[dev] Prevented tauri and rust analyzer artifact clashing

Ill look into it but why are you running release on dev?

pnpm dev fails with the same error message.

this works on windows again, ill see what I can do

hmm, i think it might be easier to override rust analyzer output location than it is to override tauri, seems there is some behavior that's not the same on linux that causes using a different output dir to break the loading


Clear label cache when fonts become available

With this pull request, web fonts will be used when they become available, just like in OpenLayers < v4.4.0 when fillText/strokeText was used. This fixes the problem mentioned in https://github.com/openlayers/openlayers/issues/7334#issuecomment-335154208.

New version of this pull request pushed, and ready for review.

In the vector labels example, I'm seeing isLoading() return true all the time.  Not clear to me that it is working as intended yet.

I'm uncertain about the return on investment here.  There are a number of cases to consider.
Below, "a label cannot be properly rendered with a font" means that none of the listed font families are available or are missing a needed glyph for one of the characters in the label.  When this happens, specifying another fallback font can change the rendering.
Consider rendering a single label.

User specifies font: 12px FontA;
i. The label can be properly rendered.
ii. The label cannot be properly rendered.
a. FontA is not available and will never be loaded.
b. FontA is loading.
a. When FontA loads, the label can be properly rendered.
b. When FontA loads, the label still cannot be properly rendered.
c. FontA will be loaded at some point in the future.  Same two sub-points as above apply here.
User specifies font: 12px FontA, FontB;
i. The label can be rendered using FontB as a fallback.
a. FontA is not available and will never be loaded.
b. FontA is loading.
a. When FontA loads, the label can be rendered without falling back to FontB.
b. When FontA loads, FontB is still used as a fallback.
c. FontA will be loaded at some point in the future.  Same two sub-points above apply here.
ii. The label cannot be properly rendered with FontA or FontB.
a. FontA and FontB are not available and will never be loaded.
b. FontA or FontB are loading.
a. When fonts load, the label can be properly rendered.
b. When fonts load, the label still cannot be properly rendered.
c. FontA or FontB will be loaded in the future.  Same two sub-points above apply here.

The trick of adding a fallback and checking if rendering width changes doesn't work for 2.i above.  And I think this would be pretty common.  If a user is loading fonts with @font-face, I imagine they will specify a fallback to be used before the preferred font loads.  If we decide whether or not we can cache rendered labels by appending another fallback, we will incorrectly cache a labels.
There are other cases above where the "cache only if adding a fallback doesn't change rendering" strategy is not so great.  In 1.ii.a, for example, we do a lot of extra work.
It feels like the only reliable way to deal with cached labels is to invalidate the cache any time a font loads.  So, where available we could:
document.fonts.addEventListener('loadingdone', invalidateLabelCache);

Of course this doesn't work for IE or Edge.  One option would be to only cache if document.fonts is available.

@tschaub I reworked this pull request again, and took advantage of the font family parser that you added. The pull request now does what you suggested: it invalidates the label cache when a previously unavailable font family becomes available, but does not rely on document.fonts for checking. In addition to that, it makes it so layer renderers listen to clear events on the label cache, and redraw to reflect changes immediately.


Execution should halt if Select2 instance is null

The following was add to the ret = line:
if (instance)

    this.each(function () {
      var instance = $(this).data('select2');

      if (instance == null && window.console && console.error) {
        console.error(
          'The select2(\'' + options + '\') method was called on an ' +
          'element that is not using Select2.'
        );
      }

**          **if (instance)** ret = instance[options].apply(instance, args);**
    });

Without that, we get a javascript error
In version 404 this is line 6417

I'm sorry, could you explain this further?  Also, not sure if you are aware, but GitHub lets you link to specific lines of code.  So rather than say:

In version 404 this is line 6417

It would be easier for others if you could link directly to that line.

sorry I'm newish to Github. We hit a scenario where the variable instance came back as null or undefined from this statement
  var instance = $(this).data('select2');

Then the line down below breaks with a null reference exception:
ret = instance[options].apply(instance, args);

So changing it to this seems to fix it with no ill effects
if (instance) ret = instance[options].apply(instance, args);

Its not obvious to me how to find that line of code since I have a select2.full.js here whereas the git entries appear to be a number of smaller files.

So actually, you can use GitHub's search to search for entire lines of code.  Plugging that line into search yields this: https://github.com/select2/select2/blob/062c6c3af5f0f39794c34c0a343a3857e587cc97/src/js/jquery.select2.js#L37
The repo actually consists of a number of smaller components (which is considered a best practice for any piece of software; see the single-responsibility principle).  These then get compiled into the monolith distribution files.
Anyway, I'm confused though by the debugging block above:
      if (instance == null && window.console && console.error) {
        console.error(
          'The select2(\'' + options + '\') method was called on an ' +
          'element that is not using Select2.'
        );
      }

I can't imagine under what conditions that window.console and console.error are not defined.  What browser are you using?  It might be better to get the Javascript error when instance is null and the console is unavailable, rather than silently skipping the .apply call.
Anyway, welcome to open source!

We mainly test with firefox and internet explorer, but also edge and chrome. Most often we'll run in internet explorer since that allows native JS debugging, so we might not (or might) have tested this specific error in the others.
If I get some time in the next few days I'll try to catch the error in the debugger, and then I'll update here.
Thanks for the explanations - makes sense :)

OK that was easier than I thought - the error happens in IE and firefox and console.error does not stop execution in either case. So it does enter that IF but it doesn't stop the following ret= assignment line from being executed.

This fiddle shows that console.error doesn't stop execution (I initially thought it might be our "window.onerror" handler, but no, it always doesn't stop execution)
https://jsfiddle.net/c0kmhdpc/1/

Thanks for looking into this.  Yeah, it's unclear whether execution should be manually halted in that check, since it will halt anyway when it fails the dereference (as you pointed out).
We do actually have a method for determining if Select2 is initialized already: https://select2.org/programmatic-control/methods#checking-if-the-plugin-is-initialized
I'm not sure which method is preferable though, yours or the one we have documented.

I suppose you could change it to a throw? Which is what we do.. but I guess you want to be consistent with the rest of the product.
  if (instance == null && window.console && console.error) {
    throw
      'The select2(\'' + options + '\') method was called on an ' +
      'element that is not using Select2.'
    ;
  }

Although execution "halts" - it raises an error unnecessarily in the case of ele.select2("close") since there's nothing to do. So its a shame to half execution of the callers product when all you need to do is return early. Which is another possible fix:
  if (instance == null && window.console && console.error) {
    console.error(
      'The select2(\'' + options + '\') method was called on an ' +
      'element that is not using Select2.'
    return;
    );
  }

Anyway, now we have a workaround, we're ok, but you might want to apply one of the fixes anyway.
thanks for helping 👍

it raises an error unnecessarily in the case of ele.select2("close") since there's nothing to do.

That's true.  This is a pretty generic issue that applies to any jQuery plugin, though.  I'd be interested in knowing how other popular jQuery plugins behave when you call a method before the plugin has been initialized on a selector.  Perhaps there is some kind of standard for this?

We don't have immediate plans to provide this. We are focused now to fix UI bugs. But if you open a PR with unit tests, I will be glad to review and approve if everything is ok.


Is there any reason why there's CancellationToken in every controller action?

Those actions are mostly short-running. Does it add any benefit?

Good point, it's probably not as useful with how short-running these actions are. I wanted to demonstrate the correct pattern of passing a CancellationToken through the layers from ASP.NET Core -> actions -> database/network code. It certainly can be removed if you don't need to handle cancellation mid-request.

So it's not that useful in this case? Alright ;)


Parse passwords containing colon character

Fixes #12379.

This PR has:

[ ] been self-reviewed.

[ ] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)

[ ] added documentation for new or modified features or behaviors.
[ ] a release note entry in the PR description.
[ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[ ] added or updated version, license, or notice information in licenses.yaml
[ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[ ] added integration tests.
[ ] been tested in a test Druid cluster.

Verified that the CodeQL warnings are misplaced.


The color of the texts in lists becomes red in PDF output

Hi,
I found another small issue, in typora, the color is good, but I am not sure why the color of the texts all become red in the pdf output, like this: :smile:

Thank you!!!

I am also seeing this behavior running Typora 0.9.68(beta) on Windows 10.

Sorry about the late response, college really takes a toll on you in the last month or so.
It seems like when it exports to a PDF (or an HTML file), it puts back the regular way how list items are rendered. In Typora, lists are organized like this:

Item 1

Item 2

Since there's no real way to style the bullet/numbering directly, this would let me do a workaround (albeit a tad hacky), by styling the list (
) with the red/blue color first, & then subsequently style anything in the list item (in this case,
) with the text color.
But when exported, it turns into this:

Item 1

Item 2

So now the workaround won't work since it will only take the list styling and not the item styling.
I could replace the bullet and numbering with psuedo-elements like before, and while this would work alright for bullet lists, it'll bring back the same problem from #4 for numbered lists. So in the end, I think the best solution to this is to set the color of the whole item (bullet and text) to the text color. I'll fix that into the next release if that's alright.
TL;DR: Lists exported from Typora bypasses the workaround to color bullets and numbering, so the best option is just make the whole item the same color as the text.


add mc tagline

add tagline to display modular credits e.g. "Modular Credits: 4"

LGTM!

Codecov Report

Merging #40 into master will not change coverage.
The diff coverage is 0.00%.

@@            Coverage Diff            @@
##             master      #40   +/-   ##
=========================================
  Coverage     71.10%   71.10%
  Complexity      373      373
=========================================
  Files            68       68
  Lines          1156     1156
  Branches        110      110
=========================================
  Hits            822      822
  Misses          304      304
  Partials         30       30

Impacted Files
Coverage Δ
Complexity Δ

src/main/java/seedu/address/ui/ModuleCard.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3b37b44...368977e. Read the comment docs.


[PE-D][Tester C] DG Format

DG would be easier to read if there was more spacing between the lines, and perhaps some keywords bolded.

Labels: severity.VeryLow type.DocumentationBug
original: jiayr99/ped#13

Update UG/DG: Bold / Spaces. It's Optional, but a good recommendation from the Tester


dotPeek shortcut not created

Possibly didn't install correctly. Check if the dependency .NET 4.5 is installed.


Feedback smiles are off-center in high contrast theme

The focus border also shifts the images a few pixels

End result for focus (first) and enabled (second) styles:

/cc @bgashler1


Don't wrap by default

I think we should NOT wrap anything but the CRUD operations by default.  I find myself basically saying wrap: false a million times for my other end points.

I'm fine with that!


change codeSystem to use inactive as well as deprecated

close #23
and cleanup new build warnings

https://build.fhir.org/ig/IHE/FormatCode/branches/moehrke-inactive/index.html

would trigger a publication of a patch release level.


Use Container#walkRules instead.

70% 1/1 build moduleshttp://localhost:8080/
webpack result is served from /
content is served from ./public
62% 45/51 build modulesContainer#eachAtRule is deprecated. Use Container#walkAtRules instead.
Container#eachRule is deprecated. Use Container#walkRules instead.
Container#eachDecl is deprecated. Use Container#walkDecls instead.
4907ms build modules
10ms seal
11ms optimize
13ms hashing
48ms create chunk assets
16ms additional chunk assets
890ms optimize chunk assets
1ms optimize assets
16ms emit

@brw31415 Thanks for reporting this. Updated and fixed in here https://github.com/alicoding/react-webpack-babel/commit/8bbfa05944ba8011d74174be14f149fff2f39d64.


[0.9.0] doc: rearranged on-chain registration

Proposed changes
Removed on-chain web app registration, and pointed to the correct location within the Autonolas Protocol docs.
Added a section to point to CLI registration, but it has not been completed so far, as some issues have been identified that require additional checks.
Fixes
If it fixes a bug or resolves a feature request, be sure to link to that issue.
Types of changes
What types of changes does your code introduce? (A breaking change is a fix or feature that would cause existing functionality and APIs to not work as expected.)
Put an x in the box that applies

[ ] Non-breaking fix (non-breaking change which fixes an issue)
[ ] Breaking fix (breaking change which fixes an issue)
[ ] Non-breaking feature (non-breaking change which adds functionality)
[ ] Breaking feature (breaking change which adds functionality)
[ ] Refactor (non-breaking change which changes implementation)
[ ] Messy (mixture of the above - requires explanation!)

Checklist
Put an x in the boxes that apply.

[ ] I have read the CONTRIBUTING doc
[ ] I am making a pull request against the main branch (left side). Also you should start your branch off our main.
[ ] Lint and unit tests pass locally with my changes
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have locally run services that could be impacted and they do not present failures derived from my changes

Further comments
If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc...

Yes. CLI documentation will be added when I have the opportunity to test what we discussed.


mesa 19.2.0

Created with brew bump-formula-pr.

failures are unrelated:
==> brew test --verbose libepoxy
Testing libepoxy
==> /usr/bin/gcc test.c -lepoxy -o test
.
Last 150 lines from /home/linuxbrew/.cache/Homebrew/Logs/libepoxy/test.01.gcc:
2019-09-28 18:55:59 +0000

/usr/bin/gcc
test.c
-lepoxy
-o
test

/home/linuxbrew/.linuxbrew/lib/../lib/libepoxy.so: undefined reference to<EMAIL_ADDRESS>/home/linuxbrew/.linuxbrew/lib/../lib/libepoxy.so: undefined reference to<EMAIL_ADDRESS>/home/linuxbrew/.linuxbrew/lib/../lib/libepoxy.so: undefined reference to<EMAIL_ADDRESS>collect2: error: ld returned 1 exit status
Error: libepoxy: failed
Failed executing:
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:1914:in `block in system'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:1851:in `open'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:1851:in `system'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libepoxy.rb:65:in `block in <class:Libepoxy>'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:1727:in `block (3 levels) in run_test'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/utils.rb:479:in `with_env'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:1726:in `block (2 levels) in run_test'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:866:in `with_logging'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:1725:in `block in run_test'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:1964:in `block in mktemp'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/mktemp.rb:57:in `block in run'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/mktemp.rb:57:in `chdir'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/mktemp.rb:57:in `run'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:1963:in `mktemp'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:1719:in `run_test'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/test.rb:30:in `block in <main>'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/2.3.0/timeout.rb:91:in `block in timeout'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/2.3.0/timeout.rb:33:in `block in catch'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/2.3.0/timeout.rb:33:in `catch'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/2.3.0/timeout.rb:33:in `catch'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/2.3.0/timeout.rb:106:in `timeout'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/test.rb:29:in `<main>'
==> FAILED


storage: a way to abort a request

It seems there is no way to abort an upload or download request created by file.createWriteStream() and file.createReadStream() respectively.
Node's native HTTP client has request.abort(), but as far as I can tell the above methods return simple through2 streams and keep the raw HTTP requests / responses hidden.
So is there a way to abort a request in progress?

You should be able to destroy the stream with stream.destroy() and assuming that validation is turned on (on by default), the upload/download will fail. Can you try that and let us know how that works?

stream.destroy() seems to work great, thanks!

Perfect! Glad we could help! :smile:


Pared down playlist with description

We should also get description from original playlist and then put it into pared down playlist.
Maybe also add link to this project page?

Added with redesign.


Ping instance owners when their instance disappears

Makyen's idea:

RFE: A single chat-ping to the owner of an SD instance once MS hasn't heard from it in a day.

Seems reasonable to me, possibly configurable at a user level. Thoughts?

Yes, just needs someone to find time to do it

Can this just be a daily job that pings all owners whose instances havn't been active for the last 24 hours but have been active in the last 48?

Wonder if there's a way to make it more immediate - would be nice to not have to wait up to 24h for the system to work.

Could always make it an adjustable threshold, i.e. "ping if instance hasn't been seen in X minutes". Either via site settings or via a runner-only or per-instance user setting so runners can choose how long they want to wait.


Add USD prices for BTC, ETH, MATIC

Tweet the us prices right after the Indian prices.
Each BTC, ETH,MATIC must be in a separate tweet.
Use the coingecko api

Hey can i get this :)

@theyashwanthsai Can I work on this

Can i get to work on this

@KunalMaverick @TejasB02 @Vaarun-Kamath
Anyone can work on the issues for the first hour :)

Does the presentation of the tweet matter? Like using the Rupee and Dollar symbols, Formatting the tweet etc.

assign me again


GEODE-3520: isValid API made public

* In case a asynchronous index update corrupts the index, the index will be marked invalid
* Queries will not use that index while executing queries.
* The API isValid had been made public so that users can find out if the index ia valid or not
* A new field has been added to the list index gfsh command which shows the isValid flag.
* If while loading the data index during its creation and it causes an exception, the index creationg will fail.

Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[x] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[x] Is your initial contribution a single, squashed commit?

[x] Does gradlew build run cleanly?

[x] Have you written or updated unit tests to verify your changes?

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to<EMAIL_ADDRESS>

Please do not review yet, as the ticket may change after the discussion is over on the dev thread and we will have to implement the same behavior for the synchronous update of the indexes too.


⚠ Bofrost.at has degraded performance

In 5bd11ed, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2806 ms

Resolved: Bofrost.at performance has improved in 7d3e732 after 7 minutes.


Feature/github javascript for github configuration refactor

Purpose
The current Github project configuration widget uses a mix of jQuery and mako. The mako template receives it's context from AddonGitHubNodeSettings#to_json, which has been deprecated.There is currently no JSON endpoint for fetching information about a project's Github addon settings. This differs from the other file storage addons. More importantly, this endpoint is necessary prerequisite if we want to render the Github configuration widget client-side
Completes milestone https://github.com/CenterForOpenScience/openscienceframework.org/milestones/Github configuration refactor
Changes
.mako files now uses Knockout data-binding to process the front-end. JSON end-points have been added & corresponding view & helper functions written.Test cases added to check the functionality of newly added view functions.

Closing this for now until this gets back into the priority queue

@lbanner @GaryKriebel We may want to revisit this next epic, as it solves a number of inconsistencies in the Github settings UI. Perhaps integrate or adapt this with the proposed project settings changes?

OK


Add precomputations to HAAQI

There are two banches working with HAAQI to improve the metric. However, after several analysis, I realised that many of the slowest parts can be improved by precomputing values. These values correspond computations that are not related to the signals. Additionally, as HAAQI works at 24000 Hz, many of the computations that depend on the sample rate can also be precomputed:

Filter coefficients in eb.env_compress_basilar_membrane:
Depends on a constant (flp = 800) and sample rate

Filter coefficients in eb.middle_ear
Depends on sample rate

Group delay vector in eb.group_delay_compensate
These are constant

Sin and Cos in eb.gammatone_bandwidth_demodulation
This is a recursive function (every 300 samples) that depends on the length of the audio signal (i.e., 24000 samples per second of audio) and the sample rate. However, considering values to the 10th decimal place, the function is periodic.

Yes. This is all true. The solution is probably to make an OOP version of these metrics.  If you look at the MSBG code you will see that there is an Ear class which has a Cochlea object. A set_audiogram method will build a new Cochlea for the Ear. This allows all the precomputation to be done in the object constructors so that the Ear's process() method doesn't repeat the parts that are signal independent.  It would be great if the Kates' models were all rewritten in this style.
Note, also that the MSBG code has its own implementation of a gammatone filterbank.  See gammatome_filterbank in cochlea.py.  This looks like the cascaded filter implementation. Here it is just a loop of lfilter commands, so no use of any njit code.  This would probably be much faster if we could show that it gave similar results. But I haven't benchmarked it.

I checked the periodicity and found that some are not "perfect" I mean, you can find the period (no more than 300 samples) However, the values for some bands are the same in each period with variation in the 15th decimal place but, for other bands, the variations are in the 3rd or 4th decimal place. repeating the first period resulted in a reduction of 0.02 HAAQI in the example I was running from 0.12 to 0.10.
Regarding refactoring HAAQI, there are many computations that can be done in the constructor and others that could be saved:

There are many constants and 3 or 4 filters' numerator and denominator that depends only on sample rate that can be precomputed and saved as constant in the class.

The sin and cos for the gammatone depends only on the length of the signal, so they could be computed only the first time the metric is called. Pre-saving this computation for a specific max length can be another option but, may increase the clarity package size. We could add this as part of the constructor and compute only if precomputations are not given.

Profiling the original HAAQI, one third of the computation are the number of correlations.

HAAQI's Ear model is hard to separate as the MSGB does.
The main issue is an alignment step that run for all the bands (32 times)
https://github.com/claritychallenge/clarity/blob/6b98a25bb13e75458fb58fdc2b76be0059a5a9cb/clarity/evaluator/haspi/eb.py#L240-L248
In this part of the Ear model, the processed signal is aligned with the reference signal. This makes this step of the process of the reference signal a requirement for properly process the processed signal.
As we can't really make the ear model independent of the signal we could have the following structure.
Note, from now on, I will called the processed signal as enhanced signal to avoid confusions
EarModel Class:

__init__: In the constructor we can compute all the parameters that depends on frequency sample to avoid repetitions.
set_audiogram: Here we can compute parameters that depend on audiogram but are common for reference and enhanced
process_reference:  Here we compute the reference ear model.
process_enhanced:  Here we compute the processed ear model. We can condition this method to the execution of process_reference

For me, it is not an ideal structure but I can't think in another. @jonbarker68, any suggestions?

Analysing the ear model, it has 2 functions using almost all the execution time: envelope_align and gammatone_basilar_membrane.
The next figure show the execution time for 15-second signals. The total execution takes 8.3 seconds.

The  envelope_align takes 2.6 seconds. However, it may not be necessary to use the whole signal for the alignment, maybe we can select one second to find the delay, which it is not expected to be more then 200 samples
The gammatone_basilar_membrane is the green box next to envelope_align and it takes 1.7 seconds to execute.

These precomputations don't improve the time take by the metric and add complexity to it.
Closing the issue


Backport ipfilter handler

Motivation:
The ipfilter handler does not exists in 4.0 yet.
Modifications:
Backport the ipfilter from 4.1 to 4.0.
Result:
It's possible to use the ipfilter handler in 4.0 as well.

@trustin please check

LGTM sans the single comment about enum naming. Could you fix it for all branches before pulling this in?

Cherry-picked as a0bde17effbedc21d8492e7f539e0824f01867a6


goneovim should open in home directory

OS: macOS Sonoma 14.3.1
Goneovim: 0.6.8, from Homebrew

Open goneovim by clicking on the icon
Type :e
This opens a list of directories and files in the root directory /
It should open a list from the home directory ~/

@carlmueller
Hi,
As a result of the discussions in #43, #325, and #337, the current working directory (CWD) of goneovim is now set to / by default. However, by including the following in init.lua, users can set the current directory to whatever they prefer.
Does this meet your expectations?
if vim.g.goneovim then
  vim.cmd('cd ~')
end

Yes, thanks.


NotoSansArabicUI: remove unused anchors that cause build error

Fixes https://github.com/googlei18n/fontmake/issues/447
This removes all the extra unused anchors from the following glyphs:
"twodotsverticalabove-ar.small",
"twodotshorizontalabove-ar.small",
"threedotsdownabove-ar.small",
"fourdotsabove-ar.small",
and only keeps the "_center" anchor, which is the only one that is actually being used to create a few other composite glyphs, namely "dyeh-ar", "dyeh-ar.fina", "nyeh-ar", "nyeh-ar.fina", "tcheh-ar", "tcheh-ar.fina", "tcheheh-ar", "tcheheh.fina".
By removing those extra anchors, we work around an error that occurs while building the mark feature, which is caused by the fact that those glyphs are being treated as mark glyphs by the MarkFeatureWriter (as they contain _-prefixed anchors), and they refer to more than one mark class (_center and _top), and feaLib rightly fails to compile when a mark lookup contain overlapping mark classes.
Besides, these particular glyphs are marked as non-export (they are only used to create composite glyphs), so ideally the feature compiler should exclude them from the generated mark feature.
Currently however in fontmake this subsetting stage that gets rid of non-export glyphs happens only after the full font has been compiled. We are going to change that (requires changes to feaLib to allow to subset the feature file source), but in the mean time this patch at least allows to build the Noto Sans Arabic UI fonts without the error, and without change in functionality.

I was able to build locally Noto Sans Arabic UI after applying this patch with these command:
$ git pull
$ ./build setup
$ ./build src/NotoSansArabicUI-MM.glyphs

@punchcutter could please you take a look and confirm these removed anchors are unnecessary and can be dropped? thanks

Yes, that's the same thing I did locally and it seems fine. Since only the center anchors are used the others are pointless. It would be nice if Glyphs said something, but that would be easy to script and check.

Thanks Zachary!

Actually, not so easy to script because these components could do Automatic Alignment with a different anchor in different composite glyphs. That would be a horrible way to set things up, but the way Glyphs does Automatic Alignment allows for that. For example you could have a dot with _top,  _center, and _bottom anchors and it could be a component in three different glyphs, one with a top anchor, one with a center anchor, and the other with a bottom anchor. That doesn't really make sense, but it's possible and in this particular case there's no way to know for sure without looking.

I did check and they appear to be used only in the _center, in the composite glyphs I mentioned above.


Prepare v0.38.0

Google Cloud PHP v0.38.0
Cloud DLP v0.2.0

Refreshed GAPIC Clients. (#658)
Updated development dependency versions. (#658)

Cloud Error Reporting v0.4.3

Updated development dependency versions. (#658)

Cloud Monitoring v0.4.3

Updated development dependency versions. (#658)

Cloud Language v0.5.0

Refreshed GAPIC Clients. (#658)
Updated development dependency versions. (#658)

Cloud Video Intelligence v0.3.3

Updated development dependency versions. (#658)

System tests are running currently. Please don't merge until I report back with results.


Unrelated Answers

Hey!
I am using this API in my discord bot, and it answers with unrelated things. Idk what's the problem. I saw another issue, which didn't continue, so I opened mine. Here is the code to the same -->
@bot.listen('on_message')
async def ai(message):
	await bot.wait_until_ready()
	if message.author == bot.user:
		return
	channel = message.channel
	if message.channel.id !=<PHONE_NUMBER>54020628:
		return
	if message.channel.id ==<PHONE_NUMBER>54020628:
		reply = await bot.rs.get_ai_response(message)
		await message.channel.send(reply)

Ik it's all messed up (the spacing and stuff), but please bear with me. If u want, contact me on discord, would be much easier. Here is me USER ID --> нυηтєя#8785. Please ping me if anyone knows the solution to this!
Thanks!

Oh sorry for the problem caused. I have fixed the thing. The problem was get_ai_response(message). The message here belongs to the discord.Message class object. I fixed it by adding the content attribute to the Message object... This fixed my problem!


Unified warning API

Feature Request
There are several different places we need to track runtime behavior to provide better warnings about incorrect usage of dioxus (https://github.com/DioxusLabs/dioxus/pull/2461, #2570, #2277). We need to support a way to opt out of these warnings for libraries and dioxus itself
Implement Suggestion
We could add an attribute like #[clippy::allow] that disables a warning within a scope with a thread local (the dioxus runtime or another local):
#[dioxus::allow(signal_write_while_running_component)]
signal.write()

The attribute syntax is not feasible because rust doesn't let you attach custom attributes to statements. We could still implement a closure based syntax for statements and an extension trait for futures. Both of which can be backed by a guard API :
#[tokio::main]
async fn main() {
    // Anything under the allow closure will not trigger the lint
    allow(&lint, || {
        trigger_warning();
    });
    // Any future executed in the future with the lint allowed will not trigger the lint
    async {
        tokio::time::sleep(std::time::Duration::from_secs(1)).await;
        trigger_warning()
    }.allow(&lint).await;
}


Beta: Stryker can get confused by its own reports directory

Question

I'm making this a question for now but perhaps it should be labelled a bug. I'm using the beta, and worked around the issue https://github.com/stryker-mutator/stryker/issues/2465 for now, but now after instrumentation nothing happens. I have detailed logging on, and I left it running for hours, but the last thing in the log is:
$ npx stryker run --fileLogLevel trace --logLevel debug
...
21:32:56 (23497) DEBUG Instrumenter Instrumented src/client/workspaceSymbols/parser.ts (120 mutant(s))
21:32:56 (23497) DEBUG Instrumenter Instrumented src/client/workspaceSymbols/provider.ts (33 mutant(s))
21:32:56 (23497) INFO Instrumenter Instrumented 768 source file(s) with 48482 mutant(s)

and nothing happens after that.
Config:
{
  "$schema"<EMAIL_ADDRESS>  "packageManager": "npm",
  "reporters": [
    "html",
    "clear-text",
    "progress"
  ],
  "testRunner": "mocha",
  "buildCommand": "npx gulp prePublishNonBundle",
  "mutate": ["src/client/**/*.ts"],
  "coverageAnalysis": "perTest"
}

This is for the Python extension for Visual Studio Code, so it should be reproducable.
It's very possible that my configuration is not right, but I still wouldn't expect a hang like this.
Stryker environment

├─┬ UNMET PEER DEPENDENCY<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ │ ├─┬<EMAIL_ADDRESS>│ │ ├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ └──<EMAIL_ADDRESS>├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>├─┬<EMAIL_ADDRESS>│ ├──<EMAIL_ADDRESS>deduped

├──<EMAIL_ADDRESS>
Additional context
I haven't added any config for the typescript checker as the build command makes use of multiple tsconfig files for different parts of the code.

check for sure if you have good packages, i can see some 3.3.1 versions here. + i think we are not using transpilers now (i dunno if it beta.4 tho). Also please provide full stryker.log file :)

I pointed you to the repo I am running against, which together with the config file is everything you would need to repro (except for bug 2465, which I had to work around - the modified repo is at this fork/branch: https://github.com/gramster/vscode-python-me/tree/stryker). I can see that the Stryker process is still consuming CPU but I don't know what it is doing. Full log file is too large to attach but I don't think it will tell you anything more useful than what I posted above. Some of the old packages are there because I started with Stryker 3 and then switched to beta; I don't think they should have any impact but I can remove them.

I removed all the Stryker packages and then ran:
npm install --save-dev @stryker-mutator/core@next @stryker-mutator/mocha-framework@next @stryker-mutator/mocha-runner@next @stryker-mutator/typescript-checker@nex

which resulted in:
➜  vscode-python-me git:(stryker) ✗ npm ls | grep stryker
├─┬ UNMET PEER DEPENDENCY<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ │ ├─┬<EMAIL_ADDRESS>│ │ ├──<EMAIL_ADDRESS>deduped
│ ├─┬<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ └──<EMAIL_ADDRESS>deduped
├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ ├──<EMAIL_ADDRESS>deduped
├──<EMAIL_ADDRESS>├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ ├──<EMAIL_ADDRESS>deduped
├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>
and stryker still gets stuck.

I broke in with a debugger and got this call stack:

It turned out it was stuck trying to process files created by Stryker itself in the reports/ directory. After deleting that folder it started making progress again. Stryker should probably be ignoring that directory.

Awesome work you're doing here @gramster This is helping us a lot 🎉

It turned out it was stuck trying to process files created by Stryker itself in the reports/ directory. After deleting that folder it started making progress again. Stryker should probably be ignoring that directory.

I think this is a bit "by design" 😅 .  So we assume you're putting reports in your ignore file. Maybe we should add it during init.
As for the strip comments preprocessor. It's actually a bug that is already fixed, but we didn't release a new version of the beta. Let me do that right now, currious to see if that fixes a lot of issues you're having

@gramster I think this can be closed, or is there still something to be done here?

Closing this for now. Don't hesitate to contact me if you think it is a mistake 👍


Recategorize API reference

Previously, our API reference had 8 sections in the main body and sidebar, and every module in the API reference was in exactly one section.
We're reducing the number of sections to 5:

Catalog
Collaborate
Analyze
Scale
Account

And reorganizing the modules within these new sections like so:

Domain-specific modules (SOMA, VCF, etc) are merged into the new "Analyze" category.
The previous "Compute", "DAG", and "Utilities" categories are merged into the new "Scale" category.
Modules of the previous "Cloud Services" category are distributed to "Catalog", "Collaborate", "Scale", and "Account".
There is overlap between the new "Catalog" and "Collaborate" sections. Some API modules are in both. That is because the Cloud-Py API has historically been organized around types of assets (groups, arrays, files, notebooks). The new categories cut across some existing modules.

Important details

This PR does not add any new API documentation for modules, it only moves existing documentation to new categories.
Placeholder descriptions for categories are used and should be replaced by real, useful text that is aligned with the website.
The quartodoc software that we're using for the API reference appears to have some deficiencies. For example, we've put a lot of work into making sure that functions in all vcf submodules are exported from tiledb.cloud.vcf, but quartodoc seems unable to produce API documentation for that high-level module.

Summary
Let's fix any obvious errors in categorization of modules now and then iterate as we progress.

Placeholder are gone, replaced by meaningful text.


Setup CI testing

Once we have a test or two committed we should turn on testing. We may want to be clever about how we do this to minimise the carbon use of our tests and use this as an example for the turing way book chapter (see #4).

Minimise carbon overhead https://github.com/marketplace/actions/pytest-last-failed

We agreed to test against Python 3.8 and 3.11.

turn on black checking too

basic CI testing is now setup, we might benefit from some more extensive testing though. But closing this issue as CI is now in place.


Drop support for Rumprun

rustc dropped the support for Rumprun in https://github.com/rust-lang/rust/pull/82594. We could drop the support on libc as we did for CloudABI.
cc #2090

:+1: to removing this support, since Rust did.
We may want to establish a policy here. Under what circumstances can we remove support for a target? Can we always do so when Rust does, or do we have any special considerations to take into account if the target was previously tier 1/2 and available in stable? Can we count on projects to have older versions of libc in their lockfiles if they need to run on older targets?
I think the answer is that we can remove targets when Rust does, but this seems worth confirming and documenting.

Note: I'm holding this off due to circumstances.
Agree, I also think we can drop support for Tier 3 once rustc does. And for tier 1/2, I'm not 100% sure but does rustc suddenly drop their support without downgrading generally? If rustc does it, I would think there was a reasonable reason (and we should consider if libc also should drop the support, it will highly depend on the situation, I imagine). I'm happy to document this as a "target support policy" or something else if needed.
@rust-lang/libc Feel free to comment here if you have different opinions.

Agree, I also think we can drop support for Tier 3 once rustc does. And for tier 1/2, I'm not 100% sure but does rustc suddenly drop their support without downgrading generally?

Historically, Rust targets have dropped from as high as tier 1 to tier 3 as a single change, e.g. https://github.com/rust-lang/rfcs/pull/2837/files but have not been abruptly removed from tier 1 or tier 2.


Samples of predictions and answers are not equal, {len(preds)}: {len(gts)}

Hi,
I am trying to run the "run.sh" file, but I have the following error:
fairseq-generate: error: argument --task: invalid choice: 'translation_from_pretrained_bart' (choose from 'translation', 'translation_lev', 'language_modeling', 'sentence_prediction', 'multilingual_translation', 'translation_moe', 'semisupervised_translation', 'sentence_ranking', 'multilingual_masked_lm', 'cross_lingual_lm', 'audio_pretraining', 'translation_from_pretrained_xlm', 'denoising', 'masked_lm', 'legacy_masked_lm') Traceback (most recent call last):   File "/content/drive/MyDrive/Colab_Notebooks/PLBART-main/scripts/text_to_code/evaluator.py", line 46, in <module>     main()   File "/content/drive/MyDrive/Colab_Notebooks/PLBART-main/scripts/text_to_code/evaluator.py", line 29, in main     assert len(preds) == len(gts), f"Samples of predictions and answers are not equal, {len(preds)}: {len(gts)}" AssertionError: Samples of predictions and answers are not equal, 0: 2000 Traceback (most recent call last):   File "/content/drive/MyDrive/Colab_Notebooks/PLBART-main/evaluation/CodeBLEU/calc_code_bleu.py", line 34, in <module>     assert len(hypothesis) == len(pre_references[i]) AssertionError
Any idea please to fix this ?

What version of Fairseq are you using?

fairseq==0.9.0

I can see the error is due to the invalid task choice "translation_from_pretrained_bart" which is actually not part of fairseq==0.9.0. I see in v0.10.0, the task was introduced (https://github.com/pytorch/fairseq/tree/v0.10.0/fairseq/tasks).
I am using fairseq 0.9.0 but probably a later commit which consists of the translation_from_pretrained_bart task. Can you please try using fairseq v0.10.0?


Can't change StatusBar backgroundColor

Applying the backgroundColor prop to StatusBar component doesn't get applied. I need to set the background colour of SafeAreaView to get the effect i want, which is not desired because on iPhone X it will have that same colour at the bottom of the screen where the "notch" is

@vTrip Could you please share with us your workaround for this issue?


All Flutter CI builds should use pinned, version-controlled CIPD versions

In Flutter's .ci.yaml format, any cipd dependency that does not have an explicit version will default to the "latest" ref (src).
This makes builds potentially non-reproducible, and also makes it difficult/impossible to determine exactly which versions of dependencies were used to make a build for artifact attestation.
Also, without pinning and version controlling CIPD versions, if a malicious CIPD package is published, new builds would immediately begin using the compromised package, without further code review.

cc @zanderso


Introduce Interfaces for AuthN and AuthZ.

Introduce the hooks that make sure, only authenticated and authorized Identities can access the resources.
Introduce test cases that test that new functionality, while maintaining backward compatibility.

This looks VERY good =).
Does any of the test check that the app/group index endpoints return only the apps/groups that the principal is allowed to see?

No there are none. If there are no general concerns, I can add tests for those cases.

@gkleiman added auth module, plugin flags and wiring. Rebased. Ready to merge.

I wonder if writing plugin interface in Scala is good decision. I think plain old Java interface suits better here because it's easier to implement in any JVM language, has less dependencies, so it's more lightweight and has backward compatibility with new releases of JVM languages which is not common in Scala world. In other world when jar with plugin-interface is build with specific version of Scala and some day you will update Scala in in this project all libs will need to be rebuild even if there were no change in interface itself.
@aquamatthias would you push this jar to some Maven snapshot repo, so I can easily start working on plugin?

Hey @janisz. I don't see a problem in defining interfaces in scala. I think the point is, that this interface should be usable from Java as well. And I completely agree with this. We will create an example Plugin in Java to show this is possible.

@janisz we will publish the interface jar publicly in the future. For now you can do an sbt plugin-interface/publishLocal to have it in the local repository.

I'm just worried about binary compatibility of this interface. Will my plugin works after you update Scala version (no changes in code)? If yes then there is no problem with Scala.

@aquamatthias I created simple Shiro based plugin https://github.com/janisz/marathon-shiro-authentication-plugin
How to enable it in marathon?
I used following configuration flags --plugin_dir /tmp/shiro --plugin_conf /tmp/plugin_conf.json and plugin_conf.json:
{
  "plugins":[
    {
      "plugin":"mesosphere.marathon.plugin.auth.Authenticator",
      "implementation":"pl.allegro.tech.marathon.plugin.auth.ShiroAuthenticator"
    },
    {
      "plugin":"mesosphere.marathon.plugin.auth.Authorizer",
      "implementation":"pl.allegro.tech.marathon.plugin.auth.ShiroAuthorizer"
    }
  ]
}

but application throws

Plugin not found: PluginDefinition(mesosphere.marathon.plugin.auth.Authorizer,pl.allegro.tech.marathon.plugin.auth.ShiroAuthorizer,None)

Hey @janisz Cool! Without looking at your code, I suspect that you did not define the ServiceLoader manifest? (https://docs.oracle.com/javase/6/docs/api/java/util/ServiceLoader.html)
I just pushed an example plugin PR here: https://github.com/mesosphere/marathon-example-plugins/pull/1
Note: everything is in flux at that moment. But feedback to the interface is highly appreciated!

Cool! I'll take a look at your example

@janisz I also added a simple java based AuthN+Z Plugin into that PR for reference.


PauliEvolutionGate default label can be too long to serialize via qpy

Environment

Qiskit version: main
Python version: 3.11
Operating system: Linux

What is happening?
To serialize an EvolvedOperatorAnsatz requires serializing a label containing the entire string representing the SparsePauliOp.  For a QAOAAnsatz, which is a subclass of EvolvedOperatorAnsatz, this might look like exp(-it (IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIZZ + IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIZ + {...} )), where I have truncated and replaced many terms with {...}.
The problem is that this label can easily exceed the maximum allowed qpy label length of 65536 after encoding (as the label length is represented by a 16-bit unsigned integer).
In terms of code, EvolvedOperatorAnsatz relies on PauliEvolutionGate, which itself creates the label.
How can we reproduce the issue?
Create a QAOAAnsatz with many terms and over many (>100 qubits).  Serialize it to qpy.
What should happen?
Somehow, no error should occur.  The user should not have to modify or discard the label in order to dump to qpy.  (Actually, it's not even straightforward how I'd go about discarding or modifying this label, since in order to do so I'd need to somehow access the PauliEvolutionGate through the EvolvedOperatorAnsatz.)
Any suggestions?
One of the following:

Truncate the label in a graceful way, at the boundary between two terms if the string is getting close to "too long" (which should be < 65536 characters, but possibly much less).
Modify the qpy spec to support longer labels.

I prefer the first option unless there is a strong case for keeping the entire label intact.

hi, @garrison as by your suggestion we can implement the truncate function like this
`def serialize(self):
label = self._get_full_label()
truncated_label = self._truncate_label(label)
return truncated_label
def _truncate_label(self, label, max_length=65536):
    if len(label) <= max_length:
        return label
    else:
        index = max_length - 1
        while index >= 0 and label[index] != '+':
            index -= 1
        if index < 0:
            raise ValueError("Cannot truncate label without breaking terms")
        return label[:index] + ' {...}'`

we can do some modification in the code according to the types of input.

I think there's probably two parts to the solution here:

PauliEvolutionGate absolutely should not be producing by default labels that use that many characters.  It's not helpful to anybody, and I imagine it's slow to make as well - I agree we should definitely change that to something more sensible, arguably avoiding eagerly serialising the inner SparsePauliOp into a less efficient representation when the gate is created, even if we do partially truncate it - it seems very wasteful and prone to this sort of explosion.  I don't use the gate, though, so I don't know what might be more useful.
QPY should have an escape mechanism to avoid hard crashing - maybe we just emit a warning on too-long labels and truncate them at the maximum length.  We could consider introducing a new QPY format version to support arbitrary-length labels in some form or another too, if it's necessary.


Random crash in FL studio

CPU meter seems to spike up to 100% (red) for an instant, then everything freezes.
Sometimes it happens, sometimes it doesn't.
PROBABLY introduced after #33 (exponential envelopes) but not sure honestly, since it appears to be a rare issue.

Not 100% sure if the following is related to this: I managed to crash it in Juce's Plugin Host (after a good 10 mins of playing).
It appears we get stuck in a loop in juce_Phase.h, following code:
    Type advance (Type increment) noexcept
    {
        jassert (increment >= 0); // cannot run this value backwards!

        auto last = phase;
        auto next = last + increment;

        while (next >= MathConstants<Type>::twoPi)
            next -= MathConstants<Type>::twoPi; // ************** LOOP HERE ****************

        phase = next;
        return last;
    }


gc.root: use RFC3339 format consistently

Signed-off-by: Akihiro Suda<EMAIL_ADDRESS>Previously, Go-specific(?) String() format and RFC3339 format were mixed up

https://github.com/containerd/containerd/pull/1690 looks like removing the label, so probably this PR should be closed.
(cc @dmcgowan)


feat: nx support

Builds on top of #230, so that should probably be merged first.
I had some issues getting NX working, so this PR adds a new NX sample.
The issue I was having turned out to be that when Vite runs within the sample directory, it tries to traverse upwards in directories and then would catch the root repo's vite.config.ts file, which would have an exclude section that would exclude **/src/** and others.
Now the global vite config has been renamed to avoid it accidentally being picked up in the samples.

This is also a big source of bad reviews: https://marketplace.visualstudio.com/items?itemName=ZixuanChen.vitest-explorer&ssr=false#review-details

@ffMathy is it intended that with these changes the working directory is the containing package rather than the monorepo root?

You can use a vitest.workspace.ts file and it should work fine from the NX monorepo root.
But there are some known bugs right now that are being fixed in the next release in the coming days.
Also note that the current extension has been deprecated and there is a new link to the new extension.

You can use a vitest.workspace.ts file and it should work fine from the NX monorepo root.
But there are some known bugs right now that are being fixed in the next release in the coming days.
Also note that the current extension has been deprecated and there is a new link to the new extension.

Thanks @ffMathy. I'm using v0.3.0 of https://marketplace.visualstudio.com/items?itemName=vitest.explorer which I think has this change in it. Using the new sample, monorepo-nx, I find that the working directory is the repository root during test execution, whereas running the test task with nx will change the working directory to each package's root. Is that inconsistency expected, and if so is there any plan to address it or should we look for workarounds?

0.3.0 doesn't have the bugfixes that are incoming.
Hmmm I didn't know about the working directory thing. Perhaps I should make a fix for that and create another test for that scenario.
Before my change, NX tests wouldn't run at all. So at least there's progress.
Can you file an issue with the working directory problem?

0.3.0 doesn't have the bugfixes that are incoming.
Hmmm I didn't know about the working directory thing. Perhaps I should make a fix for that and create another test for that scenario.
Before my change, NX tests wouldn't run at all. So at least there's progress.
Can you file an issue with the working directory problem?

Will do! Thank you!


Bug : ./deploy.sh: line 33: bundle: command not found

Works fine before.
I use macos x 10.12.2 and my ruby version is ruby 2.0.0p648 (2015-12-16 revision 53162) [universal.x86_64-darwin16].
The only thing i change is my macos X version.
Thx for the response.

I install bundler with gem install bundler and now the issue is :
bundler: command not found: middleman

When i use bundle show middleman I have middle installed : /Library/Ruby/Gems/2.0.0/gems/middleman-4.0.0

vagrant up doesn't work either :/

I upgrade my ruby version to 2.3.1 but don't work either ...

With Ruby 2.3.1, does bundle show middleman work?
Closing since this isn't really a problem with Slate (it seems).

when i make bundle show middleman result is :
/Library/Ruby/Site/2.0.0/rubygems/dependency.rb:315:in `to_specs': Could not find 'bundler' (>= 0) among 19 total gem(s) (Gem::LoadError)
Checked in 'GEM_PATH=/Users/ugobourdon/.rvm/gems/ruby-2.3.1:/Users/ugobourdon/.rvm/gems/ruby-2.3.1@global', execute `gem env` for more information
	from /Library/Ruby/Site/2.0.0/rubygems/dependency.rb:324:in `to_spec'
	from /Library/Ruby/Site/2.0.0/rubygems/core_ext/kernel_gem.rb:64:in `gem'
	from /usr/local/bin/bundle:22:in `<main>'

Ah, look like ruby 2.0.0 middleman/bundler, so maybe the ruby install didn't work? We don't support 2.0.0.

when i do gem env i have :
- RUBYGEMS VERSION: 2.6.8
  - RUBY VERSION: 2.3.1 (2016-04-26 patchlevel 112) [x86_64-darwin16]
  - INSTALLATION DIRECTORY: /Users/ugobourdon/.rvm/gems/ruby-2.3.1
  - USER INSTALLATION DIRECTORY: /Users/ugobourdon/.gem/ruby/2.3.0
  - RUBY EXECUTABLE: /Users/ugobourdon/.rvm/rubies/ruby-2.3.1/bin/ruby
  - EXECUTABLE DIRECTORY: /Users/ugobourdon/.rvm/gems/ruby-2.3.1/bin
  - SPEC CACHE DIRECTORY: /Users/ugobourdon/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /Users/ugobourdon/.rvm/rubies/ruby-2.3.1/etc
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-darwin-16
  - GEM PATHS:
     - /Users/ugobourdon/.rvm/gems/ruby-2.3.1
     - /Users/ugobourdon/.rvm/gems/ruby-2.3.1@global

ruby --version
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin16]

I upgrade my virtual box install and vagrant up works now. But ./deploy.sh don't work yet.

I redo gem install bundlerand bundle install and finally works fine. thx for help.


/* generated by CMake */   => #include "cocos2d.h"   should it not be "axmol.h"?D

/* generated by CMake */

#pragma system_header
#ifdef __cplusplus
#include "cocos2d.h"
#endif // __cplusplus

Maybe @crazyhappygame can help you

Maybe @crazyhappygame can help you

And fix it;)

Didn't this latest PR fix the issue? https://github.com/axmolengine/axmol/pull/922

Didn't this latest PR fix the issue? #922

Yes


How to pass parameters to a middleware and influence its processing logic ?

Description

Hello everyone, we know that through request.state we can pass in some custom data to handlers from middleware's before-handle process and thus influence its behaviour, I'm currently wondering how can a handler influence the middleware's logic after it.
My specific business scenario is that I have a routed address (let's take /api for example) whose function is to calculate a dynamic result that will consume a relatively long time and return a relatively large json response. Operationally, an effective way to improve efficiency is to use some buffer (like redis) to cache its results, in this way, the calculation time can be saved everytime the cache hits.
Since my memory is limited, I want to store the gzip-compressed bytes in a buffer instead of raw json streams, this will greatly increase the amount of cache I could handle. Specifically, because of there's a large number of numbers in the response, a response content is usually about 20MB in size without gzip, while after compression it is about only 1MB. This means that with 1GB of memory, I can only cache 50 different responses without compression, while with compression I can cache 1000, which is a significant difference that can hardly be ignored.
Because of these requirements, I wanted to implement a full-featured gzip middleware, but there were several technical confusions. The first is that I want to control the middleware whether to compress or not, apparently, if the response is not hitting the cache but is generated dynamically then it should be compressed, but on the contrary, it should not be compressed again since it has already been compressed once.  The second question is, even if I can control the middleware not to compress, how do I replace its results with bytes that have already been compressed when there's no need to run the compress logic?
Example
Since I yet don't know how to implement, forgive me that I can only provide some pseudo-code to illustrate my thinking.
The following code describes a relatively complex response address that does not contain middleware:
from fastapi import FastAPI, Request
import uvicorn

app = FastAPI()

@app.post("/api")
async def root(request :Request, some_args: int):

    # Using the Fibonacci series to simulate a time-consuming
    # computational operation
    def fib_recur(n):
  	if n <= 1: return n
  	return fib_recur(n-1) + fib_recur(n-2)

    # In addition to the time-consuming calculation, the size of
    # the returned content is also large.
    return {"response_content":  [fib_recur(31)] * 10000000 }

if __name__ == '__main__':
    uvicorn.run(app, host='<IP_ADDRESS>', port=8080)

I want to achieve the following after adding middleware:
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from utils import fib_recur, search_for_if_cache_hit
import uvicorn

app = FastAPI()

@app.post("/api")
async def root(request :Request, some_args: int):
    cache_hit_flag , cache_content = search_for_if_cache_hit('/api' , some_args)
    if cache_hit_flag == True:
        # Skip Calculation
        response = SomeKindOfResponse(message_content = cache_content)
        response.store['need_gzip'] = False
    else:
        # Calculate normally
        response = JSONResponse( {"response_content":  [fib_recur(31)] * 10000000 } )
        response.store['need_gzip'] = True
    return response

@app.middleware("http")
async def gzip_middleware(request: Request, call_next):
    response = await call_next(request)
    if response.store['need_gzip'] == True:
        response = gzip_handler(response)
    else:
        pass
    return response

if __name__ == '__main__':
    uvicorn.run(app, host='<IP_ADDRESS>', port=8080)

Thanks!
Environment

OS: [e.g. Linux / Windows / macOS]: Python3.8.6
FastAPI Version [e.g. 0.3.0]: 0.63.0

Have you taken background tasks into consideration? It looks applicable for your use case here.

@valentin994 Thanks for the reply, but I'm senserly consider that this isn't the answer I need. Since I need to modify the return value for each request at the moment, but not do something in the background afterwards.

@GoodManWEN Hi!
I think that it is possible to pass params via request.state, because it is the same object in middleware and handler.
Like this:
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse, Response
import random
import uvicorn

app = FastAPI()

def fib_recur(n):
  	if n <= 1: return n
  	return fib_recur(n-1) + fib_recur(n-2)

def search_for_if_cache_hit(p1, p2):
    if random.choice((0, 1)) == 1:
        return True, b'123'
    return False, b'321'

def gzip_handler(response: Response):
    # Gzip response.content here and modify response.headers
    return response

@app.get("/api")
async def root(request :Request, some_args: int):
    cache_hit_flag , cache_content = search_for_if_cache_hit('/api' , some_args)
    if cache_hit_flag == True:
        # Skip Calculation
        response = Response(cache_content)
        request.state.need_gzip = False
    else:
        # Calculate normally
        response = JSONResponse( {"response_content":  [fib_recur(3)] * 100 } )
        request.state.need_gzip = True
    return response

@app.middleware("http")
async def gzip_middleware(request: Request, call_next):
    response = await call_next(request)
    if getattr(request.state, 'need_gzip', None) == True:
        response = gzip_handler(response)
    return response

if __name__ == '__main__':
    uvicorn.run(app, host='<IP_ADDRESS>', port=8080)

@juntatalor Thanks! You reminded me that python dictionaries are structures that store pointers, which I had missed before. This means that indeed we can pass data to the middleware without relying on the return value of call_next. Now the only remaining question is how to get the middleware to return bytes directly without compression.

Now the only remaining question is how to get the gzip middleware to return bytes directly without compression.

But isn't it already done by need_gzip flag? With false value the response is returned with no changes (raw json in example). The gzip operation should be performed in gzip_handler function.

@juntatalor  Yes indeed, however I'm not familiar with http's gzip protocal, nor many python web frameworks for python implemented gzip can provide a reference. I looked a little deeper into starlette's gzip plugin source code and it seems to involve certain ASGI implementation that are generally hidden and it seems that which cannot be modified directly at the request handler level.

I admit, this is more complicated than it seemed at first.
The first thing is that starlette's GzipMiddleware works with Messages, which are generated by Response class itself. So gzip middleware does not receive Response object at all. Also there is no information about Request state in messages:
# Response.__call__
await send(
            {
                "type": "http.response.start",
                "status": self.status_code,
                "headers": self.raw_headers,
            }
        )
        await send({"type": "http.response.body", "body": self.body})

I think it is simple to use contextvars (python 3.7+) to pass use_gzip flag into middleware. It is guaranteed to be local on current async stack. Here is a minimal working example, but code should be cleared a bit:
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse, Response
import random
import uvicorn
from starlette.middleware.gzip import GZipMiddleware, GZipResponder
from starlette.datastructures import Headers

from contextvars import ContextVar

use_gzip = ContextVar('use_gzip')

app = FastAPI()

def fib_recur(n):
  	if n <= 1: return n
  	return fib_recur(n-1) + fib_recur(n-2)

def search_for_if_cache_hit(p1, p2):
    if random.choice((0, 1)) == 1:
        return True, b'123'
    return False, b'321'

@app.get("/api")
async def root(request :Request, some_args: int):
    cache_hit_flag , cache_content = search_for_if_cache_hit('/api' , some_args)
    if cache_hit_flag == True:
        # Skip Calculation
        response = Response(cache_content)
        use_gzip.set(False)
    else:
        # Calculate normally
        response = JSONResponse( {"response_content":  [fib_recur(3)] * 100 } )
        use_gzip.set(True)
    return response

class MyGZipResponder(GZipResponder):
    async def send_with_gzip(self, message) -> None:
        if use_gzip.get():
            await super(MyGZipResponder, self).send_with_gzip(message)
        else:
            await self.send(message)

class MyGzipMiddleware(GZipMiddleware):
    # subclass to modify GzipResponder -> MyGzipResponder
    async def __call__(self, scope, receive, send) -> None:
        if scope["type"] == "http":
            headers = Headers(scope=scope)
            if "gzip" in headers.get("Accept-Encoding", ""):
                responder = MyGZipResponder(self.app, self.minimum_size)
                await responder(scope, receive, send)
                return
        await self.app(scope, receive, send)

app.add_middleware(MyGzipMiddleware)

if __name__ == '__main__':
    uvicorn.run(app, host='<IP_ADDRESS>', port=8080)

@juntatalor Thanks for kind reply. I traced your code and I think we so far solved the problem. As a summary, the first problem we encountered was passing values to the middleware's post-handler logic. In the most basic way, we can called directly in the high-level api to make request.state transfer cross-sate pointers. This is a common solution, but this approach does not work since gzip a middleware based on the asgi.
As an answer to the question in the previous discussion, according to my understanding, we modify the logic of gzip by inheritance it and replace its send_with_gzip method, directly replace the message body with compressed result bytes could achieve the effect.
So the only problem left is how to pass the data. contextvars is a very nice feature to solve this problem, which I didn't know about before, thanks for introducing it to me, and thanks for the dynamic ability of python.
I'm going to close this issue, thanks once again.
btw I think you may not be aware that markdown allows you to highlight code by markup the language of code blocks, which significantly enhances the readability of the code, forgive me if you already know.

Thanks for the help here @valentin994 and @juntatalor ! 👏  🙇
Thanks for reporting back and closing the issue @GoodManWEN  👍

Sorry for the long delay! 🙈  I wanted to personally address each issue/PR and they piled up through time, but now I'm checking each one in order.


Add unit tests for main.ts

Obsolete after react rewrite https://github.com/josephrkramer/batuu-scanner2/issues/45


[SHUFFLE] [WIP] Prototype: store shuffle file on external storage like S3

What changes were proposed in this pull request?
This PR (design doc) provides support to store shuffle files on external shuffle storage like S3. It helps Dynamic
Allocation on Kubernetes. Spark driver could release idle executors without worrying about losing
shuffle data because the shuffle data is store on external shuffle storage which are different
from executors.
This could be viewed as a followup work for https://issues.apache.org/jira/browse/SPARK-25299.
There is previously Worker Decommission feature (SPARK-33545), which is a great feature to copy shuffle data to fallback storage like S3. People appreciate that work to address the critical issue to handle shuffle data on Spark executor termination. The work in the PR does not intent to replace that feature. The intent is to get further discussion about how to save shuffle data on S3 during normal execution time.
Why are the changes needed?
To better support Dynamic Allocation on Kubernetes, we need to decouple shuffle data from Spark
executor. This PR implements another Shuffle Manager and support writing shuffle data on S3.
Does this PR introduce any user-facing change?
Yes, this PR adds two Spark config like following to plug in another StarShuffleManager and store
shuffle data on provided S3 location.
spark.shuffle.manager=org.apache.spark.shuffle.StarShuffleManager
spark.shuffle.star.rootDir=s3://my_bucket_name/my_shuffle_folder

How was this patch tested?
Added a unit test for StartShuffleManager. A lot of classes are copied from Spark, thus not add tests
for those classes. We will work with the community to get feedback first, then work on removing code
copy/duplication.

Test build #146081 has started for PR 34864 at commit 4b67c8a.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50556/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50556/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50556/

@hiboyang, thanks for the work here! Could you create a design doc for this? That might help get more people attention and easier for them to understand.

@hiboyang, thanks for the work here! Could you create a design doc for this? That might help get more people attention and easier for them to understand.

Yes, good suggestion. Will create some design doc.

Test build #146081 has finished for PR 34864 at commit 4b67c8a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class ByteBufUtils
public class StarBlockStoreClient extends BlockStoreClient
public class StarBypassMergeSortShuffleWriter<K, V> extends ShuffleWriter<K, V>
public class StarLocalFileShuffleFileManager implements StarShuffleFileManager
public class StarMapResultFileInfo
public class StarS3ShuffleFileManager implements StarShuffleFileManager
   public static class S3BucketAndKey
public class StarUtils
public class StartFileSegmentWriter
class StarShuffleManager(conf: SparkConf) extends ShuffleManager with Logging
final class StarShuffleBlockFetcherIterator(
  case class FetchRequest(
  case class DeferFetchRequestResult(fetchRequest: FetchRequest) extends FetchResult

Quickly glanced through the code, seems for writing shuffle data we are writing locally first and then upload to S3, similarly for reading shuffle data we are downloading data to a local temp file first and then read.
We should be able to write/read direct to/from S3, right?

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146081/

Quickly glanced through the code, seems for writing shuffle data we are writing locally first and then upload to S3, similarly for reading shuffle data we are downloading data to a local temp file first and then read.
We should be able to write/read direct to/from S3, right?

Thanks for looking! Yes, we should be able to write/read direct on S3. This PR is a prototype. Still need to improve the code and performance of writing/reading shuffle data on S3.

Hi, All. Thank you!
BTW, for the record, Apache Spark 3.1+ already stores its shuffle files into the external storage like S3 and reads back from it.

[SPARK-33545][CORE] Support Fallback Storage during Worker decommission (Apache Spark 3.1.0)
[SPARK-34142][CORE] Support Fallback Storage Cleanup during stopping SparkContext (Apache Spark 3.2.0)
[SPARK-37509][CORE] Improve Fallback Storage upload speed by avoiding S3 rate limiter (Apache Spark 3.3.0)

It would be great not to ignore the existing Spark feature and avoid over-claiming.
Dynamic allocation is the same. Apache Spark has been supporting Dynamic Allocation in K8s too.

Right, Spark has shuffle tracking to support Dynamic Allocation on Kubernetes, but it will not work well when there is shuffle data distributed on many executors (those executors cannot be released).
The work here (storing shuffle data on S3) does not conflict with worker decommission feature. The eventual goal is to store shuffle data on S3 or other external storage directly. Before getting there, people could still use the worker decommission feature.

You are completely wrong because you already know the worker decommission feature.

but it will not work well when there is shuffle data distributed on many executors (those executors cannot be released).

You should mention this in the PR description explicitly instead of misleading the users.

The work here (storing shuffle data on S3) does not conflict with worker decommission feature. The eventual goal is to store shuffle data on S3 or other external storage directly.

Test build #146147 has started for PR 34864 at commit 8222f38.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50620/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50620/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50620/

Test build #146147 has finished for PR 34864 at commit 8222f38.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146147/

Test build #146303 has started for PR 34864 at commit 761fe2a.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50775/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50775/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50775/

Test build #146303 has finished for PR 34864 at commit 761fe2a.

This patch fails from timeout after a configured wait of 500m.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146303/

Add a design doc for this prototype.

Obviously I am biased, but I believe that rather than trying to use the AWS APIs yourself, you should just use the hadoop file system APIs and interact with S3 through the s3a connector.
For a high-performance upload of a local file, use FileSystem.copyFromLocalFile in s3a on hadoop 3.3.2 this uses the same transfer manager class as this PR does but adds: exception handling/mapping, encryption settings, auditing. And the s3a integration tests verify all this works... By the time you get to use it here you can assume the S3 upload works, and it becomes a matter of linking it up to spark.
As copyFromLocalFile is implemented for all filesystems, it means the component will also work with other stores including google cloud and azure abfs, even if they do not override the base method for a high-performance implementation -yet.
This also means that you could write tests for the feature using file:// as the destination store and include these in the spark module; if you design such tests to be overrideable to work with other file systems, they could be picked up and reused as the actual integration test suites in an external module.
And, because someone else owns the problem of the s3 connector binding, you get to avoid fielding support calls related to configuring of AWS endpoint, region, support for third-party s3 stores, qualifying AWS SDK updates, etc.
Accordingly, I would propose

ripping out StarS3ShuffleFileManager and using FileSystem APIs
write tests which can be pointed at the local FS, but then targeted at real object stores in a downstream test module, as https://github.com/hortonworks-spark/cloud-integration does for committer testing.

Getting integration tests set up is inevitably going to be somewhat complicated. I can provide a bit of consultation there.

Obviously I am biased, but I believe that rather than trying to use the AWS APIs yourself, you should just use the hadoop file system APIs and interact with S3 through the s3a connector.
For a high-performance upload of a local file, use FileSystem.copyFromLocalFile in s3a on hadoop 3.3.2 this uses the same transfer manager class as this PR does but adds: exception handling/mapping, encryption settings, auditing. And the s3a integration tests verify all this works... By the time you get to use it here you can assume the S3 upload works, and it becomes a matter of linking it up to spark.
As copyFromLocalFile is implemented for all filesystems, it means the component will also work with other stores including google cloud and azure abfs, even if they do not override the base method for a high-performance implementation -yet.
This also means that you could write tests for the feature using file:// as the destination store and include these in the spark module; if you design such tests to be overrideable to work with other file systems, they could be picked up and reused as the actual integration test suites in an external module.
And, because someone else owns the problem of the s3 connector binding, you get to avoid fielding support calls related to configuring of AWS endpoint, region, support for third-party s3 stores, qualifying AWS SDK updates, etc.
Accordingly, I would propose

ripping out StarS3ShuffleFileManager and using FileSystem APIs
write tests which can be pointed at the local FS, but then targeted at real object stores in a downstream test module, as https://github.com/hortonworks-spark/cloud-integration does for committer testing.

Getting integration tests set up is inevitably going to be somewhat complicated. I can provide a bit of consultation there.

Yes, these are great suggestions! Thanks again! I will find time to make change for this, and may also reach out to your for consultation when adding integration test :)

@hiboyang I looked at your work earlier this year and I wanted to let you know that I used it as a basis for a shuffle plugin. Ultimately I decided to rewrite the plugin from scratch (except the tests) and orient myself on the design of the Spark shuffle manager.
The code is available here: https://github.com/ibm/spark-s3-shuffle/ . It acts as an external Spark plugin and allows to be loaded into Spark binary releases as a plugin.
I'm open to contributing this work back to Apache Spark if there is any interest.

Apache Spark community is open for any contribution, @pspoerri . You can make your own PR.
BTW, do you have any sharable result in terms of the stability and performance, @pspoerri ?

Hi @pspoerri great you are working on this and thanks for letting us know! I stopped working on my previous PR due to changing of work priority, but still would like to see people continue working in this area.
There is big value if storing Spark shuffle data on S3. It will save cost and also make Spark more resilient to disk error.
During my previous experiment, shuffle data on S3 will have very worse performance. There are a lot of optimization needed, e.g. S3 key prefix randomization to avoid S3 throttling and async S3 write. Will be happy to hear your thoughts on this as well.

@michaelbilow hadoop s3a is on v2 sdk; the com.amazonaws classes are not on the CP and amazon are slowly stopping support. you cannot for example use the lower latency S3 express stores with it.
Like I say: I think you would be better off using the Hue file system APIs to talk to s3. If there are aspects of s3 storage which aren't available through the API -or just very inefficiently due to the effort to preserve the Posix metaphor, then lets fix the API so that other stores can offer the same features, and other apps can pick up.
For example, here's our ongoing delete API for iceberg and other manifest-based tables
https://github.com/apache/hadoop/pull/6726
It maps to s3 bulk delete calls, but there's scope to add to other stores (we now actually want to add it as a page-size == 1 option for all filesystems as it simplifies iceberg integration).

@steveloughran How do I call the Hue APIs from Spark? Can you point me to a package?
I agree with you that using the Hadoop APIs are not ideal performance wise, but they are great from a usability and portability perspective.
Another issue is that Hadoop wants to know the size of every file it wants to read. While this makes sense for formats like parquet where the header is located at the last few bytes of the file. It does not make sense for shuffle where you know the exact block/file you want to read.

+1, would be great to get this working and part of the Spark ecosystem.


Fix pytest warning

Thanks.!
For the record, this fixes the following warning:
WC1 None [pytest] section in setup.cfg files is deprecated, use [tool:pytest] instead.


refactoring cleaning types for SizeOf

This change is

Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


updated 404 page

Thanks @chhatrapalsinhzala007, I'll review this later today!

Hyy @jrobind.
No problem and please give suggestions to make it more attractive.

Hey @chhatrapalsinhzala007, can you please create a new branch before raising a pull request.
If you need some guidance, please follow the process outlined in this article: https://www.digitalocean.com/community/tutorials/how-to-create-a-pull-request-on-github
Thanks!

Okk @jrobind I will.


How to config  logging pattern?

logging:
  level:
    ROOT: INFO
  console: '%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n'
  file: './logs/${spring.application.name}.%d{yyyy-MM-dd}.log'

Is this a spring boot question?

spring cloud config server with spring boot

How to configure logging is a spring boot issue unless you're seeing some kind of problem.


ARROW-6960: [R] Add lz4 and zstd to R PKGBUILD

Addressing https://issues.apache.org/jira/browse/ARROW-6960 by adding lz4 and zstd, which was added in rtools via https://github.com/r-windows/rtools-packages/pull/47

There's a pretty big backup on https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/, lots of builds fighting for too few workers, so feel free to post a link to a passing build on your fork, when you get there.
Also, you can verify that the compression support is working by downloading the r\arrow_<IP_ADDRESS>00.zip artifact (the binary R package) from the Appveyor job and installing it locally (see https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/28797790/job/6l2ewckvujk48vtb/artifacts for example artifacts).
You could also add a unit test that codec_is_available("lz4") or some other exercise of the compression that only runs on Appveyor.

@gnguy I'm heading out on vacation and didn't want to leave you hanging on this so I just did the test change I suggested. Will merge when CI is green. Thanks again!

@ursabot crossbow submit macos-r-autobrew

Watching appveyor on https://ci.appveyor.com/project/nealrichardson/arrow/builds/28945982/job/iwi1txs8mscn0rl5, which will finish hours before the ASF job will start.

Appveyor passed here: https://ci.appveyor.com/project/nealrichardson/arrow/builds/28950516
Everything else was green. Merging now.
Thanks @gnguy !

Awesome, thanks so much @nealrichardson!


Ensure the external connectivity to api servers on the workload clusters

User Story
- As a Giant Swarm Engineer, I would like to know immediately if there is a problem with connectivity to an API server on a workload cluster so that I can notify the affected customer and resolve the issue.
Rationale
Currently, we only monitor the connection to the API server from the management clusters. However, it may happen that there is a connectivity issue between the API server and e.g. customer CI system. We want to know about such issues immediately.
There are potentially other systems besides the API server for which we want to monitor the external connectivity.
TODO

[ ] Talk to customers about their current monitoring for API servers externally
[ ] Find out which ready solutions are available

Ping @othylmann this is a public issue, I will be posting updates here.

@Rotfuks I think this should belong to Turtles instead if atlas. What do you think?

Hey @weseven! Since you're looking into the APIServer Alerts and Monitoring with giantswarm/giantswarm#29116 is this here also something we already looked at with that or would it be an additional check from outside the cluster?

Alright, thought as much.
So we need some external source that pings the API - could we maybe use some tekton pipelines for that? What do you think @giantswarm/team-tinkerers, would that be some check use case we could setup with your pipelines ?

It could be done using Tekton Pipeline but I don't think that is the appropriate tool for this. If I'm understanding this correctly we're basically lookup for an uptime monitor, right?
I'm having a hard time understanding what we are trying to achieve that the current monitoring stack doesn't provide.

However, it may happen that there is a connectivity issue between the API server and e.g. customer CI system.

The only way we can really monitor for this is to run something in each customers own network.
This issue seems to indicate that just because the MC can connect to the WC api-server that isn't enough to say that the customer can. Similarly, just because we're able to connect to the WC from anywhere else in our infra (Tekton) it also isn't enough to say that the customer can.
I'm also concerned at the amount of access the Tekton instance would then have. We'd be creating a single point in our infrastructure that has access to ALL of our customers clusters (assuming we're testing for actual api-server successful response).

I think one of the issue with the current monitoring stack (in-cluster agent) is that we cannot ensure that the api-server is accessible from outside. The only place we can technically act upon would be the MC so I think having something like the blackbox exporter ping the WC api server (could be once every 5 minutes) from the MC would already be a good starting point.

Hmm...Looking at the creation date of this issue it's not too critical. But I see the appeal of it. I'll add it to our Observability Improvements Epic in the backlog and we revisit it later. Thanks for bringing it up!

Thank you I removed it from Atlas board :)

@giantswarm/team-turtles for dashboarding, you might want to use this as a reference https://github.com/giantswarm/giantswarm/files/14012738/Cluster.Uptime-1705676241066.json

This is a related issue setting up infrastructure for synthetic monitoring: https://github.com/giantswarm/roadmap/issues/3225
Maybe we should wait on these results, if the priority here doesn't become more critical.

@Rotfuks on which results are you waiting? The other issue points to this one and states that you as Turtles are working on checking if Prometheus Blackbox Exporter is a good solution. To me it reads like this ticket waits on the other and the other waits on this one.
From this ticket: Maybe we should wait on these results, if the priority here doesn't become more critical.
From the other ticket: @giantswarm/team-turtles is already working on using the blackbox exporter to check that the api server is accessible and responding from the MC, let's check if that tooling is enough.
Can we just get started? We have the prometheus blackbox exporter already IIRC, it shouldn't be anything crazy to test out.

As per today's daily, waiting for the blackbox exporter to be configured & setup in MCs.


Describe how to Delete EDP from cluster

Add article to describe how user can delete EDP with all integrations and related resources (clean Keycloak, remove finalizers....).
Provide script to automate this process.
Acceptance Criteria:

mkdocs (https://epam.github.io/edp-install/) contains article of how to delete (redeploy) EDP;
provide script to remove finalizers in case if Namespace stuck in terminating state;
describe how remove finalizers in manually mode;

The "Uninstall EDP" page has been added. Please refer to the link for details.


Allow newer node versions

@markmcconachie Do you mind if I merge this back in?

I don't mind at all but it wasn't really heavily tested


IE11 crashes with syntax error in the intersection-observer package

Describe the bug
The main bundle of the @react-hook/intersection-observer package uses arrow functions that makes IE11 crash with syntax error.
Example dist/main/index.js:

const useIntersectionObserver = (target, options = {}) => {
  //                                      ERROR HERE --^^
  const {
    root = null,
    pollInterval = null,
    useMutationObserver = false,
    rootMargin = '0px 0px 0px 0px',
    threshold = 0,
    initialIsIntersecting = false
  } = options;

  // ...
}

To Reproduce
Steps to reproduce the behavior:

Load a page using this package in IE11, that is using the main package bundle.
A syntax error is thrown.

Expected behavior
The main bundle should be transpiled to a version of JS that is supported by IE11.
Desktop (please complete the following information):

OS: Windows
Browser: IE
Version: 11

Not a bug: it is 100% your responsibility to compile dependencies to your desired target. Call it controversial if you want, that’s the stance I’ve always taken and I’m not changing it.

This should be stated clearly in the README IMO.
I respect your decision to be shipping +ES6 only however I have no way of knowing it without starting to dig into (closed) issues and code.
We have hundreds of packages in our project and they all ship ES5.

I don't care about IE11. Compile your node_modules to your desired target yourself like CRA does.

Like, read my other comment. What makes you think you're going to change my mind? Use another library if you don't like it. I don't work for you. I work for me. Get over it.

Now included in the README: https://github.com/jaredLunde/react-hook/blob/master/README.md#note-on-ie11


Display Selected Date as PersianDate

When a date selected, I tried to set text property as persianDate using PersianCalendar.
In debug view , Text property has changed correctly. But control displays date as Gregorian date yet.
I tried to set DisplayText property but it is read-only property.
How can I correct it?

Could you please take a look at our sample project, it should work as you expect


Unable to reproduce. Awaiting more reports.


[ENHANCEMENT] Add EVA-CLIP support

Is your feature request related to a problem? Please describe.
New class of CLIP models https://github.com/baaivision/EVA/tree/master/EVA-CLIP
Describe the solution you'd like
Having these available either through custom loading or in the registry
Describe alternatives you've considered
It could be possible to use custom loaders but it is untested and not clear if they are compatible with open_clip
Additional context
source https://github.com/baaivision/EVA/tree/master/EVA-CLIP

it seems they have their own package eva_clip now.  The end points look exactly the same as open_clip though. I will check and find the solution.


Exception: Diamond search failed!

Dear Sam,
I'm using ggCaller v1.3.4, which I have installed using Conda (Docker is not available on the Linux cluster I'm using). I tested ggCaller with 10 Mtb genome assemblies I built using the command line:
ggcaller --refs test_input.txt --threads 4
And got the following error message:
Identifying high-scoring ORFs...
|         █          █          █          █    █| 100%
Loading gene models...
Generating initial network...
Processing paralogs...
100%|██████████| 23/23 [00:00<00:00, 3164.89it/s]
collapse mistranslations...
Processing depth:  1
Iteration:  1
100%|██████████| 7599/7599 [00:00<00:00, 10417.35it/s]
Iteration:  2
100%|██████████| 438/438 [00:00<00:00, 6477.96it/s]
Iteration:  3
100%|██████████| 15/15 [00:00<00:00, 32974.09it/s]
Processing depth:  2
Iteration:  1
100%|██████████| 7067/7067 [00:00<00:00, 9229.73it/s]
Iteration:  2
100%|██████████| 26/26 [00:00<00:00, 4977.04it/s]
Processing depth:  3
Iteration:  1
100%|██████████| 7041/7041 [00:00<00:00, 8397.15it/s]
Iteration:  2
100%|██████████| 3/3 [00:00<00:00, 6675.28it/s]
annotating gene families...
Traceback (most recent call last):
  File "/rds/homes/r/rderelle/miniconda3/envs/ggc_env/bin/ggcaller", line 33, in <module>
    sys.exit(load_entry_point('ggCaller==1.3.4', 'console_scripts', 'ggcaller')())
  File "/rds/homes/r/rderelle/miniconda3/envs/ggc_env/lib/python3.9/site-packages/ggCaller-1.3.4-py3.9-linux-x86_64.egg/ggCaller/__main__.py", line 511, in main
    run_panaroo(pool, array_shd_tup, high_scoring_ORFs, high_scoring_ORF_edges,
  File "/rds/homes/r/rderelle/miniconda3/envs/ggc_env/lib/python3.9/site-packages/ggCaller-1.3.4-py3.9-linux-x86_64.egg/panaroo_runner/__main__.py", line 114, in run_panaroo
    G = iterative_annotation_search(G, shd_arr_tup, overlap,
  File "/rds/homes/r/rderelle/miniconda3/envs/ggc_env/lib/python3.9/site-packages/ggCaller-1.3.4-py3.9-linux-x86_64.egg/panaroo_runner/annotate.py", line 192, in iterative_annotation_search
    G = run_diamond_search(G, shd_arr_tup, overlap, annotation_temp_dir,
  File "/rds/homes/r/rderelle/miniconda3/envs/ggc_env/lib/python3.9/site-packages/ggCaller-1.3.4-py3.9-linux-x86_64.egg/panaroo_runner/annotate.py", line 100, in run_diamond_search
    raise Exception("Diamond search failed!")
Exception: Diamond search failed!

I tried again with different genome assemblies but obtained the same error message. Unfortunately, I'm only interested in the gene presence/absence CSV matrix, not the gene annotation that is not performed by default according to the doc (I assume here that Diamond is used to annotate ORFs).
Any thoughts about what could have gone wrong? Happy to give more details if needed.
Thanks,
Romain

Looking at the code, file "panaroo_runner/annotate.py", I think the issue might be related to the Diamond database (which is either absent, corrupted or in a wrong format). Unfortunately, I can't figure out where it should be located.

Hi Sam,
Here are my installation commands:
conda env create -f environment_linux.yml
conda activate ggc_env
git clone --recursive https://github.com/samhorsfield96/ggCaller && cd ggCaller
python setup.py install

Thanks

A similar issue has come up before I believe, please see issues #16 and #20 and let me know if this solves your problem.