TINKERPOP-1027: Merge view prior to writing graphRDD to output format/rdd

https://issues.apache.org/jira/browse/TINKERPOP-1027
We had a bug in Spark graphRDD writing that showed itself on for particular providers. @dalaro provided realized the problem and provided a solution. This PR implements @dalaro's recommended fix. This fix also removes the need for reduceByKey() (though backwards compatible if you do still have it) and allowed us to always use GryoSerialization with Spark. This is rad. I added a few more required serialization registers to GryoSerialization and all the test cases pass. I also added some more test cases to ensure proper functioning.

Spark integration tests passed.
mvn clean install passed.

VOTE +1.

NOTE: I originally had this work in TINKERPOP-1025, but that ticket is a completely different beast so I merged the work into a new ticket. TINKERPOP-1025 is still being dealt with.

My comment on TINKERPOP-1025 now applies to this PR.   The change involving {{prepareFinalGraphRDD}} is what matters to me, and when I tested a TINKERPOP-1025 HEAD that includes both of these commits, it solved my problem.  So, +1 (non-voting) from me.

I made it so that SparkGremlin works like Spark JobServer (https://github.com/spark-jobserver/spark-jobserver/). It ensures that RDDs are not garbage collected by maintaining a static Spark class that holds a ConcurrentHashMap of RDDs. Thus, Spark is like a "file system" in that RDDs can be ls(), rm(), etc. This was necessary to get "slow" mvn clean install to build correctly where RDDs are NOT GC'd by Spark context cleaner.

Awesome, looks good and tests pass for me.
VOTE +1

Build finished. No test results found.

tests run locally for me:
VOTE +1
btw, your intellij settings might need to get fixed up - they are doing wildcards for static imports:
https://github.com/apache/incubator-tinkerpop/pull/172/files#diff-889af44a6f21b3700e537cc41765435aR40


Enabling CI

Trying to get this repo hooked up to an jenkins build and deploying:
http://angularpf-erundle.rhcloud.com/
Karma tests are currently not executing so I am working on that.

LGTM

LGTM


Restrict the project data directory configuration

Limits the configuration range for the project data directory.

Needs a rebase.


Values in tables are being splitted into multiple rows if there is \n in them

i have a table in docx file. some values in table have "\n" in them. and when i apply this script:
path = "test.docx"
from docx import Document

doc = Document(path)
table = doc.tables[0]
for row in table.rows:
    for cell in row.cells:
        print(cell.text.strip())
        print("-------------------")

i see that values that have "\n" in them are being splitted into several rows. How to solve this issue?
Example:
for this table

column name "Category/Technical \n policy" is being splitted into two rows "Category/Technical" and "policy".

You many need to replace \n by '\t" so everything stays on the same line.

cell.text contains a newline ("\n") between paragraphs in that cell. So it appears that "Category/Technical" is one paragraph and "policy" is another.
You can avoid this be making it all one paragraph. Otherwise you'll need a strategy to accommodate newlines in the text.
Closing as this is the expected behavior.


TypeError: Cannot read property 'on' of null at catchUpgradeRequest

Expected behavior
Able to proxy websockets with webpack 2.
Actual behavior
Not able to proxy websockets with webpack 2.
Setup
webpack-dev-server v2.2.1 was just pushed to stable. It depends on http-proxy-middleware ~0.17.1, which resolves to 0.17.3.
At the previous stable version, v1.16.2, this webpack config works:
  devServer: {
    https: true,
    proxy: {
      '/my-backend': {
        changeOrigin: true,
        pathRewrite: {'^/my-backend': ''},
        target: myBackendUrl,
        ws: true,
      },
  }

At v2.2.1, it does not. Instead I get:
TypeError: Cannot read property 'on' of null
    at catchUpgradeRequest (/work/my-project/node_modules/http-proxy-middleware/lib/index.js:56:19)
    at middleware (/work/my-project/node_modules/http-proxy-middleware/lib/index.js:48:13)
    at /work/my-project/node_modules/webpack-dev-server/lib/Server.js:223:15
    at Layer.handle [as handle_request] (/work/my-project/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/work/my-project/node_modules/express/lib/router/index.js:312:13)
    at /work/my-project/node_modules/express/lib/router/index.js:280:7
    at Function.process_params (/work/my-project/node_modules/express/lib/router/index.js:330:12)
    at next (/work/my-project/node_modules/express/lib/router/index.js:271:10)
    at middleware (/work/my-project/node_modules/http-proxy-middleware/lib/index.js:43:13)
    at /work/my-project/node_modules/webpack-dev-server/lib/Server.js:223:15
    at Layer.handle [as handle_request] (/work/my-project/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/work/my-project/node_modules/express/lib/router/index.js:312:13)
    at /work/my-project/node_modules/express/lib/router/index.js:280:7
    at Function.process_params (/work/my-project/node_modules/express/lib/router/index.js:330:12)
    at next (/work/my-project/node_modules/express/lib/router/index.js:271:10)
    at goNext (/work/my-project/node_modules/webpack-dev-middleware/middleware.js:28:49)

WDS 1.16.2 seems to be using HPM 0.17.1 (WDS package.json)
@tonygentilcore, can you verify if the issue disappears when you set the http-proxy-middleware explicitly to 0.17.1 in your WDS 2.2.1 setup?

Overview of code changes v0.17.1...v0.17.3

So, actually, both WDS major versions resolve to the same HPM version, 0.17.3 (because they depend on ~0.17.1 which allows the patch version to float).
Working:
└─┬<EMAIL_ADDRESS>...
  ├─┬<EMAIL_ADDRESS>  │ ├─┬<EMAIL_ADDRESS>  │ │ ├──<EMAIL_ADDRESS>  │ │ └──<EMAIL_ADDRESS>  │ └─┬<EMAIL_ADDRESS>  │   └──<EMAIL_ADDRESS>...

Not working:
└─┬<EMAIL_ADDRESS>...
  ├─┬<EMAIL_ADDRESS>  │ ├─┬<EMAIL_ADDRESS>  │ │ ├──<EMAIL_ADDRESS>  │ │ └──<EMAIL_ADDRESS>  │ └─┬<EMAIL_ADDRESS>  │   └──<EMAIL_ADDRESS>...

But just to be certain, I did try WDS 2.2.1 w/ HPM 0.17.1. Same stack.

Interesting. A large change in webpack-dev-server was that we now use HTTP/2. For that, it uses the spdy package (confusing name, but it also provides HTTP/2 support). Because http/2 only works with https, it's only activated when you use https: true. So it could be that the spdy package and http-proxy-middleware are not compatible with each other.

I reproduced it, and found out that the spdy package does not set req.connection.server, which http-proxy-middleware depends upon.

Worth investigating the spdy setup. Thanks for the background info @SpaceK33z .
With the ws: true option, HPM relies on a initial (GET) request to get the server object, so HPM can subscribe to the server's upgrade event.
In a really simplified setup, the server object is retrieved via:
middleware(req, res, next) {
   catchUpgradeRequest(req.connection.server);
}

@chimurai I filed a bug report in spdy: https://github.com/spdy-http2/node-spdy/issues/303. I'll take a look if there is another way to get the server object.

Great! Will also take a look in getting the server object.

Some sad news:

NO WEBSOCKETS OVER HTTP/2

https://daniel.haxx.se/blog/2016/06/15/no-websockets-over-http2/
https://github.com/spdy-http2/node-spdy/issues/224#issuecomment-139122247

Also some good news:
You can force spdy to use http/1.1; The server object will be available again in the request.
https://github.com/webpack/webpack-dev-server/lib/Server.js#L352
options.https.spdy = {
    // protocols: ["h2", "http/1.1"]
    protocols: ["http/1.1"]
};

This should hopefully fix the issue.

Nice, I can confirm that this config gets me going again!
  devServer: {
    https: {
      spdy: {
        protocols: ['http/1.1'],
      },
    },
    proxy: {
      '/my-backend': {
        changeOrigin: true,
        pathRewrite: {'^/my-backend': ''},
        target: myBackendUrl,
        ws: true,
      },
  }

@chimurai noo that's a shame :(. Thanks for finding that out though. I could workaround this with a huge hack by checking if ws: true is used and if so, revert to http/1.1 (insert evil laugh 😈 )

It's a shame indeed.
Glad to hear there is a workaround for this issue. (Thanks for sharing the WDS solution @tonygentilcore)
Perhaps I can catch the error and log an error so it'll be more developer friendly.
Until http2 gets official support for WebSockets there is little we can do about this.

@tonygentilcore @SpaceK33z @chimurai. By placing the spdy created server on the req.connection.server manually during express middleware execution this solved the problem for me. This is only possible if you are using node api and is a bit of a hack.
module.exports = function proxyCreator(app, server) {
  const wsProxy = proxyMiddleware('/socket.io', config);
  server.on('upgrade', wsProxy.upgrade);
  app.use((req, res, next) => {
    req.connection.server = server;
    wsProxy(req, res, next);
  });
};

Replacing the devServer option
{
  ...
  https: true
  ...
}

with
...
https: {
  spdy: {
     protocols: ['http/1.1'],
   }
}
...

worked for me.
Thank you!

This seems also be related to the order of proxy configs:
BREAKS with Cannot read property 'on' of null at catchUpgradeRequest lib\index.js:63:19:
{
  "/notification/channel": {
    "target": "wss://foobar.local/notification/channel",
    "secure": false,
    "ws": true
  },
  "/something.json": {
    "target": "https://foobar.local/something.json",
    "secure": false
  }
}

TypeError: Cannot read property 'on' of null
    at catchUpgradeRequest (node_modules\http-proxy-middleware\lib\index.js:63:19)
    at middleware node_modules\http-proxy-middleware\lib\index.js:55:13)
    at node_modules\webpack-dev-server\lib\Server.js:224:15`

WORKS:
{
  "/something.json": {
    "target": "https://foobar.local/something.json",
    "secure": false
  },
  "/notification/channel": {
    "target": "wss://foobar.local/notification/channel",
    "secure": false,
    "ws": true
  }
}

Hi @dherges,
Thanks for diving into this issue.
Can you apply your suggested change locally in you node_modules folder? And see if this indeed fixes the issue?

Hi @chimurai,
sure, we have a repro in the office at my employer, w/ wss:// server. Will check that tomorrow morning.
It also affects Angular CLI ng users, since they (angular team) are pulling in webpack which pulls in http-proxy-middleware 😸

Hi @chimurai,
if moving the call to catchUpgradeRequest(..) to inner if(shouldProxy(...)), it does not handle websocket upgrade at all. Changing the code was no solution to us. Changing the order of proxy configs is working for us.
I cannot exactly tell when it happens that req.connections.server is not initialized and has an undefined value. It seems to happen when we connect to SSL backends w/ self-signed certs (wss:// and https:// targets + "secure": false). It works fine, when connecting to ws:// and http:// targets.

Too bad the fix didn't work...
Are you using the latest version of webpack-dev-server? And can you share your devServer config?

Hi, we're using thru @angular/cli 1.0.2 default set up ... what they pull in. Would need to look at exact version next week in office.
Cheers, David

On 12. May 2017, at 17:33, chimurai<EMAIL_ADDRESS>wrote:
Too bad the fix didn't work...
Are you using the latest version of webpack-dev-server? And can you share your devServer config?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

I have this issue using bun but not node, I'm not using webpack dev server also, I'm using this library directly, latest version.


Fix ConsoleInterface binding things properly^2

Follow-up on #314 - I still misinterpreted..
Turns out the ".asInstanceOf[AnyRef].getClass.getName" implementation
was the original implementation. Then Mark switched to using bindValue
in sbt/sbt@4b8f0f3f941d5b3970fa24dfd9a34508d6974345.
Since Scala 2.11.0 (scala/scala#1648 in particular) bindValue was
removed. So we'll use NamedParam and quietBind, both which exist since
Scala 2.9.0.
Fixes sbt/sbt#2884, tested with local releases.

Why is this targetting 1.0.1?

Because @eed3si9n targeted the issue to 1.0.1: https://github.com/sbt/sbt/issues/2884#issuecomment-318778799

If it's not going to be part of 1.0.0, then I don't think we should merge it in there.
Should we create a branch for 1.0.x?

I thought about that, but I'd rather not get into the business of more persistent branches.
I think we should merge it into the 1.0.0 branch and here are my reasons:

I think the bug fix here is clean and simple, and doesn't impact anything outside of console.

If we do discover some release-blocking regression that forces our hand to do an RC4 release, this bug fix can go with it, imo. If we don't then this fix can ship as 1.0.1 or 1.1.0 at a later date.

Oh and at this point in time the only thing in the 1.x branch that's not in the 1.0.0 branch is this inconsequential test classpath change: https://github.com/sbt/zinc/pull/328.

Let's then aim for 1.0.0. I'm fine merging it and releasing RC4.

@dwijnand

I think we should merge it into the 1.0.0 branch and here are my reasons ...

So you are saying that RC-3 is still on its way to become 1.0.0 final, but you still want to merge this into 1.0.0 release branch? 1.0.0 is effectively a mutable tag that we are saying it will become 1.0.0 eventually.
If we plan to release 1.0.1 with bug fixes only, then we should create 1.0.x. Otherwise, we should merge this to 1.x and ship in 1.1.0 (in a few months?).
@jvican

Let's then aim for 1.0.0. I'm fine merging it and releasing RC4.

Releasing RC-4 means delaying the entire sbt 1.0 and Zinc 1.0 for another week or two.
I personally don't think this is worth it.

Releasing RC-4 means delaying the entire sbt 1.0 and Zinc 1.0 for another week or two.
I personally don't think this is worth it.

Hmmm... if releasing a trivial fix is going to delay sbt 1.0, then we have a real problem with our tooling. Personally, I don't like to make wrong decisions because of constraints on the tools we use -- if the current repo setup is causing too much pain, we should have a monorepo or something that emulates it.

Hmmm... if releasing a trivial fix is going to delay sbt 1.0, then we have a real problem with our tooling.

It's a matter of policy, not tooling. Even if we had a monorepo e.g. sbt/sbt, I would still push the final date by a week or two if we made any change to the code before calling it 1.0.0.

To validate this pull request against other sbt modules, please visit this link.

To validate this pull request against other sbt modules, please visit this link.


Timeout and Error counters do not increment (Always Success)

When using the --logging.debug flag, you see output like this:
Success | 3000/3000 Total Success | 0/3000 Timeout | 0/3000 Error
However, the timeouts and errors do not ever actually increment. Every miner has a perfect success rate, even though I have seen miners timeout while watching the logs.

Update: I have noticed that error DOES increment but timeout does NOT.


Fix: Cupertino Icons

Use case
When the using Cupertino Icon then that have two icons are overlapping each other on square_pencil and square_pencil_fill. This issue is continued yet.
Proposal
square_pencil and square_pencil_fill review at once.

Closing this as a duplicate of https://github.com/flutter/flutter/issues/82634


chore: minor changelog improvements

What/Why/How?
Minor improvements to the changelog.
Reference
Noticed on: https://redocly.com/docs/cli/changelog/
Testing
n/a
Screenshots (optional)
n/a
Check yourself

[ ] Code is linted
[ ] Tested with redoc/reference-docs/workflows
[ ] All new/updated code is covered with tests

Security

[x] Security impact of change has been considered
[x] Code follows company security practices and guidelines

Thank you!


DYNO_INTERFACE - how to specify copy constructor?

As in title, is it possible to do it now? Can dyno::CopyConstructible{} be used for that?
So far I wasn't able to write a compilable form of it.

Good point; the DYNO_INTERFACE macro does not allow specifying special member functions at the moment. I'll need to think of a solution.


🚀  Post Detail View

Adding a detail view for post model.
Each post will get a unique url that will only show the post itself and let the user to interact with it.
Each post detail view url will be /post/post_id/ where post_id  should be replaced with the desire post id.

Added a post detail view html template
Set up detail view for posts via views.py file, used Django's DetailView class
Set up the post detail view URL
Added tests

The post detail view screenshot:

Close #115

Sorry for force pushing

LGTM


ApplicationSet with git directories generator not recognizing exclude

I am using ArgoCD v1.7.6+b04c25e and I am just trying the POC for ApplicationSet.
Installed ApplicationSet by running kubectl
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/applicationset/v0.1.0/manifests/install.yaml

Here is my ApplicationSet file -
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: dev-application-set
  namespace: argocd
spec:
  generators:
  - git:
      repoURL: <GIT REPO URL>
      revision: HEAD
      directories:
      - path: apps/overlays/dev/*
      - path: apps/overlays/dev/common
        exclude: true
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: default
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
      source:
        repoURL: <GIT REPO URL>
        targetRevision: HEAD
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: dev

When I check in ^^ file or just type kubectl apply -f dev-app-discovery.yaml -n argocd I see that it does not recognize exclude path.
I want apps/overlays/dev/common to be excluded and I don't want an application to be created for common folder.
Getting the following error -
$ kubectl apply -f dev-app-discovery.yml -n argocd
error: error validating "dev-app-discovery.yml": error validating data: ValidationError(ApplicationSet.spec.generators[0].git.directories[1]): unknown field "exclude" in io.argoproj.v1alpha1.ApplicationSet.spec.generators.git.directories; if you choose to ignore these errors, turn validation off with --validate=false

Thanks,
Pratik

Hi @pratikzingade, the exclude feature was added to the ApplicationSet controller after 0.1.0 is released. Until the next release is available, you would need to use a development build if you wish to use exclude. See https://github.com/argoproj-labs/applicationset/blob/master/docs/Geting-Started.md#development-builds for details.

Thanks @jgwest Can you share the timeline for next release, I was able to install the dev build and get exclude working, thanks for pointing to the docs.

@pratikzingade We aim to release alongside Argo CD major releases (v2.x.0 releases, next would be v2.1)... you can track their latest milestone here: https://github.com/argoproj/argo-cd/milestone/22 (admittedly these are a bit longer than many would like, so I'm working on improving our ability to release more frequently with better test automation...)


docs(Drawer): Fix Drawer examples

Issue: none
What I did

Update Drawer docs examples

missing imports
example on how to update the aria label for the close button
styling for Drawer props so it appears on the site navigation
i18n example wasn't valuable

Checklist

[X] changeset has been added
[X] Pull request description is descriptive
[X] I have made corresponding changes to the documentation
[X] New and existing unit tests pass locally with my changes
[-] I have added tests that prove my fix is effective or that my feature works

How to test
Test the drawer docs site, clicking on all the code sandbox links and ensuring it makes sense.

I verified the changes listed above are complete and looking good.


Cranelift: Only build iconst for ints <= 64 bits

I audited the egraph "algebraic" optimization rules for any which construct an iconst on the right-hand side of the rule. In these cases we need to constrain the type passed to iconst to be both fits_in_64 and ty_int, because iconst is not defined on other types.
If there's some reason why icmp is already constrained to <=64 bits, I'd prefer to add comments explaining why instead of adding constraints to the rules.

I think this should be back-ported for the upcoming release, once we've validated this change, even though we haven't received any bug reports about these rules. I don't think it needs to be backported further since this will be the first release with e-graph optimization on by default.

If we're generalizing iconst, I'd be tempted to try for a generic const that works with floats and vectors too.

Perhaps, yeah. I just went to remind myself how fconst works and it turns out that we actually split to fconst32 and fconst64 opcodes, so the situation is even less polymorphic than I had thought. Cleanup would be nice!


Support for DB Restore from Object Stores

Support restore from an object store.  This is one of several PRs in support of the RFE for
https://bugzilla.redhat.com/show_bug.cgi?id=1513520.  Another PR against the manageiq-appliance_console is also forthcoming.
@roliveri @carbonin please review and merge when appropriate.
Links [Optional]

https://bugzilla.redhat.com/show_bug.cgi?id=1513520
https://github.com/ManageIQ/manageiq-gems-pending/pull/357 (merged)
https://github.com/ManageIQ/manageiq-gems-pending/pull/355 (merged)
https://github.com/ManageIQ/manageiq-schema/pull/227 (merged)
https://github.com/ManageIQ/manageiq/pull/17689 (merged)
https://github.com/ManageIQ/manageiq-ui-classic/pull/4300 (being reviewed)

@carbonin fixed.  good catch - thanks.


KYLIN-3531 Save uppercase of usernames to metadata

https://issues.apache.org/jira/browse/KYLIN-3531

Can one of the admins verify this patch?

Can one of the admins verify this patch?

If login failed, try this command before kylin starts:

$KYLIN_HOME/bin/metastore.sh remove /user

Restart Kylin. Now the usernames in metadata are all uppercase and kylin will ignore case in Web UI.

Facing issue while creating streaming table from Kylin
Overwriting conflict /project/KYLIN_SYSTEM.json, expect old TS<PHONE_NUMBER>000, but it is<PHONE_NUMBER>000


Negative numbers in theory atoms

What is the best way to deal with negative numbers in theory atoms, given that - is problematic due to being reserved for strong negation ?

You just declare a unary minus operator and can then use it in the theory atom. For clingo, the terms in the theory atoms are just syntax. If you want to treat them as a number or negated atom is up to you. Systems like telingo or clingcon give meaning to the terms. You can have a look how this is done in clingcon: https://github.com/MaxOstrowski/clingo/blob/master/examples/clingo/csp/csp/parsing.py

Thank you, @rkaminsk !


Master to ExMigration

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

exchange/exchange-ps/exchange/mailboxes/Import-ContactList.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/exchange-ps/exchange/mailboxes/New-InboxRule.md
:white_check_mark:Succeeded
View (exchange-ps)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Fix typos in markdown files

Thank you very much! As a not a native speaker I appreciate that.

Absolutely :smile: have a good day/night!


[data_processing] [sirene] feat: prevent new rate limiting from api insee

wait 0.5 sec between each call insee

FYI @HAEKADI


[7.x] Make valid method names for abilities with spaces

If an ability has a space in its name, such as "edit post", the current Gate implementation will check if there's a method named "edit post" in the model's policy. That method obviously doesn't exist because it's invalid PHP. I changed the formatAbilityToMethod() method to check for 'editPost' in these situations instead. I think that's intuitive behavior - more intuitive than renaming your ability to "edit-post" as you are now supposed to do.

Checks seem to be failing because it cannot install the right dependencies - not sure why that is, didn't touch any of that.

I don't think we should support gate abilities with spaces in their names.

In my particular case I have a few permissions that are quite long, such as see-opening-times-calendar, which I think is hideous with dashes but acceptable with spaces.
Spatie's popular laravel-permission package explicitly uses them, and I think for the better - $user->can('edit articles'); is (imo) good, clear, and concise code. Not sure why you would want to force people to use dashes there just so that the method name can be automatically resolved.

I personally don't use spaces in ability names but i find it nice.
I once made this PR https://github.com/laravel/framework/pull/15339 which allows to use dots but he turned it down.
He doesn't seem to like any invalid php stuff within ability names

I think he doesn't use this type of ability names in his own projects and therefore thinks it isn't necessary. Dashes are supported though, so you can have abilities named edit-articles (but not edit articles). It is gut feeling kind-of call that makes my permissions code look ugly. If you name your ability edit-articles it will check the editArticles() method on the appropriate policy to see if that action is allowed, but that doesn't work if you name the ability edit articles. Stupid.
The laravel permission package (8.2k stars) explicitly uses long ability names in their permissions, such as "edit articles", so I'm guessing I'm not the only one who would have benefited from this change.
I'm wondering if the maintainers of that package have an opinion on this.

@taylorotwell Yeah there is no harm in using spaces or dots. I hope you think about this and give anohter shot taylor.
It let us write our permissions the way we like.


fix ModelViewInheritance module spelling

this will result in braking changes and you need to reinstall the module if you use it as standalone

Closing issue for age. Feel free to reopen it at any time.
.Thank you for your contribution.

Closing issue for age. Feel free to reopen it at any time.
.Thank you for your contribution.


Remove the download plugin from the build

The github download plugin is discontinued. A new solution is required.

We could have the CI deploying both js file as part of the release in the github release.

Or we could have a 'builds' folder in which we copy hubu.js and hubu.min.js from target during the install phase ?


Remove debug msal common

This PR removes debug library from msal-common dependencies

Codecov Report

Merging #4665 (190a4dc) into dev (616e609) will not change coverage.
The diff coverage is n/a.

Flag
Coverage Δ

*Carryforward flag

msal-angular
96.39% <ø> (ø)

Carriedforward from 616e609

msal-browser
87.07% <ø> (ø)

Carriedforward from 616e609

msal-common
85.04% <ø> (ø)

Carriedforward from 616e609

msal-core
82.67% <ø> (ø)

Carriedforward from 616e609

msal-node
82.32% <ø> (ø)

msal-node-extensions
76.03% <ø> (ø)

Carriedforward from 616e609

msal-react
92.46% <ø> (ø)

Carriedforward from 616e609

node-token-validation
83.41% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.


debugging issue with git

commited project latest update, review it to double check

issue in one file where i duplicated code. please check and fix ASAP.


Dependency and Ruby 3.3.1 uplift across CI, release pipelines & image

ruby:3.3.1 released last week which was previously a blocker to us for anonymous splat forwarding in various defs.
Updated both CI/release pipelines to use 3.3.1, image uplifted and various dependencies via bundler bumped

After doing a ruby-local 🐰 🕳 - matched bundler to what is available in ruby:3.3.1 containers, and also aligned the local .ruby-version after updating on my side/conforming CI works 👍


Alien Fan Badness

X17 R2 12900HK, 3080, AFX v8.4
Laptop had been running a virus check with Alien Fans set to Manual using my curves, BIOS Fan setting to performance, Power to Bitsum Higest Performance, ThrottleStop max CPU perf settings, using a 230W power supply from an old Alienware Laptop.
Changed power config to battery, TS to ultra power savings. Unplugged laptop and moved it to my office and plugged in the 300W power supply. Walked away. When I returned some time later, Alien Fan Control had somehow dropped to ultra-low fan speeds (something like 600, 400, 1900, 500). Entire laptop was HOT soaked (most temps 75C, and it felt like most of the laptop - especially the metal parts of the base, were immediately painful to the touch), many heat-related alarms/warnings were on in TS.
I don't know if there is anything still available for diagnosis/troubleshooting. But let me know if I can send anything to assist.
P.S. I hope you have been well in my absence. My health is finally improving enough for me to start being active again. I hope it stays that way!!

Well... I'm badly occupied into my main duty this time, but still continue.
Glad to hear you back!
Looks really strange - i can not block the fans from spinning, so it should run up even for the flat curve at 75C. The only thing i can imagine is overboost, then so high boost prevents fan from running (i meet this for some hardware).
So i need reading... Scale, boost level, etc then it happened.

I have other guy to suffer the same issue, but he have G15.
We examine this, and.... this is BIOS issue - system stop react to boost/temp changes completely!
We are found to override - just run alienfan-cli unlock and it will return back to normal.
BTW, can you run alienfan-cli rpm and send me you SystemID (it's into brackets at the first line)?

I am sorry for the delay in responding.
Is there any further documentation on what, exactly, alienfan-cli unlock does? Is the only time I need to run that when the fans stop responding - or are there other instances where it would be beneficial?
D:\Program Files\Utilities\AlienFXTools>alienfan-cli rpm
AlienFan-CLI v8.5.0
Supported hardware (4800) detected, 4 fans, 2 sensors, 7 power states.
Fan CPU 0 RPM: 2000
Fan GPU 1 RPM: 2000
Fan CPU 2 RPM: 1900
Fan GPU 3 RPM: 1900

Aha, 4800. Seems like the same BIOS module.
unlock command select manual power mode. Equal to setpower=0.

I am running a quite old BIOS for the X17 R2 because Dell locked up most of the CPU tweaking after the version I have. WTF?? This is a GAMING COMPUTER, from a brand that (used to) PRIDE itself on top-tier performance. Sigh...

Offtopic a bit, i have my own teeths sharped about some things in Dell.
 And something broken into this BIOS.


New feature: Fine-Tuning BERT for NER Notebook

Description
We have added a new notebook with a tutorial on how to visualize a dataset with Argilla and use it to fine-tune a BERT for NER. This tutorial is prepared for beginners who wish to have a first approach in a simple way.
Type of change

New feature (non-breaking change which adds functionality)
Documentation update

How Has This Been Tested
This Notebook was tested using Google Colab.
Checklist

[x] I added relevant documentation
[x] follows the style guidelines of this project
[x] I did a self-review of my code
[ ] I made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I filled out the contributor form (see text above)
[ ] I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)

Hi @sdiazlor, I've returned from holidays. Have you been able to make any progress on this based on my comments?

Hi @davidberenstein1957, I have updated the notebook to remove the O-tokens when visualizing in Argilla. Please, let me know your thoughts.

Hi @sdiazlor, Thank you! I will have a look later today.

Hi @sdiazlor, the tutorial looks great already, however, there is another small caveat. Normally, we span BIOU annotations over multi-word annotation. For example, "2nd day of 2015" is now visualized as individual annotations? B-TIME (begin) I-TIME (inside) and O-TIME (outside), we want to annotate it as a single Span (set of words) with the label TIME.

Hi @davidberenstein1957, okay, I'll check it, so it can be improved.

Hi @davidberenstein1957, it should work now with the changes I made.

Awesome @sdiazlor, this looks great. Looking forward to getting started in the coming weeks


Add "verify" flag to the creation of client

Is your feature request related to a problem? Please describe.
In one of our prod environments, we have an SSL inspector, which means that sometimes we can't validate the certificate.
Right now, supabase-py just crashes when run in this environment.
Describe the solution you'd like
In gotrue's SyncClient, there is a "verify" parameter for init. I would like there to be a suitable flag in "create_client" function that will be passed on to the gotrue.
Describe alternatives you've considered
Right now, we're just overriding all the SyncClients in all the underlying clients that we use (auth, storage, postgres) with "verify=False", when needed.
Additional context
No additional context

Hey team,
Can I check if this is still an issue?

I think this is still an issue since the verify property mentioned is from the httpx library and not the gotrue-py library itself.


Create allocators for dynamic objects such as std::vector or std::unordered_map etc.

Also, I can use std::vector second template parameter to override allocator.
I can use that to provide my own std::vector wrapper with my allocator


Speed up Conda Resolving

To my knowledge, conda package resolving should speed up significantly, when we specify the way packages are pulled by a corresponding prefix, e.g.:
- conda-forge::jdk=1.X.Y
is much faster than
- jdk=1.X.Y
Maybe we should make this something that is automatically added to the environment.yml when a new repository is created to make users more aware of this?

I'd be curious to see how much of a difference this actually makes first.. 🤔

Yes, you're saying much faster, but how much faster?

I took making the Singularity container for the chipseq pipeline as an example:
Explicit paths:
real	22m35.586s
user	29m5.709s
sys	1m59.633s

Non explicit paths:
real	25m45.530s
user	31m45.949s
sys	2m36.789s

So its measurable, but not necessary in my opinion. Any thoughts on this?

Cool, thanks for checking! I think no need for now.


SQLite Grammar ignored with column type change

Laravel Version: v8.77.1
PHP Version: 8.1.1
Database Driver & Version:

Description:
When changing a column type, SQLite Grammar is ignored leading sometimes to unknown column type. There errors does not happen when creating the table.
Steps To Reproduce:
Create a migration with a simple column typed tinyText.
Create a migration to change it to another column that works anyway like "text".
Create a third migration to change it back to tinyText.
The migration will run perfectly on MySQL but not in SQLite (test database for example).
See my example-app just created https://github.com/Korko/example-app
With the feature ExampleTest sail artisan test

Unknown column type "tinytext" requested. Any Doctrine type that you use has to be registered with \Doctrine\DBAL\Types\Type::addType(). You can get a list of all the known types with \Doctrine\DBAL\Types\Type::getTypesMap(). If this error occurs during database introspection then you might have forgotten to register all database types for a Doctrine Type. Use AbstractPlatform#registerDoctrineTypeMapping() or have your custom types implement Type#getMappedDatabaseTypes(). If the type name is empty you might have a problem with the cache or forgot some mapping information.

Tinytext doesn't exists in SQLite.

That's exactly the whole point. You CAN make a Blueprint::tinyText column in SQLite because the SQLiteGrammar converts it into "text" but you CANNOT change a column into Blueprint::tinyText as the Grammar is not applied.

You really should consider reopening this ticket as it's a real issue and the reason you gave is the exact reason this ticket is.


Redis

Really appreciate you doing this. I'll try to be more diligent about updating the strings 😆

Hey @xTCry,
Thanks for your effort but it's poor quality translation.
Don't you mind if I'll rework it?

Don't you mind if I'll rework it?

Yes, you can rework
I will be glad to support your work


prototype(button): create prototype button based on MDC Web

Prototype of using MDC Web for the base of our button component. Remaining issues and todo tasks are left in the README

related to #16196


cas: save ACIIinfo and AppIndex in WriteACI and introduce latest pattern handling for image fetching.

Last commit is the interesting one.
The previous required commits were proposed in other PRs ( #342 #347 #350 #351 #352 #353)
In WriteACI save an ACIInfo and an AppIndex for the imported ACI.
On fetching, if downloaded with the "latest" pattern (no version label
specified), the ACI is saved in the store with the "latest" flag.
See appc/spec#73 for my thoughts about the need of the "latest" flag.

I'm going to replace this with #392, #393, and #394 where imagemanifest is saved in a diskv store and aciinfo is now a db table (#71). I'll close this as I think the the new ones are much better.


Releasing generated data for MBPP

Hi All,
Great work!
I am wondering if you can also release the generated data for MBPP? Similar to the ones you released for HumanEval.
https://github.com/microsoft/CodeT/tree/main/CodeT/data/generated_data
Thank you!

Hi! Thanks for your interests in our work. We have released the generated data for MBPP, APPS and CodeContests. Please check out them in the repo!


chore(buildpacks): update heroku-buildpack-scala to v67

See https://github.com/heroku/heroku-buildpack-scala/compare/v66...v67 and deis/deis#4973.

I tested this with the workflow-beta1 chart and example-scala.


fix: change vercel env variable

fix: change vercel env variable

:tada: This PR is included in version 2.1.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Nextjs - Window is not defined

im trying to use this packages in nextjs but i got error saying window is not defined

@amirizzudin4 have you tried this? https://stackoverflow.com/questions/56319997/nextjs-window-is-not-defined It seems a similar issue

This should be fixed in v3.0.2

I manage to import react particle js in client side rendering following this example
import dynamic from 'next/dynamic';

const Typewriter = dynamic(
    () => import('typewriter-effect'),
    {
        ssr: false
    }
)

I think it's not fixed yet. Css breaks when loaded over ssr. Thanks


Update doc for now allowed constant species in condition table (#65)

Codecov Report

Merging #75 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop      #75   +/-   ##
========================================
  Coverage    59.93%   59.93%
========================================
  Files            5        5
  Lines          609      609
  Branches       129      129
========================================
  Hits           365      365
  Misses         220      220
  Partials        24       24

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c7296f...c7fd647. Read the comment docs.


Make the begin/end tag dependent on the extension of the source file

In other words, allow parsing comments in other languages.

We should also add the proper class (for highlighting) with different languages.


Update README.md

Describe how to change projects.

Apologies, should have made the pull request from my fork.

Covered in #48


Separate amd64 and 386 builds

Fixes #2

Both artifacts showing with the correct names. Merging

:+1:


more than 10 entries possible ?

Are you sure you read the readme?
Yep
Android version and skin (MIUI etc... is MIUI still a thing? idk):
Aosp14
Screenshot of Play Store page of the app you are trying to detach:
Screenshot of the helper CLI showing detached apps (from Option 2):
Logs taken with su -c logcat -d -s "zygisk-detach" > /sdcard/log.txt:
Not providing this information will get your issue closed

yes?

yes?

Whatever the question mark means....you could consider it in a update maybe...

the limit is not 10 apps. what make you think its not possible to detach more than 10 apps?

the limit is not 10 apps. what make you think its not possible to detach more than 10 apps?

Because if I detached the 11th app all apps previously detached reappear in play store.... that's why I ask if there's a limit....

the limit is not 10. there is a limit but before you reach it, the cli stops you so you dont exceed it.
if you are still having problems open a proper issue with logs or it will be closed again.


Issue with Huawei phones

For android my implementation works good on all devices I tested. Except for Huawei, which I don't have myself, I'm getting the error java.lang.IllegalArgumentException: the bind value at index 1 is null in my crash reporter for updateContact. I tried exactly the same inputs as this user tried on another non-Huawei phone and it worked fine.
Does anyone know about any differences between the Huawei contacts interface and the standard Android interface?
This might be related:
https://stackoverflow.com/questions/10014832/getting-account-type-of-contacts-table-for-accessing-phone-contact

Why was this closed? Did you resolve this? @luucv


(PUP-11373) Always lock the environment before its type collection

Puppetserver in MT mode can deadlock if one thread calls into the loaders,
which locks the environment, and then tries to lock the type_collection:
- waiting to lock <3f27e688> (a org.jruby.gen.RubyObject0) owned by "qtp346358872-131" t@131
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/type_collection.rb:188
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/type_collection.rb:152
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/type_collection.rb:152
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/runtime3_type_loader.rb:79
...
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/concurrent/lock.rb:10
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/loader.rb:153

While another thread locks the type_collection and tries to lock the environment:
- waiting to lock <bd0f663> (a org.jruby.gen.RubyObject0) owned by "qtp346358872-67" t@76
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/node/environment.rb:248
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/type_loader.rb:131
...
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/type_collection.rb:203
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/concurrent/lock.rb:10
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/type_collection.rb:188

This commit orders locking so we always acquire the environment lock before its
type collection lock.

This targets main because the loaders only started locking the environment in https://github.com/puppetlabs/puppet/commit/ec319d778150eade0b6128c2ddd11cbf3926183b first released in 7.6.0

jenkins please test this with servertests

@mwaggett thanks for looking! The OP that filed the ticket reported that it was working after 5+ hours It's possible to add a test, but I think it'd take me a bit to figure that out. There's at least one other remaining MT issue (the "attempt to redefine entity" on puppetserver) so I'm inclined to merge this and "move the needle forward". Thoughts?

seconding Justin - fine with me to get this in if we've got a user successfully running with this, but any updates to the testing would be appreciated (though also fine to do that in a separate PR, if at all).

Thanks @justinstoller and @mwaggett! I added a ticket to update the test as SERVER-3155. I'll take a look at that a bit later.


Think of some solutions for the Extension Functions being Null if the extension isn't activated

If an extension isn't active, the function pointers won't be populated. And any calls to those function would crash.
Possible solutions:
1: Populate the pointers with dummy empty functions. Might be able to make some templated one and use it everywhere.
2: Throw an exception in the object of a class if its required extensions aren't activated.
Should implement both.

After thinking a lot about it, it doesn't seem worthwhile to me. I will have to make a new dummy pointer for every function. And having if else check would introduce useless checks.


Should probably throw an error if the html file referenced in the usemin block cannot be found

I made a typo like so:
        usemin: {
            html: '%<= btp.dist %>/index.html'
        },

And the output to useminPrepare was normal, but I couldn't figure out why my JS and CSS includes weren't getting replaced. Here is the (verbose) output I saw:
Running "usemin" task

Running "usemin:html" (usemin) task
Verifying property usemin.html exists in config...OK
Files: [no src] -> html
Options: type="html"
Replaced 0 references to assets

I found the bug eventually, but had usemin thrown an error saying it couldn't find/open the file in question (rather than silently saying there was nothing replaced), I'd have been better off -- and assume others would as well.
Anyway, thanks for a great plugin!

@prdonahue close this one as it is a duplicate of #452.


Usage Help: Updating An Existing Archive

tar:Archive version 0.4.20 states:

This archive can have an entry added to it

I am trying to append a file to an existing tar. Is this possible? If so, is there an example or test case demonstrating this functionality?

Thanks for the report, I think this is a bug in the documentation! Nowadays Builder is used to construct/append archives, but Archive is a read-only variant.

Ah ok gotcha. Is there a working example or test case that shows a file being appended being appended to an existing tar file using Builder? I can only see a way of creating new tar files using Builder::new.

I don't think there's an example currently, but that'd be a great one to add! You'll need to recreate the whole archive by iterating with Archive, adding to Builder, and then adding a final entry for now

You'll need to recreate the whole archive by iterating with Archive, adding to Builder, and then adding a final entry for now

@alexcrichton Is this still the preferred method for adding files to an archive? Adding in a file into an archive that is gigabytes large by dumping the original archive's contents into ram and writing out the content, with most of the contents identical to the original archive, seems a bit wasteful to me. Then again, I do not know the structure of a tar archive to know if this is even possible without needing to overwrite the entire archive.
I would also imagine that modifying a file that previously exists - or even deleting a file from an existing archive - would be a difficult task to accomplish without dumping the entire contents of the original archive into ram via the Builder, using a filter to remove unwanted files/modified files. Or maybe there is a less memory intensive method to perform actions like this?

A tar archive is "just" files concatenated together so if your archive is as big as you say it is then you should be able to seek to the end and use Builder to append more files.

That is very interesting. Then going off that knowledge, to delete a file from the archive, one would need to: (1) find out where the file starts and ends in the archive and then (2) trim the file accordingly to no longer include that file. And conversely, to modify an existing file in the tar archive, you would effectively perform the same action as delete, but: (1) shrink/expand the space needed at the region in the tar of the old file to the size that fits the change and (2) write in the change.
Is there an existing mechanism in tar-rs to perform actions like this? Or is the idea such that a user should do this operation themselves? Such as using Archive to move the file cursor to the file in question, then use Builder::get_mut to get the file handle when the file is found to be used with Builder to write the changes and the file handle itself to expand/shrink the file itself?

Yes that's all theoretically possible, but convenience APIs for doing that aren't exposed from this crate at this time. Not that they can't be, just that they weren't when I implemented everything years ago.

Hi :)
I didn't succeed adding files or directories to an existing archive. I have the same issue that in this thread but there's no answer.
To sum up, only files from the first archive can be found after opening xxx.tar file. Added files seems to be in the archive as it grown in size but can't be found.
I do not manage headers manually like an SO Thread. I only open the file in append mode and use it in the Builder to add files. What could I do wrong ?
Thanks ;)

I hope this helps out anyone looking into modify archives. I wrote this a while back and may not be the prettiest code, but it worked for me.
Below are a series of functions I used in conjunction with this library to perform more high-level access to tar archives. The public functions such as write_to_tar, move_in_tar, delete_in_tar, etc. are the commands commonly used for typical tar access.

write_to_tar will append to an existing archive if the file does not exist. Should the file already exist in the archive, the existing archive is expanded or shrunk to accommodate the new file size and spliced into the archive.
move_in_tar is used to rename/move a file in the archive.
delete_in_tar is used to delete the file and shrinking the archive.
get_in_tar is used to return a tuple of the header and file contents as a Vec<u8> if found, else Ok(None) is returned.

use std::fs::File;
use std::io::{Cursor, Error, ErrorKind, Read, Result, Seek, SeekFrom, Write};
use std::path::Path;
use std::time::{SystemTime, UNIX_EPOCH};

use more_asserts;
use take_mut::take;
use tar::{Archive, Builder, Entry, Header};

const PAGE_SIZE: u64 = 1024;

fn aligned_end(end: u64) -> u64 {
    let new_end = end & !0x1ff;
    if end & 0x1ff == 0 { new_end } else { new_end + 0x200 }
}

fn find_end(start: u64, size: u64) -> u64 {
    aligned_end(start + 512 + size)
}

fn entry_positions<R: Read>(entry: &Entry<R>) -> (u64, u64) {
    let start = entry.raw_header_position();
    let sz = entry.size();
    let end = find_end(start, sz);
    (start, end)
}

fn expand_file(cursor: &mut File, start: u64, end: u64) -> Result<()> {
    if start == end {
        return Ok(());
    }
    // No cropping needed
    more_asserts::debug_assert_gt!(end, start);
    let mut buff = [0u8; PAGE_SIZE as usize];
    let diff = end - start;
    let file_end = cursor.seek(SeekFrom::End(0))?;
    for b in (end..(file_end + diff)).step_by(PAGE_SIZE as usize).collect::<Vec<_>>().iter().rev() {
        let a = b - diff;
        cursor.seek(SeekFrom::Start(a))?;
        let n = cursor.read(&mut buff)?;
        cursor.seek(SeekFrom::Start(*b))?;
        cursor.write_all(&buff[..n])?;
    }
    Ok(())
}

fn collapse_file(cursor: &mut File, mut start: u64, mut end: u64) -> Result<()> {
    if start == end {
        return Ok(());
    }
    // No adjusting needed
    more_asserts::debug_assert_gt!(end, start);
    let current_end = cursor.seek(SeekFrom::End(0))?;
    let future_end = current_end - end + start;
    debug!("Currently at: {}, need to reduce to: {}", current_end, future_end);
    cursor.seek(SeekFrom::Start(end))?;
    let mut buff = [0u8; PAGE_SIZE as usize];
    loop {
        let n = cursor.read(&mut buff)?;
        if n == 0 { break; }
        end += n as u64;
        cursor.seek(SeekFrom::Start(start))?;
        cursor.write_all(&buff[..n])?;
        start += n as u64;
        cursor.seek(SeekFrom::Start(end))?;
    }
    debug_assert_eq!(start, future_end);
    cursor.set_len(start)
}

/// This is largely for sanity checking to see that the archive is not corrupted
fn verify_archive(cursor: &mut File) -> Result<()> {
    cursor.seek(SeekFrom::Start(0))?;
    let mut a = Archive::new(cursor);
    for entry in a.entries()? {
        entry?;
    }
    Ok(())
}

pub fn write_to_tar<P: AsRef<Path>>(
    cursor: &mut File,
    path: P,
    data: &[u8],
    mut header: Header,
) -> Result<()> {
    let path = path.as_ref();
    debug!("Attempting to write {} bytes to: {:?}", data.len(), path);
    // Lets see if the file already exists in the archive
    cursor.seek(SeekFrom::Start(0))?;
    let mut archive = Archive::new(cursor);
    let mut found = None;
    for entry in archive.entries()? {
        let entry = entry?;
        let p = entry.path()?;
        if p == path {
            let start = entry.raw_header_position();
            let sz = entry.size();
            found = Some((start, sz));
            break;
        }
    }

    // Lets pull out the file from the archive object
    let mut cursor = archive.into_inner();
    let (position, overwriting) = if let Some((start, sz)) = found {
        let current_end = find_end(start, sz); // This is where `the tar archive ends to this file
        let new_end = find_end(start, data.as_ref().len() as u64); // This is where we need to go
        if new_end > current_end {
            debug!("Expanding! >>>");
            expand_file(&mut cursor, current_end, new_end)?;
        } else if new_end < current_end {
            debug!("Collapsing! <<<");
            collapse_file(&mut cursor, current_end, new_end)?;
        } else {
            debug!("Fitting! ===");
        }
        let pos = cursor.seek(SeekFrom::Start(start))?;
        (pos, true)
    } else {
        debug!("File did not exist before. Setting to (near) the end of the archive for writing.");
        let pos = cursor.seek(SeekFrom::End(-1024))?;
        (pos, false)
    };

    debug!("The file {:?} will be written at {:x}", path, position);

    // The file was not found in the archive, so lets add this file to the end of the archive
    let mut builder = Builder::new(cursor);
    header.set_size(data.len() as u64);
    let data = Cursor::new(data);
    builder.append_data(&mut header, path, data)?;

    if overwriting {
        // The builder will always write 1024 0s at the wherever the file cursor is when the builder is dropped. So, to stop
        // the builder from incorrectly terminating the archive in the middle of the archive, we will move the cursor to the
        // actual end so the archive so it will re-terminate the end of the archive.
        let f = builder.get_mut();
        f.seek(SeekFrom::End(-1024))?;
    }
    Ok(())
}

/// This function will delete the requested file (`path`) within the tar archive (`cursor`).
/// Please note that this function will only work if you open a file with read and write permissions
pub fn delete_in_tar<P: PartialEq<Path>>(cursor: &mut File, path: P) -> Result<bool> {
    cursor.seek(SeekFrom::Start(0))?;
    let mut archive = Archive::new(cursor);
    for entry in archive.entries_with_seek()? {
        let entry = entry?;
        let p = entry.path()?;
        if path == *p {
            let (start, end) = entry_positions(&entry);
            let sz = entry.size();
            let mut cursor = archive.into_inner();
            debug!("Deleting a file from {:x} to {:x} (sz={})", start.clone(), end.clone(), sz);
            collapse_file(&mut cursor, start, end)?;
            return Ok(true);
        }
    }
    Ok(false)
}

pub fn read_from_entry<R: Read>(entry: &mut Entry<R>) -> Result<Vec<u8>> {
    let mut buff = Vec::with_capacity(entry.size() as usize);
    let _n = entry.read_to_end(&mut buff)?;
    Ok(buff)
}

fn rewind<R: Read + Seek>(archive: &mut Archive<R>) {
    take(archive, |archive| {
        let mut cursor = archive.into_inner();
        cursor.seek(SeekFrom::Start(0)).unwrap();
        Archive::new(cursor)
    });
}

pub fn get_in_tar<'a, R, P>(archive: &mut Archive<R>, to_find: P) -> Result<Option<(Header, Vec<u8>)>>
    where
        R: 'a + Read + Seek,
        P: PartialEq<Path>
{
    rewind(archive);
    for entry in archive.entries()? {
        let mut entry = entry?;
        let path = entry.path()?;
        if to_find == *path {
            let header = entry.header().clone();
            let data = read_from_entry(&mut entry)?;
            return Ok(Some((header, data)));
        }
    }
    Ok(None)
}

pub fn peek_in_tar<'a, R, P>(archive: &'a mut Archive<R>, to_find: P) -> Result<Option<Entry<'a, R>>>
    where
        R: 'a + Read + Seek,
        P: PartialEq<Path>
{
    rewind(archive);
    for file in archive.entries()? {
        let entry = file?;
        let path = entry.path()?;
        if to_find == *path {
            return Ok(Some(entry));
        }
    }
    Ok(None)
}

pub fn move_in_tar<P: AsRef<Path>>(cursor: &mut File, from: P, to: P) -> Result<bool> {
    cursor.seek(SeekFrom::Start(0))?;
    let mut archive = Archive::new(cursor);
    let mut found = None;
    for entry in archive.entries_with_seek()? {
        let entry = entry?;
        let p = entry.path()?;
        if p == from.as_ref() {
            let start = entry.raw_header_position();
            let header = entry.header().clone();
            found = Some((start, header));
            break;
        }
    }
    match found {
        None => Err(Error::new(ErrorKind::NotFound, "Could find the file requested")),
        Some((start, mut header)) => {
            let cursor = archive.into_inner();
            let deleted_old = delete_in_tar(cursor, to.as_ref().to_path_buf())?;
            header.set_path(to)?;
            cursor.seek(SeekFrom::Start(start))?;
            cursor.write_all(header.as_bytes())?;
            Ok(deleted_old)
        }
    }
}

pub fn make_generic_header(length: u64) -> Result<Header> {
    let mut header = Header::new_gnu();
    header.set_size(length);
    header.set_gid(100);
    header.set_uid(1000);
    header.set_mode(0o666);
    let now = SystemTime::now();
    let now = now
        .duration_since(UNIX_EPOCH)
        .expect("Time went backwards").as_secs();
    header.set_mtime(now);
    header.set_groupname("users")?;
    Ok(header)
}

I hope this helps out anyone looking into modify archives. I wrote this a while back and may not be the prettiest code, but it worked for me.
Below are a series of functions I used in conjunction with this library to perform more high-level access to tar archives. The public functions such as write_to_tar, move_in_tar, delete_in_tar, etc. are the commands commonly used for typical tar access.

write_to_tar will append to an existing archive if the file does not exist. Should the file already exist in the archive, the existing archive is expanded or shrunk to accommodate the new file size and spliced into the archive.
move_in_tar is used to rename/move a file in the archive.
delete_in_tar is used to delete the file and shrinking the archive.
get_in_tar is used to return a tuple of the header and file contents as a Vec<u8> if found, else Ok(None) is returned.

use std::fs::File;
use std::io::{Cursor, Error, ErrorKind, Read, Result, Seek, SeekFrom, Write};
use std::path::Path;
use std::time::{SystemTime, UNIX_EPOCH};

use more_asserts;
use take_mut::take;
use tar::{Archive, Builder, Entry, Header};

const PAGE_SIZE: u64 = 1024;

fn aligned_end(end: u64) -> u64 {
    let new_end = end & !0x1ff;
    if end & 0x1ff == 0 { new_end } else { new_end + 0x200 }
}

fn find_end(start: u64, size: u64) -> u64 {
    aligned_end(start + 512 + size)
}

fn entry_positions<R: Read>(entry: &Entry<R>) -> (u64, u64) {
    let start = entry.raw_header_position();
    let sz = entry.size();
    let end = find_end(start, sz);
    (start, end)
}

fn expand_file(cursor: &mut File, start: u64, end: u64) -> Result<()> {
    if start == end {
        return Ok(());
    }
    // No cropping needed
    more_asserts::debug_assert_gt!(end, start);
    let mut buff = [0u8; PAGE_SIZE as usize];
    let diff = end - start;
    let file_end = cursor.seek(SeekFrom::End(0))?;
    for b in (end..(file_end + diff)).step_by(PAGE_SIZE as usize).collect::<Vec<_>>().iter().rev() {
        let a = b - diff;
        cursor.seek(SeekFrom::Start(a))?;
        let n = cursor.read(&mut buff)?;
        cursor.seek(SeekFrom::Start(*b))?;
        cursor.write_all(&buff[..n])?;
    }
    Ok(())
}

fn collapse_file(cursor: &mut File, mut start: u64, mut end: u64) -> Result<()> {
    if start == end {
        return Ok(());
    }
    // No adjusting needed
    more_asserts::debug_assert_gt!(end, start);
    let current_end = cursor.seek(SeekFrom::End(0))?;
    let future_end = current_end - end + start;
    debug!("Currently at: {}, need to reduce to: {}", current_end, future_end);
    cursor.seek(SeekFrom::Start(end))?;
    let mut buff = [0u8; PAGE_SIZE as usize];
    loop {
        let n = cursor.read(&mut buff)?;
        if n == 0 { break; }
        end += n as u64;
        cursor.seek(SeekFrom::Start(start))?;
        cursor.write_all(&buff[..n])?;
        start += n as u64;
        cursor.seek(SeekFrom::Start(end))?;
    }
    debug_assert_eq!(start, future_end);
    cursor.set_len(start)
}

/// This is largely for sanity checking to see that the archive is not corrupted
fn verify_archive(cursor: &mut File) -> Result<()> {
    cursor.seek(SeekFrom::Start(0))?;
    let mut a = Archive::new(cursor);
    for entry in a.entries()? {
        entry?;
    }
    Ok(())
}

pub fn write_to_tar<P: AsRef<Path>>(
    cursor: &mut File,
    path: P,
    data: &[u8],
    mut header: Header,
) -> Result<()> {
    let path = path.as_ref();
    debug!("Attempting to write {} bytes to: {:?}", data.len(), path);
    // Lets see if the file already exists in the archive
    cursor.seek(SeekFrom::Start(0))?;
    let mut archive = Archive::new(cursor);
    let mut found = None;
    for entry in archive.entries()? {
        let entry = entry?;
        let p = entry.path()?;
        if p == path {
            let start = entry.raw_header_position();
            let sz = entry.size();
            found = Some((start, sz));
            break;
        }
    }

    // Lets pull out the file from the archive object
    let mut cursor = archive.into_inner();
    let (position, overwriting) = if let Some((start, sz)) = found {
        let current_end = find_end(start, sz); // This is where `the tar archive ends to this file
        let new_end = find_end(start, data.as_ref().len() as u64); // This is where we need to go
        if new_end > current_end {
            debug!("Expanding! >>>");
            expand_file(&mut cursor, current_end, new_end)?;
        } else if new_end < current_end {
            debug!("Collapsing! <<<");
            collapse_file(&mut cursor, current_end, new_end)?;
        } else {
            debug!("Fitting! ===");
        }
        let pos = cursor.seek(SeekFrom::Start(start))?;
        (pos, true)
    } else {
        debug!("File did not exist before. Setting to (near) the end of the archive for writing.");
        let pos = cursor.seek(SeekFrom::End(-1024))?;
        (pos, false)
    };

    debug!("The file {:?} will be written at {:x}", path, position);

    // The file was not found in the archive, so lets add this file to the end of the archive
    let mut builder = Builder::new(cursor);
    header.set_size(data.len() as u64);
    let data = Cursor::new(data);
    builder.append_data(&mut header, path, data)?;

    if overwriting {
        // The builder will always write 1024 0s at the wherever the file cursor is when the builder is dropped. So, to stop
        // the builder from incorrectly terminating the archive in the middle of the archive, we will move the cursor to the
        // actual end so the archive so it will re-terminate the end of the archive.
        let f = builder.get_mut();
        f.seek(SeekFrom::End(-1024))?;
    }
    Ok(())
}

/// This function will delete the requested file (`path`) within the tar archive (`cursor`).
/// Please note that this function will only work if you open a file with read and write permissions
pub fn delete_in_tar<P: PartialEq<Path>>(cursor: &mut File, path: P) -> Result<bool> {
    cursor.seek(SeekFrom::Start(0))?;
    let mut archive = Archive::new(cursor);
    for entry in archive.entries_with_seek()? {
        let entry = entry?;
        let p = entry.path()?;
        if path == *p {
            let (start, end) = entry_positions(&entry);
            let sz = entry.size();
            let mut cursor = archive.into_inner();
            debug!("Deleting a file from {:x} to {:x} (sz={})", start.clone(), end.clone(), sz);
            collapse_file(&mut cursor, start, end)?;
            return Ok(true);
        }
    }
    Ok(false)
}

pub fn read_from_entry<R: Read>(entry: &mut Entry<R>) -> Result<Vec<u8>> {
    let mut buff = Vec::with_capacity(entry.size() as usize);
    let _n = entry.read_to_end(&mut buff)?;
    Ok(buff)
}

fn rewind<R: Read + Seek>(archive: &mut Archive<R>) {
    take(archive, |archive| {
        let mut cursor = archive.into_inner();
        cursor.seek(SeekFrom::Start(0)).unwrap();
        Archive::new(cursor)
    });
}

pub fn get_in_tar<'a, R, P>(archive: &mut Archive<R>, to_find: P) -> Result<Option<(Header, Vec<u8>)>>
    where
        R: 'a + Read + Seek,
        P: PartialEq<Path>
{
    rewind(archive);
    for entry in archive.entries()? {
        let mut entry = entry?;
        let path = entry.path()?;
        if to_find == *path {
            let header = entry.header().clone();
            let data = read_from_entry(&mut entry)?;
            return Ok(Some((header, data)));
        }
    }
    Ok(None)
}

pub fn peek_in_tar<'a, R, P>(archive: &'a mut Archive<R>, to_find: P) -> Result<Option<Entry<'a, R>>>
    where
        R: 'a + Read + Seek,
        P: PartialEq<Path>
{
    rewind(archive);
    for file in archive.entries()? {
        let entry = file?;
        let path = entry.path()?;
        if to_find == *path {
            return Ok(Some(entry));
        }
    }
    Ok(None)
}

pub fn move_in_tar<P: AsRef<Path>>(cursor: &mut File, from: P, to: P) -> Result<bool> {
    cursor.seek(SeekFrom::Start(0))?;
    let mut archive = Archive::new(cursor);
    let mut found = None;
    for entry in archive.entries_with_seek()? {
        let entry = entry?;
        let p = entry.path()?;
        if p == from.as_ref() {
            let start = entry.raw_header_position();
            let header = entry.header().clone();
            found = Some((start, header));
            break;
        }
    }
    match found {
        None => Err(Error::new(ErrorKind::NotFound, "Could find the file requested")),
        Some((start, mut header)) => {
            let cursor = archive.into_inner();
            let deleted_old = delete_in_tar(cursor, to.as_ref().to_path_buf())?;
            header.set_path(to)?;
            cursor.seek(SeekFrom::Start(start))?;
            cursor.write_all(header.as_bytes())?;
            Ok(deleted_old)
        }
    }
}

pub fn make_generic_header(length: u64) -> Result<Header> {
    let mut header = Header::new_gnu();
    header.set_size(length);
    header.set_gid(100);
    header.set_uid(1000);
    header.set_mode(0o666);
    let now = SystemTime::now();
    let now = now
        .duration_since(UNIX_EPOCH)
        .expect("Time went backwards").as_secs();
    header.set_mtime(now);
    header.set_groupname("users")?;
    Ok(header)
}

Oh ! Thank you for that ! I'll dive into it during the week but no doubt it will help me fix my program :) On top of that I'm pretty sure that I'll have a better understanding after reading your code.
Thanks again, I'll comment again if needed ;)


[BUG] I am unable to init project as described in CHEATSHEET.md

Describe the bug
I get an error when trying to initialize a shopware-pwa project using shopware-pwa init. Attempted on Windows 10 64-bit, node version 10.16.3, yarn version 1.19.0
How to reproduce

Clone repo
In project root, execute 'yarn' and 'yarn build --types'
Make a new project folder outside of shopware-pwa project root
cd inside new project folder
Execute 'shopware-pwa init'

Expected behavior
A new project to be initialized
Actual behavior
Got 'command failed' error (see screenshot)
Screenshots

Devices:

Device: Desktop
OS: Windows 10 x64
Browser: N/A

Windows problems with CLI has been fixed, I'm closing this issue.
Here's instruction on how to start: https://github.com/DivanteLtd/shopware-pwa#quick-start
@FlushBG in case of further troubles please let us know :)


bugs in market BTS:CNY

Not really a bug but a "feature" of the market. The bids are too high to fullfill the criteria needed for them to match the margin orders. There needs to be a buy in the range 0.0228 - 0.0230, and nothing above that in order for the call orders to fill.


v2.1.0 improvements

Further improve library usage, something like this...

>>> import scanless
>>> sl = scanless.Scanless()
>>> sl.scan('scanme.nmap.org', scanner='hackertarget')
{'pretty_output': '...', 'raw_output': [(22, 'open'), (23, 'closed'), ...]}

Output colors on CLI using crayons.
Move random.choice funcs into a util.py or something
3.6 in setup.py, whoops.

Done :)
Most of the core work in these commits:

https://github.com/vesche/scanless/commit/cceaee3a59a7f87c8532c882896710b13660be20
https://github.com/vesche/scanless/commit/70876ab5522bb31b9f6911d48e794622d44e0e07


[SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

What changes were proposed in this pull request?
Before this PR ShuffleBlockFetcherIterator was partitioning the block fetches into two distinct sets: local reads and remote fetches. Within this PR (when the feature is enabled by "spark.shuffle.readHostLocalDisk.enabled") a new category is introduced: host-local reads. They are shuffle block fetches where although the block manager is different they are running on the same host along with the requester.
Moreover to get the local directories of the other executors/block managers a new RPC message is introduced GetLocalDirs which is sent the the block manager master where it is answered as BlockManagerLocalDirs. In BlockManagerMasterEndpoint for answering this request the localDirs is extracted from the BlockManagerInfo and stored separately in a hash map called executorIdLocalDirs. Because the earlier used blockManagerInfo contains data for the alive block managers (see org.apache.spark.storage.BlockManagerMasterEndpoint#removeBlockManager).
Now executorIdLocalDirs knows all the local dirs up to the application start (like the external shuffle service does) so in case of an RDD recalculation both host-local shuffle blocks and disk persisted RDD blocks on the same host can be served by reading the files behind the blocks directly.
How was this patch tested?
Unit tests
ExternalShuffleServiceSuite:

"SPARK-27651: host local disk reading avoids external shuffle service on the same node"

ShuffleBlockFetcherIteratorSuite:

"successful 3 local reads + 4 host local reads + 2 remote reads"

And with extending existing suites where shuffle metrics was tested.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13491/
Test PASSed.

Test build #108396 has started for PR 25299 at commit 7210742.

Test build #108396 has finished for PR 25299 at commit 7210742.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class GetLocalDirs(executorIds: Array[String]) extends ToBlockManagerMaster
  case class BlockManagerLocalDirs(localDirs: Seq[Array[String]])

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108396/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13497/
Test PASSed.

Test build #108403 has started for PR 25299 at commit d509278.

Test build #108403 has finished for PR 25299 at commit d509278.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108403/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13502/
Test PASSed.

Test build #108406 has started for PR 25299 at commit 894e772.

Test build #108406 has finished for PR 25299 at commit 894e772.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108406/
Test FAILed.

Test build #108407 has started for PR 25299 at commit 17552ca.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13503/
Test PASSed.

Test build #108407 has finished for PR 25299 at commit 17552ca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108407/
Test PASSed.

does this work with external shuffle service?

Yes.
Basically external shuffle service basically does the same: I mean it reads the local disks of the host local executors directly (it has its own mapping from app ID and executor ID tuples to local disk). With this feature a regular executor will be able to do the same (here the mapping from executor to local disk is at the block manager master).

Do you mean people can already avoid the unnecessary network if they turn on external shuffle service?

No. To fetch a block from the external shuffle service the network must be used.
Here the executor can read the block for itself from the disk directly so the fetching via network is saved. What I meant the mechanism how it is done (reading the disk of the other executors) is the very same.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13562/
Test PASSed.

Test build #108468 has started for PR 25299 at commit 193a0eb.

Test build #108468 has finished for PR 25299 at commit 193a0eb.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class BlockManagerLocalDirs(localDirs: Map[String, Array[String]])

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108468/
Test FAILed.

@tgravescs I think we talked about this a long time ago, and you thought spark already had this optimization in place, so you might be interested in this change

yes we had talked about this and I looked briefly, but like mentioned in the description you have to have a way for each to know about different executor locations.
@attilapiros  I'm curious did you run any performance numbers to see if it makes a difference?

@tgravescs unfortunately haven't run any as for this case I have no good performance tests.
But I am positive it makes a difference especially for small clusters.

I actually do not see much value in adding the extra metric

I'm not totally sold on it either - I can see some value, but maybe not enough to justify all the hassle around actually adding it...

I can revert the metric changes and can count the host-local related bytes and number of blocks into the local ones. So should I go ahead and do the revert?

Sounds good to me.

Test build #108748 has started for PR 25299 at commit 7dded2c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13831/
Test PASSed.

Test build #108748 has finished for PR 25299 at commit 7dded2c.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108748/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13850/
Test PASSed.

Test build #108769 has started for PR 25299 at commit 50bca42.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108769/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13970/
Test PASSed.

Test build #108896 has started for PR 25299 at commit c51f107.

Test build #108896 has finished for PR 25299 at commit c51f107.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108896/
Test FAILed.

just FYI: I remember there was the similar work for this improvement a few years ago: SPARK-6521(https://github.com/apache/spark/pull/9478). We closed the ticket cuz we didn't get actual performance gains. Anyway, I added a link to it in the jira.

@maropu I think the performance gains would be significant if the block size is over "spark.maxRemoteBlockSizeFetchToMem" (which default is 2GB), then without this change the shuffle block during fetching:

would be read from disk
sent via the network
streamed to disk
and finally re-read from disk when it is used.

With this change it would be just read from disk directly.

jenkins retest this please

jenkins retest this please

retest this please.

Test build #4831 has started for PR 25299 at commit 40b7339.

ok to test

Test build #4831 has finished for PR 25299 at commit 40b7339.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14238/
Test PASSed.

Test build #109169 has started for PR 25299 at commit 496c0f9.

Test build #109169 has finished for PR 25299 at commit 496c0f9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109169/
Test FAILed.

retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14292/
Test PASSed.

Test build #109225 has started for PR 25299 at commit af99e79.

Test build #109225 has finished for PR 25299 at commit af99e79.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109225/
Test FAILed.

The last three builds were failing for different reasons and all seams to me unrelated for my change. I trigger the build again without any change (previously rebased on top of master).
jenkins retest this please

jenkins retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14339/
Test PASSed.

Test build #109271 has started for PR 25299 at commit af99e79.

Test build #109271 has finished for PR 25299 at commit af99e79.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109271/
Test PASSed.

Thanks for looking @cloud-fan , I think there actually is a very interesting idea about avoiding asking the driver.  The executors could ask each other for the local dirs (or ask the external shuffle service, so we know those dirs even after the executor goes down, for the cases where that is useful).
Another idea would be to have the executors communicate via the filesystem -- there is some agreed upon location where each exec writes its set of local dirs.  This optimization is only useful where the executors share a filesystem.  This may also indirectly help with the k8s case you mentioned.
On k8s, its a good question.  I don't think the performance benefit will work, as the pods generally are isolated from each other, and can't read each others filesystems.  But we have to make sure things aren't broken by this because they try to read from a place they don't have access to.

Another idea would be to have the executors communicate via the filesystem

This seems a better idea to me. Furthermore, if executor's data dir path is constructed deterministically with the executor id, then we can access data files of other executors directly within the same host.

So the downside to using a disk is that at least on yarn containers may not see the same disks. one containers might start and get all the disks.  Then a disk perhaps gets temporarily full and another container starts and it has all the disks minus that full one.  The other downside is that you have to go scan all the disks in your list to find whatever file you are using to communicate with, so the question is scanning all the disks slower then getting the list from the driver.  I know some people run with 20+ disks, although at my previous job most hosts were 4 to 6.  It would definitely be nice to not have to go to the driver though.  We could try a hybrid approach where it looks for a file in the application level directory by scanning the disks and uses that if it finds it and if it doesn't falls back to the shuffle service or falls back to asking the driver.  I would hope its fairly rare to not find it so just falling back to the shuffle service would make it simpler.  I'm definitely in favor of not hitting the driver if we can help it.
Really on yarn, other then the issue of different containers getting different disks, there is no reason you need to know the local dirs at all as it could just scan the disks, but that would require change to the hashing of filename code in the blockmanager.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15244/
Test PASSed.

Test build #110247 has started for PR 25299 at commit ba8a73e.

The current state only covers the original idea to have communication with the driver: no files (https://github.com/apache/spark/pull/25299#issuecomment-527029909) and no async parts (https://github.com/apache/spark/pull/25299#discussion_r318239195).
If you agree I would do the mentioned improvements in a separate PR / Jira. What do you think?

Test build #110247 has finished for PR 25299 at commit ba8a73e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110247/
Test PASSed.

I would do the mentioned improvements in a separate PR

If you want to do that I'd have this feature disabled by default until those are implemented. I'm not sure the current approach is a win, since executors may block at the start of a shuffle waiting for a reply from the driver.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15657/
Test PASSed.

Test build #111775 has started for PR 25299 at commit 54542f9.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16758/
Test PASSed.

Test build #111775 has finished for PR 25299 at commit 54542f9.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111775/
Test FAILed.

Test build #111778 has started for PR 25299 at commit dca309e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16761/
Test PASSed.

Test build #111778 has finished for PR 25299 at commit dca309e.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111778/
Test FAILed.

Test build #111779 has started for PR 25299 at commit 4ad9f22.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16762/
Test PASSed.

Test build #111779 has finished for PR 25299 at commit 4ad9f22.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111779/
Test PASSed.

+1 for go ahead with the current implementation and disable the feature by default.

I am working on a solution where the local directories for the executor (which are not cached at the executor side) is loaded from the external shuffle service asynchronously.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18605/
Test PASSed.

Test build #113725 has started for PR 25299 at commit 4a0979a.

 And it is uses the external shuffle service where directories are cached for all host local executors (as org.apache.spark.network.shuffle.ExternalShuffleBlockResolver#executorRemoved does not remove any entry from executors) so this way no fallback to remote fetch is needed.

Test build #113725 has finished for PR 25299 at commit 4a0979a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class GetLocalDirsForExecutors extends BlockTransferMessage
public class LocalDirsForExecutors extends BlockTransferMessage

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113725/
Test FAILed.

Test build #113729 has started for PR 25299 at commit eafeb47.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18609/
Test PASSed.

Test build #113729 has finished for PR 25299 at commit eafeb47.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18689/
Test PASSed.

Test build #113817 has started for PR 25299 at commit 8792931.

Test build #113817 has finished for PR 25299 at commit 8792931.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113817/
Test FAILed.

jenkins retest this please

Test build #113880 has started for PR 25299 at commit 8792931.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18748/
Test PASSed.

Test build #113880 has finished for PR 25299 at commit 8792931.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113880/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18895/
Test PASSed.

Test build #114034 has started for PR 25299 at commit aa8c4c7.

Test build #114034 has finished for PR 25299 at commit aa8c4c7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114034/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19022/
Test PASSed.

Test build #114168 has started for PR 25299 at commit 1f53e06.

Test build #114168 has finished for PR 25299 at commit 1f53e06.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114168/
Test PASSed.

Test build #114290 has started for PR 25299 at commit d0d5ce7.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19131/
Test PASSed.

Test build #114290 has finished for PR 25299 at commit d0d5ce7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114290/
Test PASSed.

jenkins retest this please

Test build #114310 has started for PR 25299 at commit 52037b7.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19150/
Test PASSed.

Test build #114310 has finished for PR 25299 at commit 52037b7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114310/
Test PASSed.

I only skimmed this but overall looks good, like this approach.
When fetching the host local blocks do we have any limits on # that can happen concurrently?  I didn't see anything config on just skimming it. If we have a lot of other executors on the same node we end up bypassing the remote fetch configs as to # in parallel so now we are doing more work at once here which could potentially cause more local disk activity on both the read and write sides.

@tgravescs The block content reading will occur at the client side of the 'ShuffleBlockFetchIterator' as the blocks are wrapped into 'FileSegmentManagedBuffer'.
Over and above on one specific executor the host local blocks with cached executor directories are handled synchronously in one single thread (just like local blocks) and host local blocks for which directories are not cached are handled on a separate but single thread (on the thread which used to get all the missing directories via one RPC call).

ok that makes sense. thanks.  So it seems we fetch all the local blocks in first and then the host local blocks and all that happens in the current thread while the remote block fetcher happens in separate.
My concern could go both ways then. Now we are having to wait to fetch the local blocks first before the host local blocks where as before that could have been in parallel with the local blocks.  Or if we end up with a lot of host local blocks then within those fetches they happen serially now whereas they would have happened in parallel before.
Mostly just an observation to something we may want to enhance later if we see it as an issue.

Now we are having to wait to fetch the local blocks first before the host local blocks where as before that could have been in parallel with the local blocks.

"Host local" blocks are basically local blocks after you fill in the local dirs cache. And "fetch" is a misnomer in this case; you'll get a pointer to the data (which mostly involves metadata operations like finding the file, but not actually opening it), and it will only be actually opened when the task starts running. So not really any waiting involved.
Sounds like a gain to me - while before you'd have RPC + open on the remote side + transfer data over socket + cache it in memory in executor (or, worst, write it to disk again + open another file), now you only do an "open file".

Definitely agree, sounds like gain and been wanting this for a while so thanks @attilapiros  for working on it. And you're right I wasn't thinking about this not even opening the file so should be very fast.

Test build #114429 has started for PR 25299 at commit 22aa383.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19265/
Test PASSed.

Test build #114429 has finished for PR 25299 at commit 22aa383.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114429/
Test PASSed.

Test build #114464 has started for PR 25299 at commit da93837.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19298/
Test PASSed.

jenkins retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19302/
Test PASSed.

Test build #114468 has started for PR 25299 at commit da93837.

test this please

Test build #114470 has started for PR 25299 at commit da93837.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19304/
Test PASSed.

The error is:
[ERROR] Failed to execute goal on project spark-catalyst_2.12: Could not resolve dependencies for project org.apache.spark:spark-catalyst_2.12:jar:3.0.0-SNAPSHOT: The following artifacts could not be resolved: org.codehaus.janino:commons-compiler:jar:3.0.15, com.univocity:univocity-parsers:jar:2.8.3, org.apache.arrow:arrow-vector:jar:0.15.1: Could not transfer artifact org.codehaus.janino:commons-compiler:jar:3.0.15 from/to central (https://repo.maven.apache.org/maven2): Connection timed out (Read failed) -> [Help 1]

Meanwhile this link is working: janino 3.0.15.
Could it be the local maven repo in the Jenkins which runs the PR builder should be purged?

Test build #114464 has finished for PR 25299 at commit da93837.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114464/
Test PASSed.

Test build #114468 has finished for PR 25299 at commit da93837.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114468/
Test FAILed.

Test build #114470 has finished for PR 25299 at commit da93837.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114470/
Test PASSed.

Merging to master.