[SPARK-10586] fix bug: BlockManager ca't be removed when it is re-registered, then disassociats

Scene: When the executor has been removed, but it still exists on the SparkUI web.
Process:

Driver Lost executor because heartbeat timed out;
Executor received SIGNAL 15: SIGTERM and re-registered the blockmanager;
Driver Lost executor because remote Rpc client disassociated and attempt to remove blockmanager but not success;

Reason:
The first time lost executor, the SchedulerBackend reviveOffers, when the host also has other executors, it won't add this executor. So the second time lost executor, DAGScheduler won't remove BlockManager because the executorId has been in failedEpoch.

Merged build triggered.

Merged build started.

Test build #42416 has started for   PR 8741 at commit bb562a2.

Test build #42416 has finished for   PR 8741 at commit bb562a2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class Stddev(child: Expression) extends StddevAgg(child)
case class StddevPop(child: Expression) extends StddevAgg(child)
case class StddevSamp(child: Expression) extends StddevAgg(child)
abstract class StddevAgg(child: Expression) extends AlgebraicAggregate
abstract class StddevAgg1(child: Expression) extends UnaryExpression with PartialAggregate1
case class Stddev(child: Expression) extends StddevAgg1(child)
case class StddevPop(child: Expression) extends StddevAgg1(child)
case class StddevSamp(child: Expression) extends StddevAgg1(child)
case class ComputePartialStd(child: Expression) extends UnaryExpression with AggregateExpression1
case class ComputePartialStdFunction (
case class MergePartialStd(
case class MergePartialStdFunction(
case class StddevFunction(
case class IntersectNode(conf: SQLConf, left: LocalNode, right: LocalNode)
case class SampleNode(
case class TakeOrderedAndProjectNode(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42416/
Test FAILed.

Merged build triggered.

Merged build started.

Test build #44350 has started for PR 8741 at commit bb562a2.

Test build #44350 has finished for PR 8741 at commit bb562a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44350/
Test PASSed.

Merged build finished. Test PASSed.

Hi @XuTingjun , can you add a test case for this, perhaps to TaskSchedulerImplSuite?  If I understand correctly, the key issue is when you have two executors on one host, so that if you lose one and try to add it back the re-registration is wrong.  But I'm not sure I follow the details -- a test case would really help clarify.

@XuTingjun I'm also not sure if I follow. If the driver loses an executor, it should just remove it from all of its schedulers' data structures. When the executor re-registers, it's a new executor according to the driver, so there's no problem. Actually, can you test out the latest code in master and verify whether this problem still exists?


angel rotate when stun

That was because of something weird with slide Stop method. removed that for the time being


Page freeze when loading several failed jobs

When loading several failed jobs (38 in our instance), the entire page freezes until request is completed.
No console errors (aside from React Dev messages) and the request eventually returns (after 2 minutes).

There is the potential for a lot of data to be transmitted here, based on the size of your job payload and the error trace.   By default, the page will try to load 100 failed jobs before pagination https://github.com/evantahler/ah-resque-ui/blob/master/actions/ah-resque-ui.js#L150-L177.  You can change this in either the action's default or in the font-end's request.
When you have the page open, look at your network requests tab in chrome and look the resqueFailed request.  Where is that 2 minutes being sent?  Is it 2 min waiting for the server to send data to the browser, 2 min for the browser to parse and render that data, etc.

Issue was the result of large resque tasks.
Refactored to move data out of task and not long have issues.


Failed to acquire resource (boskos / mason)

https://k8s-gubernator.appspot.com/build/istio-prow/pull/istio_istio/4778/e2e-simpleTests/2443/
I0410 22:26:41.233] failed to get a Boskos resource
I0410 22:26:41.233] failed to kill mason client
I0410 22:26:41.234] time="2018-04-10T22:16:51Z" level=info msg="Attempting to acquire resource"
I0410 22:26:41.234] time="2018-04-10T22:16:51Z" level=info msg="Failed to acquire resource"

...

I0410 22:26:41.245] time="2018-04-10T22:26:24Z" level=info msg="Failed to acquire resource"
I0410 22:26:41.245] time="2018-04-10T22:26:34Z" level=info msg="Attempting to acquire resource"
I0410 22:26:41.245] time="2018-04-10T22:26:40Z" level=info msg="Failed to acquire resource"
I0410 22:26:41.246] time="2018-04-10T22:26:41Z" level=panic msg="unable to find a resource" error="context deadline exceeded"
I0410 22:26:41.246] panic: (*logrus.Entry) (0x8cd4c0,0xc4202e4dc0)
I0410 22:26:41.246]
I0410 22:26:41.246] goroutine 1 [running]:
I0410 22:26:41.246] istio.io/test-infra/vendor/github.com/sirupsen/logrus.Entry.log(0xc42009e140, 0xc420322ea0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
I0410 22:26:41.246] 	/home/bootstrap/go/src/istio.io/test-infra/vendor/github.com/sirupsen/logrus/entry.go:128 +0x59c
I0410 22:26:41.246] istio.io/test-infra/vendor/github.com/sirupsen/logrus.(*Entry).Panic(0xc4202e4d70, 0xc4201add88, 0x1, 0x1)
I0410 22:26:41.246] 	/home/bootstrap/go/src/istio.io/test-infra/vendor/github.com/sirupsen/logrus/entry.go:173 +0xaa
I0410 22:26:41.246] istio.io/test-infra/vendor/github.com/sirupsen/logrus.(*Entry).Panicf(0xc4202e4d70, 0x8e9a01, 0x19, 0x0, 0x0, 0x0)
I0410 22:26:41.247] 	/home/bootstrap/go/src/istio.io/test-infra/vendor/github.com/sirupsen/logrus/entry.go:221 +0xed
I0410 22:26:41.247] main.main()
I0410 22:26:41.247] 	/home/bootstrap/go/src/istio.io/test-infra/boskos/cmd/mason_client/main.go:147 +0x8c4
W0410 22:26:41.247] + echo 'failed to get a Boskos resource'

I think this has been fixed, and boskos resources can be monitored at https://velodrome.istio.io/dashboard/db/boskos-dashboard?orgId=1.


egraphs: transpose select on operation to operation on select when it increases sharing

This is a second attempt of https://github.com/bytecodealliance/wasmtime/pull/6219, but this time only includes rewrites that are obviously beneficial by reducing number of instructions

cc: @elliottt?


No CA / SSL Certs (can't git clone etc)

Hi chaps,
I can't download files from https websites with this image. Do you know if there is a package I can install with apt-get to get around this?
# cd .; git clone https://github.com/tools/godep /go/src/github.com/tools/godep
Cloning into '/go/src/github.com/tools/godep'...
fatal: unable to access 'https://github.com/tools/godep/': Problem with the SSL CA cert (path? access rights?)
package github.com/tools/godep: exit status 128

Thanks,
Alex

apt-get install ca-certificates

?

This image provides only a redis database so there is no need for git clone and other packages.
If you would like to use it as base for developing you can easily create a new image with FROM hypriot/rpi-redis. But you can also develop in another container and link them together.

OK I raised this on the wrong repo (window) - I meant to raise it against hypriot/rpi-raspbian, wrong window - but usually I'd check with the person who raised the bug before closing immediately.

The image I wanted to mention was with the base image.. that you guys are using, not the redis container. I think I raised this on the wrong repo - but you don't have one for the base image.


Updating solvers in transient block-problems correctly

Hi again,
I want to solve a very simple linear PDAE with MFEM with time adaptivity, which I model as a block problem. However it seems like I do something wrong when updating the linear operator for my solver, as changing the time step results in divergence at some random timepoint. As a side-note, the linear PDAE results from an operator splitting of a degenerate reaction-diffusion system. After discretizing the system with an implicit Euler I obtain the following linear problem:
|    T     Δt*K1 |  |  V2 | = | M V1 |
|  Δt*K1   Δt*KK |  | φe2 |   |   0  |

where K1 and KK are discretizations of diffusion operators and T is the mass matrix and another discretization of a diffusion operator, so everything is symmetric (utilizing linear Lagrange elements).
With constant Δt I do not run into much issues solving the problem with CG or PCG (with a block diagonal preconditioner).  However if Δt is perturbated in each timestep, then neither CG or PCG converge, sometimes. For testing purposes I have a very small Δt=0.005 and my perturbation is randomly sampled from the uniform distribution [-0.001, 0.005].
Here the relevant lines of my Euler step
if (Δt_prev != Δt)
{
    // Assemble atomar matrices
    T = std::unique_ptr<mfem::HypreParMatrix>(Add(1.0, M, Δt / (model->χ * model->Cₘ), K1_V));

    ΔtK1_V = std::unique_ptr<mfem::HypreParMatrix>(
        Add(Δt / (model->χ * model->Cₘ), K1_V, 0.0, K1_V));
    ΔtK1_φe = std::unique_ptr<mfem::HypreParMatrix>(
        Add(Δt / (model->χ * model->Cₘ), K1_φe, 0.0, K1_φe));

    KK = std::unique_ptr<mfem::HypreParMatrix>(
        Add(Δt / (model->χ * model->Cₘ), K1_φe, Δt / (model->χ * model->Cₘ), K2));

    // Set preconditioner - removed for CG run
    invP->SetOperator(*T);
    invE->SetOperator(*KK);
    LPEB_prec->SetDiagonalBlock(0, invP.get());
    LPEB_prec->SetDiagonalBlock(1, invE.get());
    LPEB_solver.SetPreconditioner(*LPEB_prec);
    // End of set preconditioner - removed for CG run

    LPEBOp->SetBlock(0, 0, T.get());
    LPEBOp->SetBlock(0, 1, ΔtK1_φe.get());
    LPEBOp->SetBlock(1, 0, ΔtK1_V.get());
    LPEBOp->SetBlock(1, 1, KK.get());
}

// Assemble RHS Block Vector
mfem::BlockVector rhs = u;
M.Mult(u.GetBlock(0), rhs.GetBlock(0));
rhs.GetBlock(1) = 0.0;

// Solve.
LPEB_solver.SetOperator(*LPEBOp);
LPEB_solver.iterative_mode = true;
LPEB_solver.Mult(rhs, u_new);
if(!LPEB_solver.GetConverged())
{
    if(fespace.GetMyRank() == 0)
    {
        std::cout << "BackwardEuler did not converge. Norm: " << LPEB_solver.GetFinalNorm() << std::endl;
    }
    MPI_Abort(MPI_COMM_WORLD, -1);
}

Δt_prev = Δt;

I do not understand what is going wrong. I append two runs with the iterative solver output (level 1).
CG.log
PCG.log

If the block system is symmetric, then K1_V and K1_φe must be transpose of each other. On the other hand, they are also added to the diagonal blocks, so the must be symmetric as well. Therefore, it looks like K1_V and K1_φe must be the same which does not seem right. Is this correct?

Thanks for the quick response! Indeed, I applied Dirichlet BCs to K1_φe, while I do not apply any BCs to K1_V, which seems to be the culprit. Interesting that this did not crash the solver earlier. Substituting the CG with GMRES works like a charm. Thanks for the pointer, I should have noticed earlier.
My follow up question relating to the code snippet would be: Is there a more efficient way to update the matrices and BoomerAMG? All I want to do is to rescale my matrices (to conform to the new time step length) without allocating new matrices and especially without rebuilding the AMG (what happens when SetOperator is called, if I understand correctly).

You may be okay just reusing the preconditioner built for some fixed dt, even when the actual dt is changing. Another option is to only rebuild the preconditioner when dt changes (increases or decreases) more than some factor, say 10.

Good point, that should do it too. Thanks for the help!


Chat flags.

Default config no longer has features on
Players will now have their chat name prefixed with "[You]", "[Friend]" or "[Cheater]" when they type in chat.
Fixed a crash with KeyValues (the issue isn't fixed but it doesn't crash and proxy skins work)
Antiwarp works while abusing z angles, and no longer does anything if the player will stop moving in the next tick anyway.

Merge conflicts 😍


How should the library deal with pre-FF4 properties?

SGFC converts to FF4 automatically, and writes FF4 only. A case can be made from this that the library should only support FF4 properties.
What would this mean for the current implementation?

The SgfcPropertyType enumeration contains not only game-specific properties, it also contains pre-FF4 properties.
If pre-FF4 properties go away, some of the internal value type descriptors might also no longer be needed. But should they be removed? What if FF5 ever happens?

SGFC has an enumeration token that contains the following elements that are not part of FF4:
TKN_CH, TKN_SI, TKN_BS, TKN_WS, TKN_ID, TKN_TC, TKN_OM, TKN_OP,
TKN_OV, TKN_LT, TKN_RG, TKN_SC, TKN_SE, TKN_EL, TKN_EX

The current status is that if the SGF content contains a property that is not known to SGFC it will be passed on to libsgfc++ without any particular handling. This is relatively OK for single value properties that have the value type SimpleText and Text, but for all other properties with more restrictive requirements this is a problem because libsgfc++ expects that SGFC performs a certain pre-processing, according to the internal value type descriptor assigned to that particular property.
Examples:

A Double value type property where SGFC does not check that the property value is "1" or "2".
A Color value type property where SGFC does not check that the property value is "B" or "W".
A Number or Real value type property where SGFC does not check that the property value is numeric.
A composed value property where SGFC does not perform the separation into value 1 and value 2.

The last example is the most critical, because it causes libsgfc++ (SgfcPropertyDecoder) to throw an exception (std::domain_error) because it detects an interfacing problem with SGFC.

SGFC performs the following conversions and passes on the converted properties to libsgfc++:

L to LB
M to MA
KI to KM

The other properties listed in the previous comment are not converted and passed on to libsgfc++ without change.

Decision: libsgfc++ supports the same property set as SGFC, i.e. it continues to support pre-FF4 properties. Library clients that do not wish to see pre-FF4 properties can use SgfcArgumentType::DeleteObsoleteProperties when reading in SGF content.


Feat/edb component

Codecov Report
Attention: Patch coverage is 94.73684% with 8 lines in your changes are missing coverage. Please review.

Project coverage is 81.56%. Comparing base (b65db6c) to head (5e5bf08).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4509      +/-   ##
==========================================
+ Coverage   81.46%   81.56%   +0.09%
==========================================
  Files         110      110
  Lines       53460    53519      +59
==========================================
+ Hits        43553    43653     +100
+ Misses       9907     9866      -41


there is a bug when method has the keyword 'from'

before call isort:
from a import d
from test import get_email_from, get_email_to
from b import c

after call isort:
from a import d
from b import c
from test import get_email_, get_email_to

isort version is: 3.9.4

below function has some error:
    def _strip_syntax(self, import_string):
        import_string = import_string.replace("_import", "[[i]]")
        for remove_syntax in ['\\', '(', ')', ",", 'from ', 'import ']:
            import_string = import_string.replace(remove_syntax, " ")
        import_string = import_string.replace("[[i]]", "_import")
        return import_string

I have gived my commit

Hi @Zuckonit,
Thanks for catching this! I will make sure the fix is integrated before the next release.
~Timothy

Pull request seems to be https://github.com/timothycrosley/isort/pull/216

@Zuckonit -- I extracted the minimal bit from this PR that I needed to fix my issue (https://github.com/timothycrosley/isort/issues/224). I also added a test for at last part of your issue; see https://github.com/timothycrosley/isort/pull/230. If your other changes are necessary, I think they may belong in a separate issue. At least, we need a test to see what problem they are solving.

This is fixed officially in release 3.9.5.
Thanks again guys for your help!
~Timothy


Nuxt Build Error : "create" is not exported by @intlify/shared

Reporting a bug?
Looks like this happens since v10.0.5.
I had to delete my yarn.lock file to recreate one, then yarn automatically got the latest version
Now nuxi generate renders this error
<EMAIL_ADDRESS>(6:42): Error when using sourcemap for reporting an error: Can't resolve original location of error.
ℹ ✓ 1909 modules transformed.                                                                                9:53:22 AM

 ERROR  x Build failed in 4.15s                                                                                9:53:22 AM

[9:53:22 AM]  ERROR  Nuxt Build Error<EMAIL_ADDRESS>(6:42): "create" is not exported by<EMAIL_ADDRESS>imported by<EMAIL_ADDRESS>file: /PROJECT_PATH/node_modules/@intlify/core-base/dist/core-base.mjs:6:42

4:   * Released under the MIT License.
5:   */
6: import { getGlobalThis, hasOwn, isNumber, create, isObject, isString, isBoolean, warn, format as format$1, isFunction...
                                             ^
7: import { detectHtmlTag, defaultOnError, baseCompile as baseCompile$1, COMPILE_ERROR_CODES_EXTEND_POINT, createCompile...
8: export { CompileErrorCodes, createCompileError } from "@intlify/message-compiler";

  file: node_modules/@intlify/core-base/dist/core-base.mjs:6:42

  4:   * Released under the MIT License.
  5:   */
  6: import { getGlobalThis, hasOwn, isNumber, create, isObject, isString, isBoolean, warn, format as format$1, isFunction...
  ^
  7: import { detectHtmlTag, defaultOnError, baseCompile as baseCompile$1, COMPILE_ERROR_CODES_EXTEND_POINT, createCompile...
  8: export { CompileErrorCodes, createCompileError } from "@intlify/message-compiler";

  at getRollupError (node_modules/rollup/dist/es/shared/parseAst.js:396:41)
  at error (node_modules/rollup/dist/es/shared/parseAst.js:392:42)
  at Module.error (node_modules/rollup/dist/es/shared/node-entry.js:15593:16)
  at Module.traceVariable (node_modules/rollup/dist/es/shared/node-entry.js:16042:29)
  at ModuleScope.findVariable (node_modules/rollup/dist/es/shared/node-entry.js:13825:39)
  at Identifier.bind (node_modules/rollup/dist/es/shared/node-entry.js:5071:40)
  at CallExpression.bind (node_modules/rollup/dist/es/shared/node-entry.js:2658:23)
  at CallExpression.bind (node_modules/rollup/dist/es/shared/node-entry.js:11289:15)
  at VariableDeclarator.bind (node_modules/rollup/dist/es/shared/node-entry.js:2658:23)
  at VariableDeclaration.bind (node_modules/rollup/dist/es/shared/node-entry.js:2654:28)

Expected behavior
No build error
Reproduction
Use Nuxt 3 and install latest version of @nuxtjs/i18n
Rebuild the yarn.lock file
System Info
npm 10.8.2

node v20.11.0

yarn 4.5.0

nuxt 3.14.159

@nuxtjs/i18n ^9.1.0

Screenshot
No response
Additional context
No response
Validations

[X] Read the Contributing Guidelines
[X] Read the Documentation
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussions

Thank you for your reporting!
Sorry, that was a mistake in the tagging at the time of my release. I have just tagged latest as 10.0.5.
Please try to remove the lock file in your package manager and re-install your project.

It's working now :) thanks a lot for the quick fix


YARN-11178. Avoid CPU busy idling and resource wasting in DelegationTokenRenewerPoolTracker thread

Description of PR
The DelegationTokenRenewerPoolTracker thread is busy wasting CPU resource in empty poll iterate when there is no delegation token renewer event task in the futures map:
// org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.DelegationTokenRenewerPoolTracker#run
@Override
public void run() {
  // this while true loop is busy when the `futures` is empty
  while (true) {
    for (Map.Entry<DelegationTokenRenewerEvent, Future<?>> entry : futures
        .entrySet()) {
      DelegationTokenRenewerEvent evt = entry.getKey();
      Future<?> future = entry.getValue();
      try {
        future.get(tokenRenewerThreadTimeout, TimeUnit.MILLISECONDS);
      } catch (TimeoutException e) {

        // Cancel thread and retry the same event in case of timeout
        if (future != null && !future.isDone() && !future.isCancelled()) {
          future.cancel(true);
          futures.remove(evt);
          if (evt.getAttempt() < tokenRenewerThreadRetryMaxAttempts) {
            renewalTimer.schedule(
                getTimerTask((AbstractDelegationTokenRenewerAppEvent) evt),
                tokenRenewerThreadRetryInterval);
          } else {
            LOG.info(
                "Exhausted max retry attempts {} in token renewer "
                    + "thread for {}",
                tokenRenewerThreadRetryMaxAttempts, evt.getApplicationId());
          }
        }
      } catch (Exception e) {
        LOG.info("Problem in submitting renew tasks in token renewer "
            + "thread.", e);
      }
    }
  }
}

A better way to avoid CPU idling is waiting for some time when the futures map is empty, and when the renewer task done or cancelled, we should remove the task future in futures map to avoid memory leak:
@Override
public void run() {
  while (true) {
    // waiting for some time when futures map is empty
    if (futures.isEmpty()) {
      synchronized (this) {
        try {
          // waiting for tokenRenewerThreadTimeout milliseconds
          long waitingTimeMs = Math.min(10000, Math.max(500, tokenRenewerThreadTimeout));
          LOG.info("Delegation token renewer pool is empty, waiting for {} ms.", waitingTimeMs);
          wait(waitingTimeMs);
        } catch (InterruptedException e) {
          LOG.warn("Delegation token renewer pool tracker waiting interrupt occurred.");
          Thread.currentThread().interrupt();
        }
      }
      if (futures.isEmpty()) {
        continue;
      }
    }
    for (Map.Entry<DelegationTokenRenewerEvent, Future<?>> entry : futures
        .entrySet()) {
      DelegationTokenRenewerEvent evt = entry.getKey();
      Future<?> future = entry.getValue();
      try {
        future.get(tokenRenewerThreadTimeout, TimeUnit.MILLISECONDS);
      } catch (TimeoutException e) {

        // Cancel thread and retry the same event in case of timeout
        if (future != null && !future.isDone() && !future.isCancelled()) {
          future.cancel(true);
          futures.remove(evt);
          if (evt.getAttempt() < tokenRenewerThreadRetryMaxAttempts) {
            renewalTimer.schedule(
                getTimerTask((AbstractDelegationTokenRenewerAppEvent) evt),
                tokenRenewerThreadRetryInterval);
          } else {
            LOG.info(
                "Exhausted max retry attempts {} in token renewer "
                    + "thread for {}",
                tokenRenewerThreadRetryMaxAttempts, evt.getApplicationId());
          }
        }
      } catch (Exception e) {
        LOG.info("Problem in submitting renew tasks in token renewer "
            + "thread.", e);
      }
      // remove done and cancelled task
      if (future.isDone() || future.isCancelled()) {
        try {
          futures.remove(evt);
          LOG.info("Removed done or cancelled renew tasks of {} in token renewer thread.", evt.getApplicationId());
        } catch (Exception e) {
          LOG.warn("Problem in removing done or cancelled renew tasks in token renewer thread.", e);
        }
      }
    }
  }
}

some screenshots and CPU profile as following:

all screenshots images and CPU profile report HTML files were attached into issue: YARN-11178

before optimized the ACTIVE ResourceManager process will occupy CPU core 100% continuous：

after optimized the code, CPU occupation decreased to normal：

for CPU profile, before optimized:

after optimized:

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 42s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
38m 36s

trunk passed

+1 :green_heart:
compile
1m 25s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
1m 11s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m  5s

trunk passed

+1 :green_heart:
mvnsite
1m 18s

trunk passed

+1 :green_heart:
javadoc
1m 16s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
1m  1s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
2m 26s

trunk passed

+1 :green_heart:
shadedclient
21m 56s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m  2s

the patch passed

+1 :green_heart:
compile
1m  9s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javac
1m  8s

the patch passed

+1 :green_heart:
compile
1m  0s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
1m  0s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 50s
/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 33 unchanged - 0 fixed = 35 total (was 33)

+1 :green_heart:
mvnsite
1m  4s

the patch passed

+1 :green_heart:
javadoc
0m 54s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 46s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
2m  9s

the patch passed

+1 :green_heart:
shadedclient
21m 55s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
97m 39s
/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
hadoop-yarn-server-resourcemanager in the patch passed.

+1 :green_heart:
asflicense
0m 45s

The patch does not generate ASF License warnings.

199m 35s

Reason
Tests

Failed junit tests
hadoop.yarn.server.resourcemanager.TestClientRMTokens

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4435

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux b5f71f736421 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 9db8110f450a23449e4f37fdb0e89c51623b7b19

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/1/testReport/

Max. process+thread count
989 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@LennonChin I ask a question, Why do you need to add the synchronized keyword?

@LennonChin I ask a question, Why do you need to add the synchronized keyword?

@slfan1989  Hi, synchronized code block is needed for waiting operation, Object.wait() need a monitor object, otherwise it will throw java.lang.IllegalMonitorStateException.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 37s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
37m 40s

trunk passed

+1 :green_heart:
compile
1m 15s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
1m 18s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m 10s

trunk passed

+1 :green_heart:
mvnsite
1m 20s

trunk passed

+1 :green_heart:
javadoc
1m 15s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 57s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
2m 24s

trunk passed

+1 :green_heart:
shadedclient
22m 19s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m  3s

the patch passed

+1 :green_heart:
compile
1m 13s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javac
1m 13s

the patch passed

+1 :green_heart:
compile
0m 59s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
0m 59s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 50s

the patch passed

+1 :green_heart:
mvnsite
1m  0s

the patch passed

+1 :green_heart:
javadoc
0m 47s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 44s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
2m 14s

the patch passed

+1 :green_heart:
shadedclient
22m 25s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
98m  3s
/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
hadoop-yarn-server-resourcemanager in the patch passed.

+1 :green_heart:
asflicense
0m 51s

The patch does not generate ASF License warnings.

199m 43s

Reason
Tests

Failed junit tests
hadoop.yarn.server.resourcemanager.TestClientRMTokens

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4435

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux e5d0d06d30cd 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / b72827c5f739634e704d6b512998cc2c25b0a5a7

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/2/testReport/

Max. process+thread count
959 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@slfan1989 @dineshchitlangia @brumi1024 @9uapaw @ashutoshcipher Could you please to help review the code?

@slfan1989 @dineshchitlangia @brumi1024 @9uapaw @ashutoshcipher Could you please to help review the code?

I have understood your changes, but can the token also add some user information?

@slfan1989 @dineshchitlangia @brumi1024 @9uapaw @ashutoshcipher Could you please to help review the code?

I understand your change, and it seems feasible, can we give the data of the cpu before and after the pr change？

@slfan1989 @dineshchitlangia @brumi1024 @9uapaw @ashutoshcipher Could you please to help review the code?

I understand your change, and it seems feasible, can we give the data of the cpu before and after the pr change？

Hello, I added some screenshots and CPU profile as following:

all screenshots images and CPU profile report HTML files were attached into issue: YARN-11178

before optimized the ACTIVE ResourceManager process will occupy CPU core 100% continuous：

after optimized the code, CPU occupation decreased to normal：

for CPU profile, before optimized:

after optimized:

@LennonChin  LGTM. Thanks for your contribution, it looks like it's working well, let's wait for other buddies' suggestions.

I can reproduce the issue and I would like to know the current status of this PR. Thanks.

@slfan1989
I'm using Hadoop 3.3.4 with kerberos, and the issue is happened, the DelegationTokenRenewer use 100% CPU.
I use jstack to check threads and find the DelegationTokenRenewer thread cause this issue, like @LennonChin said.

The issue still exists but the PR is not being merged, I am not quite sure why it is not merged, you can manually merge this patch to solve this problem.

@slfan1989
I'm using Hadoop 3.3.4 with kerberos, and the issue is happened, the DelegationTokenRenewer use 100% CPU.
I use jstack to check threads and find the DelegationTokenRenewer thread cause this issue, like @LennonChin said.

The issue still exists but the PR is not being merged, I am not quite sure why it is not merged, you can manually merge this patch to solve this problem.
After merging the patch, we have been running in the production environment for half a year, and no problems have been found so far.

Thank you for your work on this issue, I had added the empty "futures" checking code, it works well.

@slfan1989 @dineshchitlangia @brumi1024 @9uapaw @ashutoshcipher Could you please to help review the code?

This issue makes version 3.3.4 unable to run directly on the production environment without a manual patching。
Could I know the current review status of this PR and whether it can be incorporated into version 3.3.5?

I have made two commits to perform your suggestions,  @slfan1989 please cc

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 36s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 58s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
25m 48s

trunk passed

+1 :green_heart:
compile
9m 47s

trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04

+1 :green_heart:
compile
8m 29s

trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
checkstyle
1m 48s

trunk passed

+1 :green_heart:
mvnsite
3m 20s

trunk passed

-1 :x:
javadoc
1m  2s
/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt
hadoop-yarn-server-resourcemanager in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.

+1 :green_heart:
javadoc
2m 47s

trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
spotbugs
6m 12s

trunk passed

+1 :green_heart:
shadedclient
21m 27s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 28s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 11s

the patch passed

+1 :green_heart:
compile
9m 43s

the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04

+1 :green_heart:
javac
9m 43s

the patch passed

+1 :green_heart:
compile
8m 34s

the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
javac
8m 34s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m 39s

the patch passed

+1 :green_heart:
mvnsite
3m  1s

the patch passed

-1 :x:
javadoc
0m 56s
/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt
hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.

+1 :green_heart:
javadoc
2m 34s

the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
spotbugs
6m 23s

the patch passed

+1 :green_heart:
shadedclient
21m 35s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
1m 11s

hadoop-yarn-api in the patch passed.

+1 :green_heart:
unit
5m 44s

hadoop-yarn-common in the patch passed.

-1 :x:
unit
99m 49s
/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
hadoop-yarn-server-resourcemanager in the patch passed.

+1 :green_heart:
asflicense
0m 57s

The patch does not generate ASF License warnings.

266m 11s

Reason
Tests

Failed junit tests
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4435

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint

uname
Linux a20d734265b5 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / ef0d71469bfe38fd23bc6573a377f91fbb442435

Default Java
Private Build-1.8.0_352-8u352-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/3/testReport/

Max. process+thread count
945 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 39s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
15m  8s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
26m  8s

trunk passed

+1 :green_heart:
compile
10m  5s

trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04

+1 :green_heart:
compile
9m 16s

trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
checkstyle
1m 45s

trunk passed

+1 :green_heart:
mvnsite
3m 13s

trunk passed

-1 :x:
javadoc
1m  1s
/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt
hadoop-yarn-server-resourcemanager in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.

+1 :green_heart:
javadoc
2m 35s

trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
spotbugs
6m 12s

trunk passed

+1 :green_heart:
shadedclient
21m 53s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 31s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 19s

the patch passed

+1 :green_heart:
compile
9m 39s

the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04

+1 :green_heart:
javac
9m 39s

the patch passed

+1 :green_heart:
compile
8m 35s

the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
javac
8m 35s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m 35s

the patch passed

+1 :green_heart:
mvnsite
2m 56s

the patch passed

-1 :x:
javadoc
0m 56s
/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt
hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.

+1 :green_heart:
javadoc
2m 32s

the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
spotbugs
6m 18s

the patch passed

+1 :green_heart:
shadedclient
22m 15s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
1m 12s

hadoop-yarn-api in the patch passed.

+1 :green_heart:
unit
5m 43s

hadoop-yarn-common in the patch passed.

-1 :x:
unit
102m 26s
/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
hadoop-yarn-server-resourcemanager in the patch passed.

+1 :green_heart:
asflicense
0m 56s

The patch does not generate ASF License warnings.

270m 53s

Reason
Tests

Failed junit tests
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4435

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint

uname
Linux f5e8a8b302d5 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 84d4d555f62e29fdc19947869d02ffd33b9ac4c5

Default Java
Private Build-1.8.0_352-8u352-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/4/testReport/

Max. process+thread count
950 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/4/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@LennonChin Thank you for your contribution! Is the unit test error caused by this pr?

@LennonChin Is this junit test error related to this pr?

@LennonChin Is this junit test error related to this pr?

Looks like it is, I'll check it out.
I'm a little busy lately, sorry...

I had the same problem. hadoop 3.3.5 was released and had the same problem

I had the same problem. hadoop 3.3.5 was released and had the same problem

We will continue to follow up this pr. @LennonChin Can we rebuild this pr?

TestDelegationTokenRenewer. TestTokenThreadTimeout need RM_DT_RENEWER_THREAD_IDLE_BACKOFF_MS is set to less than RM_DT_RENEWER_THREAD_TIMEOUT

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
35m 39s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
15m 44s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
26m 33s

trunk passed

+1 :green_heart:
compile
8m 57s

trunk passed

+1 :green_heart:
checkstyle
1m 52s

trunk passed

+1 :green_heart:
mvnsite
3m 25s

trunk passed

+1 :green_heart:
javadoc
3m  1s

trunk passed

+1 :green_heart:
spotbugs
6m 25s

trunk passed

+1 :green_heart:
shadedclient
21m 35s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 29s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 10s

the patch passed

+1 :green_heart:
compile
8m 17s

the patch passed

+1 :green_heart:
javac
8m 17s

the patch passed

+1 :green_heart:
blanks
0m  1s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m 41s

the patch passed

+1 :green_heart:
mvnsite
3m  7s

the patch passed

+1 :green_heart:
xmllint
0m  0s

No new issues.

+1 :green_heart:
javadoc
2m 32s

the patch passed

+1 :green_heart:
spotbugs
6m 29s

the patch passed

+1 :green_heart:
shadedclient
21m 40s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
1m 16s

hadoop-yarn-api in the patch passed.

+1 :green_heart:
unit
5m 50s

hadoop-yarn-common in the patch passed.

-1 :x:
unit
100m 27s
/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
hadoop-yarn-server-resourcemanager in the patch passed.

+1 :green_heart:
asflicense
0m 59s

The patch does not generate ASF License warnings.

278m 47s

Reason
Tests

Failed junit tests
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer

Subsystem
Report/Notes

Docker
ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4435

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint

uname
Linux 18558f9ceeee 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 84d4d555f62e29fdc19947869d02ffd33b9ac4c5

Default Java
Red Hat, Inc.-1.8.0_362-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/5/testReport/

Max. process+thread count
943 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/5/console

versions
git=2.9.5 maven=3.6.3 spotbugs=4.2.2 xmllint=20901

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Thanks to LennonChin for his contributions and ideas, but why close this commit? I don't think the problem has been fixed. Can I continue the work?

As already mentioned, this issue make 1vcpu to be always occupied due to busy loop. The makes the adoption of 3.3 line difficult in production environments. Can it be fixed?

This bug seems to have been solved by https://github.com/apache/hadoop/pull/5629 (https://issues.apache.org/jira/browse/YARN-11489), after which the futures are held in LinkedBlockingQueue. Elements are taken from the list using take method, that is documented like following:

Retrieves and removes the head of this queue, waiting if necessary until an element becomes available.

I confirmed in our environments that after patching 3.3.6 with this fix the CPU utilization is back to normal. I believe that https://issues.apache.org/jira/browse/YARN-11178 can be closed.


[PUB-1] community.namespace.so: The initial connection between Cloudflare's network and the origin web server timed out.

Following link in ns (installed via homebrew)

PUB-1

If I visit https://community.namespace.so/discord/ I see the same message; checking

I've fixed the 302 for /discord/(note forward slash); but this is a bit too complicated on our side. Filed NSL-1166 for a follow up.


Automatically run earthaccess.login()?

I've run into a few times where either I, or a user I was working with, forgot to run earthaccess.login() and get an error like
File "/opt/coiled/env/lib/python3.10/site-packages/earthaccess/api.py", line 184, in open
    results = earthaccess.__store__.open(granules=granules, provider=provider)

AttributeError("'NoneType' object has no attribute 'open'")

I think a better user experience would be to automatically try earthaccess.login() with the netrc and environment strategies. That way, at least for users who have a .netrc file or have intentionally set the right environment variables, earthaccess.login() isn't needed. This removes the "I forgot to run earthaccess.login()" issue and cuts down on boilerplate code (at least by 1 line 🙂).
@betolink @MattF-NSIDC I'm curious what you think about this

Sounds like a good idea, are you thinking on executing it at import time? I'm planning on working on a PR for this paired with "too verbose" output that you mentioned on #283

are you thinking on executing it at import time?

My thinking is avoiding a long import time would be nice, as along as the code needed to do things on-demand isn't super complicated

I'm planning on working on a PR for this paired with "too verbose" output that you mentioned on https://github.com/nsidc/earthaccess/issues/283

Ah, that'd be great. FWIW I played around a little with the automatic login stuff already. This seems to work for on-demand login:
diff --git a/earthaccess/__init__.py b/earthaccess/__init__.py
index ef4fdfb..dd248b0 100644
--- a/earthaccess/__init__.py
+++ b/earthaccess/__init__.py
@@ -36,7 +36,28 @@ __all__ = [
     "Store",
 ]

-__auth__ = Auth()
-__store__: Any = None
-
 __version__ = version("earthaccess")
+
+_auth = None
+_store = None
+
+def __getattr__(name):
+
+    global _auth, _store
+
+    if name == "__auth__" or name == "__store__":
+        if _auth is None or not _auth.authenticated:
+            for strategy in ["environment", "netrc"]:
+                try:
+                    _auth = Auth().login(strategy=strategy)
+                except Exception:
+                    continue
+                else:
+                    if not _auth.authenticated:
+                        continue
+                    else:
+                        _store = Store(_auth)
+                        break
+        return _auth if name == "__auth__" else _store
+    else:
+        raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

What do you think?
To be clear, that diff ^ doesn't reduce any verbosity during the login process

:heart:  I like this idea. If we don't do automatic login, we should at least make that error message better :)

Nice! @jrbourbeau if you start a PR with this it would be great, reducing the verbosity should be trivial.

also, what do you think @JessicaS11? I don't think this should disrupt icepyx


About info layout

Issue
About info is compact and not showing properly

[x] Fixed

Fixed on commit: d6860aae2851dcdeda2d5885487b9e477bcfcdc8


Recommend any jQuery tree for represent getFoldertree

I would like to represent my folder descendant with jQuery tree in a php page.
What javascript tree recommend for use it with this library?
Thanks.

Hi. This is a issue tracker and not a support forum.

@aivdev That question is not related to the cmis client anyway.


UX is very difficult to use

Hi, guys!
Thanks for publish new version
But there are some features, which is not very convenient to use:

no tip of pakage in search during searching function
The text size is too big, information is spread on the page
function below package description are in 3 rows in quite big tabs, which is difficult to focus

Sorry, but I used the site a lot, now will switch to rdrr.io, dissapointed

Thank you for your feedback. I think you have a good point regarding the font size. I just merged a PR (#62) to decrease it from 16px to 14px. Other UX improvements might take more time to implement.

The new UX looks nice, but functionally is bad. I prefer to have all of the functions listed in a single column (easier to view) that could be sorted. Also, it's now not possible to open a link in a new tab (right click, open link in new tab).


/reload command

when send /reload command on any chat bot are clearing admins dict for any groups.

It is /admincache

Sorry, i renamed this command on my private repository

I merged pull request


Fix docs

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 52.396%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
503

Relevant Lines:
960

💛  - Coveralls


fix: use proper variable names

E2E integrations require those variables to contain the string "SERVICE" to work. This was missing in two of the release service variables.

/retest

/retest


Convert grpc to alluxio exceptions in polling ping

We retry alluxio.exception.status.UnavailableException, but not the gRPC StatusRuntimeException equivalent. This converts the gRPC exception to an AlluxioStatusException so that we can properly retry when the status is Unavailable

I can see this becoming a popular type of issues. @ggezer Is it possible we provide some pattern to automatically convert the exception or make it harder to forget about the conversion?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/2009/
Test PASSed.

StatusRuntimeException should have been converted to AlluxioStatusException at the AbstractClient.
@aaudiber, Do you have a stack-trace that shows otherwise?

@aaudiber, I see the ::connect() call in AbstractClient::retryRpc is not under try-catch. Do you think catching all exceptions in the below catch block in ::connect() would make it work?
...
...
} catch (IOException e) {
LOG.warn("Failed to connect ({}) with {} @ {}: {}", retryPolicy.getAttemptCount(),
getServiceName(), mAddress, e.getMessage());
}

I can see this becoming a popular type of issues. @ggezer Is it possible we provide some pattern to automatically convert the exception or make it harder to forget about the conversion?

@bf8086, @aaudiber suggested implementing an exception handling client that we can wrap over naked service stubs. Which could save us from future instances of such issues as long as everyone obeys and don't use a naked stub.
Such work could be impactful so I'm fine letting this one go as it is, and we can open an issue to track that.


Attempt again to reproduce #4264.

This time, use integer timestamps for the entity keys.
The new system test passes.
Closes #4264.

Rebased to pick up the BQ systest fix from #5400.


A after hook to change the received value

It would be nice to have some sort of an after hook added to promptui.Prompt{} struct so we can for example cast a received string to a number. So some sort of value a transformer:
AfterHook: func(input string) (error, uint64) {
	transformed, err := strconv.ParseUint(input, 10, 64)
	if err != nil {
		return fmt.Errorf("failed transforming received input to uint64: %v", input), 0
	}
	return nil, transformed
}

Or does this not fall into what promptui should do?

I'd quite like this; I'm facing a scenario where I'd like to downcase strings. Right now I need to do this in my validate function and in whatever receives this input.


Idle notifications unfocused window

Is your feature request related to a problem? Please describe.
I wouldn't consider it as a problem precisely but huge inconvenience sometimes. As I imagine a lot of people use idle notifications, and it seems to be to unfocus the window in order to get them. I and I imagine quite a few other people don't tend to click outside RuneLite window while watching youtube or movies much.
Describe the solution you'd like
A clear and concise description of what you want to happen.
I would like a tick box that says "notify while focused" which would on top of getting notifications unfocused would also add them if runelite is the window you're focused on.
Describe alternatives you've considered
Clicking outside runelite window to unfocus it, but human errors such as forgetting to do so end up happening which is why I'm requesting this.

Oh, I didn’t see it cause I thought it would be in the idle notifications category I guess, thanks for letting me know


On branch edburns-msft-ibm-7891-support-pre-pushed-acr-image WIP: support deployment of pre-pushed image to pre-existing Azure Container Registry, such as ejb02ibm781.azurecr.io/jakartaee-microprofile-example:v1

On branch edburns-msft-ibm-7891-support-pre-pushed-acr-image Skip inspecting the manifest if the image comes from Azure Container Registry.
modified:   pom.xml

Increment version.

modified:   src/main/scripts/preflight.sh

Assume that if SOURCE_IMAGE_PATH comes from azurecr.io it is assumed to meet the requirements of being a valid container image for running Liberty on Kubernetes.

modified:   src/main/bicep/mainTemplate.bicep

Fix issue where fully qualified azurecr.io tags were incorrectly prepended with docker.io.

Testing
Verified the following 6 test cases with preview offer:

ACR
app docker image
expected deployment status

ejb02ibm781.azurecr.io
ejb02ibm781.azurecr.io/jakartaee-microprofile-example:v1
Succeeded

create new
icr.io/appcafe/open-liberty/samples/getting-started
Succeeded

acr9dadaa.azurecr.io
icr.io/appcafe/open-liberty/samples/getting-started
Succeeded

new
ejb02ibm781.azurecr.io/jakartaee-microprofile-example:v1
Failed

acr9dadaa.azurecr.io
ejb02ibm781.azurecr.io/jakartaee-microprofile-example:v1
Failed

ejb02ibm781.azurecr.io
ejb02ibm781.azurecr.io/jakartaee-microprofile-example:v1
Failed


Calculator.py

It is what you would think it to be: A calculator .   2 numbers, 5 operations
Relevant Issues

Description

Did you:

[ ] Join the Python Discord Community?
[ ] Read all the comments in this template?
[ ] Ensure there is an issue open, or link relevant discord discussions?
[ ] Read and agree to the contributing guidelines?

Duplicate of #807.
Hey! Once again thanks for your suggestions, but these projects are not something we can integrate into our bots.
The bots are discord.py bots which interact with discord, with no real place for standalone scripts such as this one.
Im going to close this issue, as well as #808 which has the same problem. Please do not open any more PRs without talking to the core devs about your ideas on server.
Thanks!


Add video, video-series, and videos components

Closes #10

[x] Create videos folder in /src/components
[x] Create  component in new videos folder
[x] Create  component in new videos folder
[x] Create  component in new videos folder
[x] Attach  component to path /videos in App.js
[x] Attach  component to path /videos/:id/series in App.js
[x] Attach  component to path /videos/:slug in App.js

Sure thing! Amending now. :)

Thanks! Glad to make my first contribution.


Detectron2 multinode run with slurm

Dear detectron users,
I'm using detectron2 in order to train a net using a custom dataset, according to coco format. Until the run is intranode it works properly. But I don't understand how to set and launch on multinode hpc cluster and Slurm scheduler, in particular the dist_url flag in launch command:
``  launch(main,
           num_machines = 2,
           num_gpus_per_machine = 4,
           dist_url = ???)

The "auto" options is not accepted when num_machines is more than 1 , and if dist_url is missing, I get the following error:

``    if num_machines > 1 and dist_url.startswith("file://"):
AttributeError: 'NoneType' object has no attribute 'startswith'

The following is my Slurm script:
#!/bin/bash

#SBATCH --time=00:10:00
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=8
#SBATCH --exclusive
#SBATCH -N 2
#SBATCH --gres=gpu:4

srun python3.8 ./train_multinode.py

So, which is the right way to set and launch detectron2 in multinode way and Slurm scheduler? I think Slurm itself should pass the url of the nodes  to detectron in some manner, like an MPI application.
I'm not sure also about ntasks-per-node and cpus-per-task. Are there properly set? Is it not clear how many cores detectron uses during a training session. Thanks.

Closing due to inactivity, please reopen if there are any further problems.

Hi, I met the same issues with you, have you solved it?


[Request] Include release year

Hello. I have this website bookmarked and visit it every day. It would be amazing if the lists included the release year of the game. Because now I'm pretty much clicking on the links to load the game page and see the release year.
E.g. when it's -80% off for a game released in 2000 or 2015 (or in between) it's really important.
Also adding some icon indicating that the game has Steam trade cards also is important reason to buy the game, especially when it's on sale.
Cheers

Games already show support for trading cards, in their header here: https://steamdb.info/app/570/, the image is https://steamstore-a.akamaihd.net/public/images/v6/ico/ico_cards.png

Thanks for reply. I meant in /sales/ page.


Metrics for managed PV and LV

What should the feature do:
Metrics for:

Nr. of TopoLVM managed pv
Nr. of LogicalVolumes

What is use case behind this feature:
If the retention policy is Retain and a user deletes a pvc/pv the corresponding LogicalVolume is not deleted, this way we can detect lingering LVs on the cluster.

Good. If you submit a PR, we are happy to review it.


SSLError

When sending test data to coveralls, from the CLI (using bundle exec coveralls push), the following error is raised:
Coveralls encountered an exception:
OpenSSL::SSL::SSLError
SSL_connect returned=1 errno=0 state=error: no protocols available

Version installed: 0.17.0
Rails version: <IP_ADDRESS>
Ruby version: 2.6.5

@sylvainbx thanks for the PR
Could you please also give a try to this branch?  chore/refactor-http-client
I've switched to auto negotiation and removed some code that I think it was related to Ruby < 2.0

Of course!
I've just made a test using the chore/refactor-http-client branch, as you asked, and it worked fine.

@sylvainbx thanks! I'm going to merge and release a new version of this gem

0.18.0 released


fix: [do not merge] test PR to see if codeowners file does its thing

Closing this PR as the CODEOWNERS set up is working as expected! 🎉


Why the way of getting coco train dataset  is using "build_dataset" rather than "build_selfdet"?

The coco dataset is intended for supervised finetuning whereas the coco_pretrain is intended for DETReg pretraining.


'org-roam-node-buffer' is not a valid language name

Encountered in neovim 0.10 when attempting to expand a backlink. Must be something I missed when adding support for neovim 0.10. I'm assuming they changed how creating a parser works. I'm assuming that I'm setting the filetype to this somewhere, and neovim is trying to do virtual identation, a feature new to neovim 0.10 that I have enabled. Not sure why it's happening for a non-org buffer, though.
Error executing vim.schedule lua callback: ....10.0/share/nvim/runtime/lua/vim/treesitter/language.lua:101: 'org-r
oam-node-buffer' is not a valid language name
stack traceback:
        [C]: in function 'error'
        ....10.0/share/nvim/runtime/lua/vim/treesitter/language.lua:101: in function 'add'
        ...0/share/nvim/runtime/lua/vim/treesitter/languagetree.lua:111: in function 'new'
        .../neovim/0.10.0/share/nvim/runtime/lua/vim/treesitter.lua:41: in function '_create_parser'
        .../neovim/0.10.0/share/nvim/runtime/lua/vim/treesitter.lua:108: in function 'get_parser'
        .../neovim/0.10.0/share/nvim/runtime/lua/vim/treesitter.lua:386: in function 'get_node_at_cursor'
        .../nvim/lazy/orgmode/lua/orgmode/utils/treesitter/init.lua:51: in function 'closest_headline_node'
        ...hare/nvim/lazy/orgmode/lua/orgmode/ui/virtual_indent.lua:86: in function 'set_indent'
        ...hare/nvim/lazy/orgmode/lua/orgmode/ui/virtual_indent.lua:154: in function <...hare/nvim/lazy/orgmode/lu
a/orgmode/ui/virtual_indent.lua:153>

The problem seems to be this line: https://github.com/nvim-orgmode/orgmode/blob/e365b85b4a5bfea507d77a16a7b66d8c6860e9b7/lua/orgmode/utils/treesitter/init.lua#L21:
function M.get_node_at_cursor(cursor)
  M.parse_current_file()
  if not cursor then
    return vim.treesitter.get_node()
  end

  return vim.treesitter.get_node({
    bufnr = 0,
    pos = { cursor[1] - 1, cursor[2] },
  })
end

where orgmode calls vim.treesitter.get_node() twice without specifying the language. The behavior of get_node() in that case is to deduce the language from the filetype. For that it checks the table filled via vim.treesitter.language.register().
The way I see it, there are two ways to fix this:

in orgmode proper, we pass the language explicitly where it makes sense, eg: get_node({lang = 'org'})
in org-roam, we register our custom filetypes with the org language: vim.treesitter.language.register('org', {'org-roam-node-buffer'})

Personally, I think the first option makes more sense. I've submitted https://github.com/nvim-orgmode/orgmode/pull/795 so if and when that PR is accepted, this should work without updates to org-roam itself.

The fix just got merged, so the error should no longer appear after an update!

Can confirm that as of https://github.com/nvim-orgmode/orgmode/commit/56c8246c633bcde845cefc2090979ec23bd9034e, this has been fixed. Thanks @troiganto for getting this resolved upstream!


Increase CameraMoveTo speed and fix lock to player speed

Please check if the PR fulfills these requirements

[x] The commit message follows our guidelines
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

What kind of change does this PR introduce? (Bug fix, feature.)
Changes an oversight where the fastest camera move to speed was every second frame.
Also fixes Camera : lock to player only supporting instant, or every frame.

What is the current behavior?
Code is calculated in ScriptBuilder.js,
6th bit set for transitioning,
low 4 bits are masked, then bitwise & against time binary, if=0, moves cam 1px.
Speed 0 (instant), value 0 (instant)binary 0000-0000
Speed 1 (Faster), 32+1 (every 2)   binary 1000-0100
Speed 2, 32+ 3    ( move every 4 )   binary 1100-0100
Speed 3, 32+ 7   (every 8 frames)  binary 1110-0100
Speed 4, 32+15 (every 16 frame)  binary 1111-0100
Speed 5, 32+31 (every 16 frame)  binary 1111-1100

What is the new behavior (if this is a feature change)?
Speed 0 (instant) value 0 (instant) binary 0000-0000
Speed 1, 32 + 0 (every 1  frame )    binary 0000-0100
Speed 2 32 + 1 (every 2 frames)    binary 1000-0100
Speed 3 32+ 3 (every 4 frames)    binary 1100-0100
Speed 4 32+ 7 (every 8 frames)    binary 1110-0100
Speed 5 32+15 (every 16 frame)  binary 1111-0100
Also adds camera_speed = (UBYTE)script_cmd_args[0] & CAMERA_SPEED_MASK; to Script_CameraLock_b

Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
May need migration,
If you want to keep old speeds for camera move, Instant and speed 5 no change,
speed 1 is equivalent speed 2 and so on.
Camera lock to player was always instant, or every frame, so no change, users will have more options.

Other information:
Accidentally includes static multiframe actor fix Commit.

Awesome! Thanks for this, as suggested I've added a migration for this change and added some test cases for the migration.


Error with define_clonotypes on small datasets

Dear scirpy team, I'm reporting an issue that has popped up a few times in our repo mvTCR: here and here
Describe the bug
We've been encountering a TypeError: iteration over a 0-d array error when using scirpy.tl.define_clonotypes on smaller datasets. The error seems to stem from how clonotype distances are calculated and aggregated under certain conditions.
To Reproduce
Additional information such as full error codes can be found in the referenced issues above. If you want to directly reproduce:

conda create --name mvTCR python=3.10 -y && conda activate mvTCR && pip install mvtcr==0.2.1 && conda install nb_conda_kernels -y (important that mvtcr v0.2.1 is used because there is a fix in the newest version)
Download and follow the tutorials 00_10x_data_preprocessing.ipynb to generate the data and 01_preprocessing.ipynb to encounter the bug.
In tutorial 01_preprocessing in cell 3 switch selected_patients = ['AP1', 'CV0062'] to selected_patients = ['AP1'] (downsampling to only one patient aka a small dataset)
Follow along and cell 11 should then fail with the error as well as cell 13 (which do the same thing)

In one of our issues someone encountered the same issue on a different dataset.
Additional context/information
The error occurs when the dataset for computing clonotypes is too small. Specifically, under this condition:
if self.n_jobs == 1 or n_clonotypes <= 2 * self.chunksize (your_env\Lib\site-packages\scirpy\ir_dist\ _clonotype_neighbors.py line 213, see error traces in the issues on how to get there)
If this condition is true, scirpy calculates distances for clonotypes in a single process and returns them as a generator object, which cannot be stacked by scipy.sparse.vstack. This results in the error: TypeError: iteration over a 0-d array.
However, when the dataset (n_clonotypes) is large enough and the condition is false, distances are computed using multi-processing and returned as a list, which can be properly stacked by scipy.sparse.vstack.
For me, it worked when casting the row distances as a list again:
dist_rows = tqdm(                 list(self._dist_for_clonotype(i) for i in range(n_clonotypes)),                 total=n_clonotypes,             )
(row 214-17)
As a temporary hack, we set the chunksize to 1 in the define_clonotypes function to force the condition to evaluate as false, bypassing the bug even when working with small datasets. However, it would be great if you could verify whether this is indeed a bug on your end and potentially roll out a fix.
Cheers & thanks,
Jan

Thanks for the detailled analysis! I'll try to patch this within the next week.

Hi Jan,
where can I find the data from your preprocessing tutorial?
path_base = '../data/10x_CD8TC/raw/'

Which version of scirpy are you testing? I tried looking into the code section you are describing, but both branches are covered by test cases and the result is explicitly casted to a list before it is vstacked:
dist = sp.vstack(list(dist_rows))

Hi thanks for the quick reply! The dataset can be downloaded from the 10x website under and the keywords "a new way immune profiling".
We are still running on scirpy v0.13.1, where this is an issue, but as you pointed out, it is not with the current version. Sorry to steal your time in this manner...

No worries :)


Alright v2

I just added a second @font-face for the  'Alright-v2-Normal-Regular' that tells the browser to use the Alright-v2-Normal-Bold.woff2 for bold weights.
This way I won't have to rewrite the CSS heading styles. Many of our 'bold' text comes from the default bold stylings on h1, h2. I would have to make a reset to stop the system from doing that. I really don't know which way is more semantically correct, but at least this way, nothing should change from how it was before.
"Ready For Review" checklist

[ ] PR title accurately summarizes changes
[ ] PR description

Does this fix any open issues?
What changes are you making?
Why did you implement it this way?
Is there anything reviewers need to know or pay attention to?
Does anything special need to happen for deployment?

[ ] Optional: Inline github PR comments explaining context of tricky / complicated / unexpected lines so reviewers can follow along.
[ ] If appropriate, new tests were added for isolated methods or new endpoints
[ ] All tests green
[ ] Percy changes are purposeful or explained
[ ] I booted up the branch locally, exercised any new code I wrote.
[ ] If css was touched, I verified changes are happy on mobile (via Percy is ok)
[ ] I opened an issue for any logical followups

@ofsound Looking at Percy, it looks like the black font is having a bit of trouble. Maybe it's being double bolded? Maybe we should include it with the regular and bold as weight 700/800?


(fix) semantic scanner abstraction

Pull Request Test Coverage Report for Build<PHONE_NUMBER>1
Details

35 of 44 (79.55%)  changed or added relevant lines in 2 files are covered.
5 unchanged lines in 1 file lost coverage.
Overall coverage increased (+1.9%) to 31.235%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

scanner/sentences.go
34
43
79.07%

Files with Coverage Reduction
New Missed Lines
%

scanner/scanner.go
5
83.33%

Totals

Change from base Build<PHONE_NUMBER>8:
1.9%

Covered Lines:
134

Relevant Lines:
429

💛  - Coveralls


readme ansible role type fix.  change certain default values

@kaijparo MONIT-6719, I have added the flag to finish the yum-complete-transaction.


Allow passing a JSONEncoder and JSONDecoder

The python json module supports a custom JSONEncoder and JSONDecoder. This enables to serialize / deserialize custom data formats. Since requests just passes JSON data directly to json.dumps and json.loads, it is not possible to customize JSON serialization at all. As a result one must configure configure the JSON request manually, meaning the content type needs to be set manually.
I suggest to add the ability to pass custom json_encoder and json_decoder properties to requests. This handles pretty much all other arguments that could be passed to json.dumps and json.loads as well.
This example:
response = requests.post(
    'https://www.example.com',
    content_type='application/json',
    data=json.dumps(data, cls=MyEncoder))
result = json.loads(response.text, cls=MyDecoder)

would become:
response = requests.post(
    'https://www.example.com',
    json=data,
    json_encoder=MyEncoder,
    json_decoder=MyDecoder)
result = response.json()

I could even imagine that it's useful to pass these arguments to the session constructor, because data from one session might often contain the same types of data serialization.
session = requests.Session(json_encoder=MyEncoder, json_decoder=MyDecoder)

response = session.post('https://www.example.com', json=data)
result = response.json()

Thanks for this suggestion!
It's been provided before and was rejected (see #2755): the same argument applies here, I'm afraid. This is just too minor in utility to justify adding the extra kwargs: json= triggers only a tiny amount of code that can easily be replicated in your own application. :smile:

Agreed!

@Lukasa I have to disagree. It is probably simple workaround for request JSON. But what about response JSON? requests  uses built-in decoder and if you want later inspect and transform that JSON response - it will require recursively analyze and convert them (in case of complex nested json objects) - what is definitely not what we want .
Consider following scenario:

POST JSON object to server, which contains strings with zoned datetime
Database does know nothing about timezone of user and convert to internal representation (UTC)
GET  JSON object from server, zoned datetime will be the same, but with different string representation
Compare that two JSON objects are identical
Step 4 will require some kind of normalization of JSONs - for example convert strings to datetime or  Arrow object and compare that objects.
If requests doesn't provide custom decoders - user have recursively (and safe, without stack overflow) walk over JSON tree structure (same job done twice).
I think it greatly complicates things.

@likern Not really. Just use response.content instead of response.json() and pass to your own custom decoder instead. One extra line of code. Doesn't seem to be a great increase in complexity to me. :wink:


Single-app automation script updated

The single app automation script is updated to clear app data + cache after every capture.

Looks good. I have a feeling that we might be running too many tests, but more data is always better.

@n-aggarwal The fact that "save permissions" also deletes the data isn't clear. It would be better to move the clearing of data out of the function.


Add versioning to tools

Each extension should be available at roam.davidvargas.me/1.0.0/extension.js. master defaults to latest

This is actually live!


fixed indexing issues with masked meshes

The skeleton quality function didn't handle indices on masked meshes correctly. I updated the function in a few ways:

Added default values that exist for a skeleton created from a default use of skeletonize.skeletonize_mesh
Expect the skeleton to know about unmasked mesh indices, and use the existing mesh to do the filtering. This uses the idea that the skeleton should always use unmasked indices to not need to care about the exact filters used. This is able to be toggled off, in case someone did something odd with the skeletonization, although it's hard to see what that would be.
Improved (existant!) docstring explaining arguments.
Adding warning that a non-default distance normalization needs a different likelihood ratio/bin value.

Codecov Report

Merging #28 into fcc_skel_qual will decrease coverage by 0.44%.
The diff coverage is 0%.

@@                Coverage Diff                @@
##           fcc_skel_qual      #28      +/-   ##
=================================================
- Coverage          67.02%   66.58%   -0.45%
=================================================
  Files                 13       13
  Lines               1929     1942      +13
=================================================
  Hits                1293     1293
- Misses               636      649      +13

Impacted Files
Coverage Δ

meshparty/skeleton_quality.py
0% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fce2352...a10c792. Read the comment docs.

can you take a stab at having a test on this?

I added a simple skeleton quality check by running it on the test mesh and the test skeleton. To get everything in sync, I updated the test skeleton to be generated by a current version of skeletonize_mesh and updated various skeleton tests as needed for the new value. Because I generated the skeleton with the link_edges, I also saved the link edges to a npy binary file and added a fixture for getting the existing test mesh with correct link edges.

Lots of bug fixes more broadly, pivoting this to being a bug-fixing branch in general.


Caret isn't visible when user first click the text area

When I click on the designated editable area, the flashing caret that you usually get when a textarea gets focus is not visible in the editable area. Only when I start typing a letter the caret appears. So the problem is that when the user clicks the designated text area with his mouse, it looks like the text area loses focus because there is not visual cue showing the user that he can start typing text.

Chrome latest version
Windows 7
1920x1080

Hi, it's just a guess, but is your textarea inside a html label? And if so, did you try to move it out of the label container?

Thank you!!! that saved me a lot of time!

I can also confirm this was an issue we were having and it was due to the textarea being wrapped in a label.

So you should use:
<label for="my-textarea">...</label><textarea id="my-textarea" class="editor"></textarea>


UVI Request

--- UVI JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from v4.14 to before v4.19.198",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=0c3fc4d5fa26092853278145aca9b21fa52a3e93",
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=04bd77ef4f4d9fc6102023b85f4590fc2130aac5"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "0c3fc4d5fa26092853278145aca9b21fa52a3e93",
      "note": "introduced"
    },
    {
      "type": "commit",
      "value": "04bd77ef4f4d9fc6102023b85f4590fc2130aac5",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "coresight: tmc-etf: Fix global-out-of-bounds in tmc_update_etf_buffer()\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v4.19.198 by commit 04bd77ef4f4d9fc6102023b85f4590fc2130aac5, it was introduced in version v4.14 by commit 0c3fc4d5fa26092853278145aca9b21fa52a3e93. For more details please see the references link."
}
--- UVI JSON ---

This issue has been assigned UVI-2021-1001395


chore(plugins): Update dropdown control for BigNumber with Time Comparison range

SUMMARY
We are making the chart controls more usable and thus we are renaming some options and adding some information that would help users to understand the meaning of the Range for Comparison dropdown.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
https://github.com/apache/superset/assets/38889534/ddfd1cc3-26fc-4ac9-8c10-7ca7a292193d
TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

/testenv up FEATURE_CHART_PLUGINS_EXPERIMENTAL=true

🏷 preset:2024.7


Fix CAS toggle broken by retained render world

Objective
Fixes #16531
I also added change detection when creating the pipeline, which technically isn't needed but it felt weird leaving it as is.
Solution
Remove the pipeline if CAS is disabled. The uniform was already being removed, which caused flickering / weirdness.
Testing
Tested the anti_alias example by toggling CAS a bunch on/off.

Would that work correctly if the CasUniform component is removed and re-added in the same frame?

Would that work correctly if the CasUniform component is removed and re-added in the same frame?

No, that would break. But that can't happen with ExtractComponentPlugin afaik, so you'd have to add a custom extraction or render system just to break it.
I think it just works if we run removals first, because in the "added then removed" case the query would be empty.


GeeksforGeeks Clone Project

Title and Issue number

Close #<issue_no>

Checklist:

[] I have mentioned the issue number in my Pull Request.
[] I have commented my code, particularly in hard-to-understand areas
[] I have created a helpful and easy to understand README.md

Title - Clone of GeeksforGeeks
Close Issue -> #3129


SSH Segment or SSH handling in Session

Prerequisites

[x] I have read and understand the CONTRIBUTING guide
[x] I looked for duplicate issues before submitting this one

Description
I'd like to add some special handling if detected running in an ssh session.
Would an ssh specific segment be best for this? Or would some properties on the hostname in the session segment be more appropriate.  I'm thinking like ssh-foreground, ssh-background, ssh-icon.
Environment

Oh my Posh version: 3.11.2
Theme: all
Operating System: all
Shell: all
Terminal: all

properties on the hostname in the session segment

I would opt for that as well. It's part of your session. Not sure of we should have color overrides specifically although It could make sense for foreground rather than background (might get messy and as soon as you're allowing partly overrides for all the colors a separate segment makes more sense imho).

Thanks!


Social Engineering Plan

Incomplete

What's incomplete about it? you said my social engineering campaign was great before? what else could i have even added to it?

From: ben-AI-cybersec @.>
Sent: Monday, 28 October 2024 5:46 PM
To: Redback-Operations/redback-cyber @.>
Cc: MattMina04 @.>; Author @.>
Subject: Re: [Redback-Operations/redback-cyber] Social Engineering Plan (PR #49)
Incomplete
—
Reply to this email directly, view it on GitHubhttps://github.com/Redback-Operations/redback-cyber/pull/49#issuecomment-2440685868, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMJ32JORHJ3OQ2W44QPLKTDZ5XMUXAVCNFSM6AAAAABPRP4ZQKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBQGY4DKOBWHA.
You are receiving this because you authored the thread.Message ID: @.***>


Add inventory

Show all the items collected and allow them to be interacted with, e.g. dropped, or potions mixed
Will need to decide how to show/hide inventory

Also show coins collected and current health

This has been dropped in favour of status bar


Rollback the metrics refactoring

In testing within the Netflix environment, it appears that 1.4.8 introduced a memory leak due to the refactoring around metrics collection.  This commit rolls back all of those changes (and those changes only) while I chase those down.

NetflixOSS » Hystrix » Hystrix-pull-requests #130 SUCCESS
This pull request looks good

NetflixOSS » Hystrix » Hystrix-pull-requests #131 SUCCESS
This pull request looks good


⚠ Buyfrom.io has degraded performance

In 6e36f7e, Buyfrom.io ($BUYFROM_IO_URL) experienced degraded performance:

HTTP code: 200
Response time: 5052 ms

Resolved: Buyfrom.io performance has improved in 13a3e43.


Making stunnel port configureable

Fixes #116
The new approach uses ConfigMap for stunnel config to replace the default configuration at /etc/stunnel/stunnel.conf in the container. This allows users to replace the default 2222 port with the use of stunnel_rsync_port var, in case the default port is pre-occupied by other services.

completed investigations on this, synced up with @pranavgaikwad earlier and dumping my finding here.
We use stunnel on source side in client mode which accepts connections on 2222 and forwards it to the route URL on destination port 2222. If the source side node has other services running on 2222, stage 3 will fail abruptly. We only need to make stunnel_port configurable on source side, as destination side networking will be handled by kube/openshift networking. However, to keep source and destination port consistent we'll make it configurable on both source and destination.
With above, this PR is ready to be merged.
cc @eriknelson @jwmatthews


Power BI URLs for Government

We receive many questions from our government customers about whitelisting URLs for GCC tenants. Can you provide info for government tenants in the doc.
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 297c1372-b1bb-57c4-9c31-840b4cad2c37
Version Independent ID: 0bb282a3-4aa6-1094-d8a4-652ec726734f
Content: Power BI URLs for whitelisting - Power BI
Content Source: powerbi-docs/admin/power-bi-whitelist-urls.md
Service: powerbi
Sub-service: powerbi-service
GitHub Login: @kfollis
Microsoft Alias: kfollis

Thanks for the comment, @haider392. There is a separate article for US Gov here: https://docs.microsoft.com/en-us/power-bi/admin/service-govus-overview#connect-government-and-global-azure-cloud-services that refers users to a downloadable file with IP ranges and service tags. Does that provide the information that you need?

Thanks @kfollis - I am aware of the Azure IP ranges. But since the IP list is dynamic (changes on a weekly basis), customers would need to update those frequently in their firewalls. Whitelisting URLs on the other hand is more permanent solution.

Closing this issue. The URL list to add to allow lists is in tech review and will publish as a part of the US Gov document.


Figure out how to support OSB 2.12

Currently, the catalog supports the 2.11 version of the OSB API spec.  This issue is to determine how we will support the 2.12 version once it is released.  Some background:

Minor version releases of the spec (ie 2.11 -> 2.12) should be backward compatible (ie, not change the behaviors of existing fields or introduce new mandatory fields)
We already have support for an alpha feature (accepted to the OSB spec, but not released): see (https://github.com/kubernetes-incubator/service-catalog/pull/686)

Initial suggestions for how to handle this (and in general, minor version bumps):

Add new fields to the OSB API client package in our code
Change the hard-coded version string that we send in OSB API call request headers
Rename any related flags that include alpha which are not release to no longer include alpha

There's are a couple other facets beyond code changes:

How do we message to users about OSB API version bumps?
How do we decide that we feel the catalog is ready to 'support' a particular API version?

We should devote some thought to this, because 2.12 is likely to be released soon.

Removing from 0.1.0 - @pmorie will fill in details on the plan here

The 2.12 issue is resolved by #971, but there seems to be a lot to unpack in here to multiple new issues.


Modify project structure, add gitignore, and update readme

Modify project structure
As I was setting up the project, I ran into some issues with where pom.xml is located. The problem is that when you open this file for IntelliJ to build the project, it leaves out those documents that are outside chat-web-app/chat-web-app. Now, by moving all contents inside chat-web-app/chat-web-app it is easier and faster to set the project up.
Before

After

Add gitignore
I added a basic list of files that are not needed so we do not push things that we should not.
Update readme
I added a setup guide! Check it out here and please leave your feedback! Let me know if it worked for you 😅

That's great thank you! 🙂 I followed the ReadMe and it worked well for me 👍🏽


"Vector Records" should be defined in the standard

Should clarify that it is a vector of data sets as opposed to just a vector of (integer or real or whatever) values.

I am a bit confused: The standard currently defines already scalar and tensor records, which includes the latter. You want to mention just the latter fact more clearly?
Just as background: For the data type of a record, many important data formats such as ADIOS2 do not yet support structs or n-floats as individual element-value, therefore we do not chose it. Also, we have more flexibility of scaling the individual components by mapping them to scalar components.

My apologies. I was vague. In the standard, under the section "Scalar, Vector and Tensor Records", the following or something like it should be added (and the existing first three paragraphs should be appropriately modified):
A record is a data structure that stores data. A scalar record stores a single type of datum. For example, a scalar record may store temperature which will be a set of temperatures measured on a grid. Notice that while this record holds an array of temperatures, it only holds one type of datum so it is called scalar.
A vector record holds an an array of sub-records. For example, a position vector record can hold position/x, position/y, and position/z scalar sub-records. Similarly a tensor record holds an array of vector records or tensor records.


retain temp_target if exists

in some environments, this can already exist as a link, and making it a
directory destroys the link (e.g. a shared folder under Vagrant)

Rebased.


fix: airflow metadata

Description

Fix failing airflow build: https://github.com/kedro-org/kedro-plugins/issues/494
Tests blocked by https://github.com/kedro-org/kedro/pull/3460
Development notes

Tests fail because as part of the tools process, a hook is used to remove dependencies from any tools not selected (we currently select none) from pyproject.toml. However, pyproject.toml is not found when an airflow project is generated, hence trying to remove requirements from a non-existent file causes errors.
Possible quick fix: include tool 1-5 in test project setup to avoid triggering the error-causing condition
Alternatively, we can use one of the starters, that protects against tool selection and doesn't run the hook - QQ would this introduce a new dependency?
Checklist

[x] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
[ ] Updated the documentation to reflect the code changes
[ ] Added a description of this change in the relevant RELEASE.md file
[ ] Added tests to cover my changes

< However, pyproject.toml is not found when an airflow project is generated, hence trying to remove requirements from a non-existent file causes errors.
How was this airflow project generated? Is it the kedro-starter starters?

CC: @noklam

< However, pyproject.toml is not found when an airflow project is generated, hence trying to remove requirements from a non-existent file causes errors.
How was this airflow project generated? Is it the kedro-starter starters?

It uses a link to the kedro template in framework's kedro/templates/project

@SajidAlamQB Unsure, maybe @ankatiyar can shed some light on this when she's back. Ultimately I do think using kedro new instead of cookiecutter aligns more closely with the rest of our testing practices, but it is odd that it resolved that particular issue.