[SPARK-9104][SPARK-9105][SPARK-9106][SPARK-9107][CORE][WIP] Netty network layer memory usage on webUI

This is a sub-task of SPARK-9103, we'd like to expose the memory usage for spark running time, this is the first step to expose the netty buffer used both with on-heap and off-heap memory. Also the metrics are showed on WebUI. In this PR, a new web Tab name Memory is added. Which is used to show the memory usage of each executors (can be in more details in future). the screenshot is like the following:

This is for each stages memory info:

This is History View:

This is WIP because with unit tests left.

Build triggered.

Build started.

Test build #38858 has started for   PR 7753 at commit 17e5b97.

Test build #38858 has finished for   PR 7753 at commit 17e5b97.

This patch fails Python style tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
class ExecutorMetrics extends Serializable
class MemoryListener extends SparkListener
class MemoryUIInfo

Build finished. Test FAILed.

A simple question, is it enough to only expose the maximum memory usage of Netty layer?
Besides, IMHO I think it would be better to separate getting Netty memory usage and displaying in web portal into two different PR.

@jerryshao , thanks for your review and feedback, I'll separate it into different PRs. And I will keep this PR open and waiting for others to see whether they have different thoughts on the whole implementation.

@squito mind helping to review this?

Jenkins, retest this please

Build triggered.

Build triggered.

Since we'll eventually want to add more metrics, can you put all the netty metrics into another case class inside ExecutorMetrics?
Also, I'm wondering if we want to use "netty" in the name -- I think most users won't know or care about netty in particular.  It should it just be named "network" or "transport", and the nio implementation should indicate that metrics are missing.
I guess altogether this means doing something like:
class ExecutorMetrics {
  var transportMetrics: Option[TransportMetrics] = ...
}

@jerryshao I'm not entirely sure I know what you mean by:
| A simple question, is it enough to only expose the maximum memory usage of Netty layer?
can you elaborate?  Obviously we'd always like more metrics, but are you saying this isn't useful?

Build started.

Build started.

Test build #164 has started for   PR 7753 at commit 17e5b97.

Test build #164 has finished for   PR 7753 at commit 17e5b97.

This patch fails Python style tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
class ExecutorMetrics extends Serializable
class MemoryListener extends SparkListener
class MemoryUIInfo

Build finished. Test FAILed.

Test build #39044 has started for   PR 7753 at commit 17e5b97.

Test build #39044 has finished for   PR 7753 at commit 17e5b97.

This patch fails Python style tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
class ExecutorMetrics extends Serializable
class MemoryListener extends SparkListener
class MemoryUIInfo

Build finished. Test FAILed.

Hi @squito , IIUC here the code always set the max memory usage:
 @volatile private var _nettyReadOnheapSize: Long = _
  def nettyReadOnheapSize: Long = _nettyReadOnheapSize
  private[spark] def setNettyReadOnheapSize(value: Long, time: SysTime): Unit = {
    _nettyReadOnheapSize = value
    if (value > _maxNettyReadOnheapSizeTime._1) {
      _maxNettyReadOnheapSizeTime = (value, time)
    }
  }

If current memory usage is smaller than the previous one, simple will discard it.
From my understanding it is better to transfer the raw metrics to the driver, and further processed by each listener as they wanted, that will be more flexible. If people want the minimum memory usage for example, they could process it by their logic in listener. Here only get the max memory usage will restrict the flexibility for different user.

@jerryshao I don't think that's true.  The executors metrics that get sent back to the driver have both the max value, and the current value.  So a custom listener could already track the minimum memory usage if it wanted.
I wonder if we should even calculate the max at all inside the executor, or just leave it to the listeners.  On one hand, max is what you will usually want, so might as well make it more convenient.  OTOH, maybe it would be cleaner to leave it to the listeners ... eg. I can already imagine in the future we may want to track max memory usage during each stage, and I definitely think that logic should be left to the listeners.
what do you think @liyezhang556520 ?  I will think about it more, but now I'm leaning towards just leaving max to the listeners ...

Merged build triggered.

Merged build started.

Test build #41313 has started for   PR 7753 at commit 7491279.

Test build #41313 has finished for   PR 7753 at commit 7491279.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ExecutorMetrics extends Serializable
case class TransportMetrics(
class MemoryListener extends SparkListener
class MemoryUIInfo
class transportMemSize
case class MemTime(memorySize: Long = 0L, timeStamp: Long = 0L)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41313/
Test FAILed.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #41315 has started for   PR 7753 at commit 424c172.

Test build #41315 has finished for   PR 7753 at commit 424c172.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ExecutorMetrics extends Serializable
case class TransportMetrics(
class MemoryListener extends SparkListener
class MemoryUIInfo
class transportMemSize
case class MemTime(memorySize: Long = 0L, timeStamp: Long = 0L)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41315/
Test FAILed.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #41372 has started for   PR 7753 at commit f21a804.

Test build #41372 timed out     for PR 7753 at commit f21a804     after a configured wait of 175m.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41372/
Test FAILed.

Jenkins, retest this please

Merged build triggered.

Merged build started.

Test build #41402 has started for   PR 7753 at commit f21a804.

Test build #41402 has finished for   PR 7753 at commit f21a804.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ExecutorMetrics extends Serializable
case class TransportMetrics(
class MemoryListener extends SparkListener
class MemoryUIInfo
class transportMemSize
case class MemTime(memorySize: Long = 0L, timeStamp: Long = 0L)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41402/
Test PASSed.

Merged build finished. Test PASSed.

Hi @liyezhang556520 , sorry for the long delay on the review.  I think this is progressing in the right direction, I really think this is going to help a lot of users understand memory usage out a lot.  I left a handful of comments on the code -- honestly I have not been through it in a ton of detail, but hopefully that helps you keep going.  Also I have some high-level comments:

I don't think there is any need to separate out the memory used by the client and server portions.  These are internal details that the end-user doesn't care about -- in fact you're already correctly simplifying by the time it gets to the web UI, but you could really combine them immediately in TransportMetrics.  That would help simplify the code I think.

We shouldn't say "Net" memory used, users might think that means total memory used.  I guess we should say "Network"

I've never heard "direct-heap" before, it seems strange -- I'd say "off-heap" or just "direct".

Could the stage table include the max memory per executor per stage as well?  That would be great to help users quickly identify the stages which require the most memory

How many additional events are getting logged?  With the current architecture, there is some pressure to not log too much -- both to keep log sizes small for later processing, and also to make sure the driver doesn't get too busy just logging (which eventually leads to it dropping events).

I don't think there is any need to separate out the memory used by the client and server portions. These are internal details that the end-user doesn't care about -- in fact you're already correctly simplifying by the time it gets to the web UI, but you could really combine them immediately in TransportMetrics. That would help simplify the code I think.

This make sense, I'll merge them in the Metrics itself.

We shouldn't say "Net" memory used, users might think that means total memory used. I guess we should say "Network"

Oh, I forgot thet "net" has another meaning.... Sorry for that.

4.Could the stage table include the max memory per executor per stage as well? That would be great to help users quickly identify the stages which require the most memory

Yes, User can see the max memory per executor per stage, there is a StageMemoryPage attached to MemoryTab, which will show the executors' status (max memory) for each finished stage.

How many additional events are getting logged? With the current architecture, there is some pressure to not log too much -- both to keep log sizes small for later processing, and also to make sure the driver doesn't get too busy just logging (which eventually leads to it dropping events).

additional events are only logged when there is stages complete or executors removed. Please see the design doc but the code is not aligned with the doc yet, I'll update the code first.

Merged build triggered.

Merged build started.

Test build #44815 has started for PR 7753 at commit a8fcf74.

Test build #44815 has finished for PR 7753 at commit a8fcf74.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * class ExecutorMetrics extends Serializable \n  * case class TransportMetrics(\n  * class MemoryListener extends SparkListener \n  * class MemoryUIInfo \n  * class TransportMemSize \n  * case class MemTime(memorySize: Long = 0L, timeStamp: Long = 0L)\n  * case class Corr(\n  * case class Corr(left: Expression, right: Expression)\n

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44815/
Test FAILed.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #44860 has started for PR 7753 at commit f2f0e64.

Merged build triggered.

Merged build started.

Test build #44866 has started for PR 7753 at commit 5f7a999.

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44870/
Test FAILed.

Test build #44860 has finished for PR 7753 at commit f2f0e64.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * class ExecutorMetrics extends Serializable \n  * case class TransportMetrics(\n  * class MemoryListener extends SparkListener \n  * class MemoryUIInfo \n  * class TransportMemSize \n  * case class MemTime(memorySize: Long = 0L, timeStamp: Long = 0L)\n

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44860/
Test FAILed.

Merged build finished. Test FAILed.

Test build #44866 has finished for PR 7753 at commit 5f7a999.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * class ExecutorMetrics extends Serializable \n  * case class TransportMetrics(\n  * class MemoryListener extends SparkListener \n  * class MemoryUIInfo \n  * class TransportMemSize \n  * case class MemTime(memorySize: Long = 0L, timeStamp: Long = 0L)\n

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44866/
Test FAILed.

Merged build triggered.

Merged build started.

Test build #44890 has started for PR 7753 at commit c836fb9.

Test build #44890 has finished for PR 7753 at commit c836fb9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * class ExecutorMetrics extends Serializable \n  * case class TransportMetrics(\n  * class MemoryListener extends SparkListener \n  * class MemoryUIInfo \n  * class TransportMemSize \n  * case class MemTime(memorySize: Long = 0L, timeStamp: Long = 0L)\n

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44890/
Test FAILed.

jenkins, retest this please

Merged build triggered.

Merged build started.

Test build #44912 has started for PR 7753 at commit c836fb9.

Test build #44912 has finished for PR 7753 at commit c836fb9.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * class ExecutorMetrics extends Serializable \n  * case class TransportMetrics(\n  * class MemoryListener extends SparkListener \n  * class MemoryUIInfo \n  * class TransportMemSize \n  * case class MemTime(memorySize: Long = 0L, timeStamp: Long = 0L)\n

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44912/
Test PASSed.

@squito, sorry for long delay updating, any further comments?

Hi @liyezhang556520 thanks for updating.  Can you add some unit tests, especially MemoryListener?  Eg., push some events into an EventLoggingListener, then read those events back into the MemoryListener and make sure the stages have the right metrics.  You can use the cases you describe in your design doc.
also all the info available in the memory tab should also be available through the json api as well ... that can wait for a followup if you really need to.

Test build #46091 has started for PR 7753 at commit 75e63c3.

Test build #46091 has finished for PR 7753 at commit 75e63c3.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * class ExecutorMetrics extends Serializable \n  * class TransportMetrics (\n  * class MemoryListener extends SparkListener \n  * class MemoryUIInfo \n  * class TransportMemSize \n  * case class MemTime(memorySize: Long = System.currentTimeMillis, timeStamp: Long = 0L)\n

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46091/
Test FAILed.

Test build #46092 has started for PR 7753 at commit 0c1241c.

Test build #46092 has finished for PR 7753 at commit 0c1241c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * class ExecutorMetrics extends Serializable \n  * class TransportMetrics (\n  * class MemoryListener extends SparkListener \n  * class MemoryUIInfo \n  * class TransportMemSize \n  * case class MemTime(memorySize: Long = System.currentTimeMillis, timeStamp: Long = 0L)\n

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46092/
Test FAILed.

Test build #46290 has started for PR 7753 at commit c78628e.

Test build #46290 has finished for PR 7753 at commit c78628e.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * class ExecutorMetrics extends Serializable \n  * class TransportMetrics (\n  * class MemoryListener extends SparkListener \n  * class MemoryUIInfo \n  * class TransportMemSize \n  * case class MemTime(memorySize: Long = 0L, timeStamp: Long = System.currentTimeMillis)\n

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46290/
Test FAILed.

Test build #46292 has started for PR 7753 at commit a93bd96.

Test build #46292 has finished for PR 7753 at commit a93bd96.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * class ExecutorMetrics extends Serializable \n  * class TransportMetrics (\n  * class MemoryListener extends SparkListener \n  * class MemoryUIInfo \n  * class TransportMemSize \n  * case class MemTime(memorySize: Long = 0L, timeStamp: Long = System.currentTimeMillis)\n

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46292/
Test FAILed.

Merged build finished. Test FAILed.

Test build #46522 has started for PR 7753 at commit 1ed48c1.

@squito thank you for your comments, it helped a lot, I updated some unit tests. If you got time, can you help to review? Thanks.

Test build #46522 has finished for PR 7753 at commit 1ed48c1.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):\n  * class ExecutorMetrics extends Serializable \n  * class TransportMetrics (\n  * class MemoryListener extends SparkListener \n  * class MemoryUIInfo \n  * class TransportMemSize \n  * case class MemTime(memorySize: Long = 0L, timeStamp: Long = System.currentTimeMillis)\n

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46522/
Test FAILed.

Jenkins, retest this please.

Test build #46533 has started for PR 7753 at commit 1ed48c1.

Test build #46533 has finished for PR 7753 at commit 1ed48c1.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):\n  * class ExecutorMetrics extends Serializable \n  * class TransportMetrics (\n  * class MemoryListener extends SparkListener \n  * class MemoryUIInfo \n  * class TransportMemSize \n  * case class MemTime(memorySize: Long = 0L, timeStamp: Long = System.currentTimeMillis)\n

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46533/
Test FAILed.

Test build #46576 has started for PR 7753 at commit b438077.

Test build #46576 has finished for PR 7753 at commit b438077.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * class ExecutorMetrics extends Serializable \n  * class TransportMetrics (\n  * class MemoryListener extends SparkListener \n  * class MemoryUIInfo \n  * class TransportMemSize \n  * case class MemTime(memorySize: Long = 0L, timeStamp: Long = System.currentTimeMillis)\n

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46576/
Test PASSed.

JsonProtocolSuite should also have a backward compatability test for the new fields

Test build #47305 has started for PR 7753 at commit 4123ac7.

Test build #47305 has finished for PR 7753 at commit 4123ac7.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):\n  * class ExecutorMetrics extends Serializable \n  * class TransportMetrics (\n  * class MemoryListener extends SparkListener \n  * class MemoryUIInfo \n  * class TransportMemSize \n  * case class MemTime(memorySize: Long = 0L, timeStamp: Long = System.currentTimeMillis)\n

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47305/
Test FAILed.

Test build #47415 has started for PR 7753 at commit 4b3dbe4.

Test build #47415 has finished for PR 7753 at commit 4b3dbe4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * class ExecutorMetrics extends Serializable \n  * class TransportMetrics (\n  * class MemoryListener extends SparkListener \n  * class MemoryUIInfo \n  * class TransportMemSize \n  * case class MemTime(memorySize: Long = 0L, timeStamp: Long = System.currentTimeMillis)\n

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47415/
Test PASSed.

One thought here:  it'd probably be nice to have a json history with the new events as part of the history server suite regression tests —that'll catch any changes to the new events which will break backwards compatibility

Test build #49235 has started for PR 7753 at commit 5e031ce.

Test build #49235 has finished for PR 7753 at commit 5e031ce.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49235/
Test FAILed.

Test build #49239 has started for PR 7753 at commit 87f8172.

Hi @steveloughran , thank you for your attention and your comments, and your further comments on this PR are pretty appreciated. Your advice is quite correct, we need to make a regression test for history server suite. I think this is related with what @squito mentioned that we need to add json api support. I'm wondering whether we can make it done in another PR after this PR be able to merge or this feature totally accepted by the community. And of course, rest api support is supposed to be one part of this feature.
@squito, what's your opinion on this?

Test build #49239 has finished for PR 7753 at commit 87f8172.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49239/
Test PASSed.

Merged build finished. Test PASSed.

@liyezhang556520 can you remove the "WIP" from the title.
cc @andrewor14

Thanks for the pull request. I'm going through a list of pull requests to cut them down since the sheer number is breaking some of the tooling we have. Due to lack of activity on this pull request, I'm going to push a commit to close it. Feel free to reopen it or create a new one. We can also continue the discussion on the JIRA ticket.
Note - I actually think this is a pretty useful feature, and in the future maybe we should switch over to consolidate memory management entirely with Netty.


Submitting multiple dask.xgboost calls causes fewer worker utilization

I have created a random classification dataset with 100,000 rows and 30 columns and I am training the distributed xgboost on this dataset. My system has 8 workers, therefore I created 8 partitions of my dataset. While running dask.xgboost only once on this dataset, each of the 8 workers get one part of the DaskDmatrix (total it has 8 parts). But when I submit multiple dask.xgboost calls, then the 8 partitions get randomly divided among a subset of the workers. Therefore, only those workers end up utilizing CPU and runtime becomes very high.
Here is a reproducible example,
from dask.distributed import Client
import xgboost as xgb
from sklearn.datasets import make_classification
import dask.array as da
from dask.distributed import get_client
import dask

n_samples, n_features, num_test_samples = 100000, 30, 100
dask.config.set({'distributed.worker.daemon': False})

def invoke_dis_xgboost(X, y, number_of_classes, number_of_estimators):
  client_xgboost = get_client()
  dtrain = xgb.dask.DaskDMatrix(client_xgboost, X, y)
  xgb_params = {
                  'n_estimators': number_of_estimators,
                  'num_class': number_of_classes,
                  ## all other xgb parameter
              }
output = xgb.dask.train(client_xgboost, xgb_params, dtrain, num_boost_round=100)
return

def main(client):
    print(f'n_samples={n_samples}, n_features={n_features}')
    X_local, y_local = make_classification(n_samples=n_samples, n_features=n_features, random_state=12345)
    number_of_classes = len(set(y_local))
    X = da.from_array(X_local, chunks=(n_samples//8,n_features), name='train_feature')
    y = da.from_array(y_local, chunks=(n_samples//8,), name='train_label')

    futures = []
    results = []

    for i in range(100, 105):
        f1 = client.submit(invoke_dis_xgboost, X, y, number_of_classes, i)
        futures.append(f1)
    for i, f in enumerate(futures):
        results.append(f.result())

return

if __name__ == '__main__':
    client = Client('<IP_ADDRESS>:8786')
    main(client)

Looking at this part of the source code of xgboost.dask.train
key_to_partition = {part.key: part for part in parts}
who_has = await client.scheduler.who_has(keys=[part.key for part in parts])
worker_map: Dict[str, "distributed.Future"] = defaultdict(list)
for key, workers in who_has.items():
     worker_map[next(iter(workers))].append(key_to_partition[key])

I was hoping to see each part will be distributed to one worker only. However, I am getting the following output,
('tuple-4d2a6701-77c1-41f6-9566-b5d360cde213', ('tcp://<IP_ADDRESS>:43719',),
('tuple-f301e12c-b53d-4520-a764-383d3a5e1785', ('tcp://<IP_ADDRESS>:46133',),
('tuple-4f5d4be3-4e51-4b49-87ea-ef4bf2460f21', ('tcp://<IP_ADDRESS>:32889',),
('tuple-2ad89ca9-2e8a-4486-bfd4-5e36eed92576', ('tcp://<IP_ADDRESS>:32889',),
('tuple-5856e611-f229-4c41-bf49-a39e4d3c6f9a', ('tcp://<IP_ADDRESS>:46133',),
('tuple-d176cd30-4049-40c2-927e-49d06088dc8b', ('tcp://<IP_ADDRESS>:46133',),
('tuple-aba3b2a6-d245-44a9-b3ec-9231b6637c88', ('tcp://<IP_ADDRESS>:43719',),
('tuple-f82b4ee7-47f4-4ccf-a431-efe5b930bd34', ('tcp://<IP_ADDRESS>:32889',)

Since there are only 3 unique workers getting all the parititions, number of dispatched_train is 3, therefore only 3 workers are utilizing CPU according to the dask dashboard.

My system has 8 workers, therefore I created 8 partitions of my dataset.

Can you try to let dask decide the partitioning?  Dask usually has a more fine-grained partitioning scheme and can saturate all the workers.

My system has 8 workers, therefore I created 8 partitions of my dataset.

Can you try to let dask decide the partitioning? Dask usually has a more fine-grained partitioning scheme and can saturate all the workers.

I tried with the default auto chunks of dask array. Runtime varied from 117 sec to 235 sec to  more than 5 minutes. From the dashboard, I have sometimes seen only 2 workers actually handling the dispatch_train call and in those cases, overall CPU utilization was very low.

For xgboost, it doesn't move the data. XGBoost will take whatever dask provide and run on corresponding workers

Is there a way to make sure that dask uniformly partitions my data to each worker?

There are some ongoing efforts on client.rebalance like https://github.com/dask/distributed/issues/4906 and some other techniques like repartition that you have already been using.  Feel free to participate in the discussion in dask/distributed.
I will close this issue now since it's about dask data management instead of XGBoost.


Link to Standards Document

MS1 Criteria

[x] The model exists
[x] There is a Catena-X standard referencing this model
[x] The model is in status "release"
[x] The model is not yet in status "standardize"
[x] Release note entry for model version referenced in standard received link to standard document

CX - 0093 Aspect Model Traction Battery Code

@jonbckr @johannsvarela Could you please raise the PR for Standardization of this model.

@agg3fe The model is standardized. It is the standard CX-0093. If you can not find it in the standard library, you'll be able to find it here: https://github.com/catenax-eV/product-standardization-prod/tree/main/standards/CX-0093-AspectModelTractionBatteryCode .

@jonbckr @johannsvarela Could you please raise the PR for Standardization of this model.

Done: https://github.com/eclipse-tractusx/sldt-semantic-models/pull/432

@jonbckr since this model is written in bamm, I need to create a new model with major version in samm. is that ok for you or you have any plans already?

@agg3fe this is fine by us. We do not have any plans for it.


Update and refactor

As useful as this repo and scivision have been for getting started, we are increasingly running into issues which would be much easier to resolve if we had our own 'version' of this model to build on and keep up to date.
As a first step, I've made minimal changes so that it runs without complaining using Python 3.11 and PyTorch 2.4, switched to the pyproject.toml standard for packaging and replaced requirements.txt with requirements.in.

Just a note to self about the labelling in the utils module. There are 5 different 'experiments' which correspond to a different number of output classes, from label1 which is just plankton vs detritus, up to label3_detritus which has 38 plankton classes + detritus.
The important place these labels are used is in the model __init__, where the final Linear layer (called fc) of ResNet is replaced by another Linear layer with an output size that is the number of classes for which predictions are needed.
However, afaik the only weights that have been published are here and correspond to label2_detritus.
So, if we are starting from scratch we remove all of the logic around the labels, though keeping these experiments in mind for future reference.

Since I have broken it anyway, I am removing the .scivision directory which contains the configuration that allows this model to be loaded through scivision.
This is something I want to revisit in future, ideally in a world in which the rewrite of intake has been completed and scivision have updated their spec.

There is not much to really do with the benchmark / test set of 26 images, except check that the results are reasonably consistent with a model that's been well-trained. To this end I added a couple of metrics to the benchmark script.
I used torchmetrics because I agree with their principle of standardising metrics across ML, but unless this repo ends up using any more metrics I will probably remove in future in favour of pure-torch implementations of accuracy and confusion matrix.

Thought about making torchmetrics and matplotlib optional dependencies, but it's just overengineered. There is no need to use either of them for the benchmark as it currently stands so just dropping them for now.

The current version has been tested with a fresh 3.11.9 venv and a python -m pip install . followed by running the benchmark cefas-benchmark.

I have checked that pip installing from the repository (@update) and running the benchmark works..


Add support for WPFs XAML Whitespace Handling

Is your feature request related to a problem? Please describe.
To support WPFs more complex text formatting (or any other form of it, really), Avalonia needs an opt-in way for a control to receive the whitespace that is present in the XAML content for itself. Otherwise, the following:
<TextBlock>
<Bold>BoldText</Bold>
Text
</TextBlock>

Will strip all whitespace, so that it becomes <Bold>BoldText</Bold>Text, making it effectively impossible to have a whitespace between BoldText and Text.
Describe the solution you'd like
Support having whitespace preserved as an opt-in feature on a control-by-control basis. WPF uses two attributes for this:
WhitespaceSignificantCollectionAttribute and TrimSurroundingWhitespaceAttribute.
But overall, white space processing is more complex and might require more work.
I.e., I've noticed that Avalonia does NOT collapse inner white space, so that Hello     World will become Hello World in WPF, but remains as Hello     World in Avalonia.
Additional Context
WPF Documentation: https://docs.microsoft.com/en-us/dotnet/desktop/xaml-services/white-space-processing
edit: Partial duplicate of #2619 (only mentions whitespace preservation, but not collapsing whitespace).

Duplicate of #2619


Goodbye Raccoon v3.x, Hello Raccoon v4.x

There will be no further development on Raccoon v3. This repository is only kept here for legacy reasons. Development on Raccoon v4 is continued in another repository:
https://github.com/onyxbits/raccoon4/
Please head over there.

Would you please write a release list notes to describe new features of Raccoon4?

Raccoon v4 is a complete rewrite that goes beyond being a mere APK downloader:

Integrated App manager / browser (no need to bother with the filemanager and cryptic filenames any longer).
Integrated webserver (for downloading APKs / other files from PC to Droid)
Integrated ADB frontend (installing apps from within Raccoon, sending urls, taking screenshots)
Way improved market search (including screenshots now, permission tree, meta information)

thank you

@onyxbits Better merge #53 and archive this project to make this more clear / visible.


Change image signals

Signals for custom Image and Rendition models are connected automatically now.

Thanks @mikedingjan - looks good to me! Merged in 8c970e4848b9cedd5251082c1a64dfcc907e1afa.
I've also removed (in 25c2be10c3486fc3228251db7211df5286931c14) the docs about setting up the signal handler for image feature detection, as I believe these are now obsolete too.


All validation loss reported as 0.0000e+00 in keras / TF

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
This is running on Google Colab. I am running macOS 10.15.6. Colab is opened in Chrome 84.0.4147.105 .
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary):
Not installed by me as it is running in Colab.
TensorFlow version (use command below):
v2.3.0-0-gb36436b087 2.3.0
Python version: 3.6.9
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory: N/A although this also happens when using Tesla P100 on Colab

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with:

TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior
During training, val_loss is reported as 0.0000e+00.
When printing model.history.history['val_loss'], an array of [0.0,..., 0.0] is printed.
However, when evaluating the model on examples val_x, val_y, a non-zero loss is reported.
This happens for at least 4 different notebooks, including notebooks that worked properly earlier today.
Describe the expected behavior
During training, a non-zero validation loss should be reported in most cases.
These non-zero losses should be reported in the model history.
When evaluating the model against the validation examples val_x, val_y, a loss similar to that reported in the model history should be reported.
Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.
https://colab.research.google.com/drive/1YP_dR7O9KsqSeLyrgFTzOSna0EggL9Gs?usp=sharing
Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

I have tried in colab with TF version 2.2,2.3 and was able to reproduce the issue.Please, find the gist here..
However i am seeing the below error message with TF nightly version(2.4.0-dev20200804).
ValueError: Layer functional_1 expects 1 input(s), but it received 2 input tensors. Inputs received: [<tf.Tensor 'IteratorGetNext:0' shape=(10, 1, 10) dtype=float32>, <tf.Tensor 'IteratorGetNext:1' shape=(10, 1, 1) dtype=float32>]
Thanks!

@becker929 Thanks for creating this issue.
Root-cause of this issue is that the validation_data was provided as a list where as  validation_data needs to be a tuple. Please check the argument of modell.fithere.

tuple (x_val, y_val) of Numpy arrays or tensors
tuple (x_val, y_val, val_sample_weights) of Numpy arrays

I updated validation_data argument within the model.fit from validation_data=[val_x, val_y] to validation_data=(val_x, val_y) . There was no error after this modification.
Please check the gist here.
Please verify once and close the issue if this resolved your issue. Thanks!

Hi, thanks! Indeed, after trying this with some other train/val data, this does solve the problem. I did not have to install the nightly build to solve the problem. I updated the notebook in the issue description demonstrating the change. Thanks again.

FYI, completely not the same reason, but I got the exact same problem when I failed to specify a loss function (in tensorflow 2.3.0) during the call to .compile() on the model.

@brethvoice Please open a new issue with a simple standalone code to reproduce the issue. thanks!

@brethvoice Please open a new issue with a simple standalone code to reproduce the issue. thanks!

Actually it was my failure to read the API that caused the issue.  This is not something Tensorflow can or should "fix."

Hello I am also having a similar issue. The validation loss is always 0 even though i use keras and not tf.keras. But i have values for validation accuracy.
I am using tf-version 2.1.0 and keras 2.3.1.
Can anyone help me with this?

Hello I am also having a similar issue. The validation loss is always 0 even though i use keras and not tf.keras. But i have values for validation accuracy.
I am using tf-version 2.1.0 and keras 2.3.1.
Can anyone help me with this?

Check your .compile() call and make sure you have a loss function specified there.

Hello I am also having a similar issue. The validation loss is always 0 even though i use keras and not tf.keras. But i have values for validation accuracy.
I am using tf-version 2.1.0 and keras 2.3.1.
Can anyone help me with this?

Check your .compile() call and make sure you have a loss function specified there.

I have a loss function specified but still i have the issue.

Hello I am also having a similar issue. The validation loss is always 0 even though i use keras and not tf.keras. But i have values for validation accuracy.
I am using tf-version 2.1.0 and keras 2.3.1.
Can anyone help me with this?

Check your .compile() call and make sure you have a loss function specified there.

I have a loss function specified but still i have the issue.

@ArpithaFAU another thing I would check is that your validation data are being presented in a tuple (x, y).  If they are not then the validation calculation will not be possible.

Hello I am also having a similar issue. The validation loss is always 0 even though i use keras and not tf.keras. But i have values for validation accuracy.
I am using tf-version 2.1.0 and keras 2.3.1.
Can anyone help me with this?

Check your .compile() call and make sure you have a loss function specified there.

I have a loss function specified but still i have the issue.

@ArpithaFAU another thing I would check is that your validation data are being presented in a tuple (x, y). If they are not then the validation calculation will not be possible.

@brethvoice i have done that as well but i still have the same issue. If you are aware of any other solution pls let me know.

I  am facing the same issue.
WIth tf.keras I used validation_data as a  tuple to get
/lib/python3.7/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 630, in_is_list_of_scalars
return ListsOfScalarsDataAdapter._is_list_of_scalars(inp[0])
IndexError: list index out of range
With keras and on using validation data as a tuple I get the same error
With keras and on using validation data as a list I get the 0 validation loss.
Would greatly appreciate any help.

I  am facing the same issue.
WIth tf.keras I used validation_data as a  tuple to get
/lib/python3.7/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 630, in_is_list_of_scalars
return ListsOfScalarsDataAdapter._is_list_of_scalars(inp[0])
IndexError: list index out of range
With keras and on using validation data as a tuple I get the same error
With keras and on using validation data as a list I get the 0 validation loss.
Would greatly appreciate any help.

What i tried is upgrading to tensorflow and keras to the latest version and
that fixed my problem.
On Wed, Jan 20, 2021 at 9:16 PM Nirav Diwan<EMAIL_ADDRESS>wrote:

I am facing the same issue.
WIth tf.keras I used validation_data as a tuple to get
/lib/python3.7/site-packages/tensorflow/python/keras/engine/data_adapter.py",
line 630, in_is_list_of_scalars
return ListsOfScalarsDataAdapter._is_list_of_scalars(inp[0])
IndexError: list index out of range
With keras and on using validation data as a tuple I get the same error
With keras and on using validation data as a list I get the 0 validation
loss.
Would greatly appreciate any help.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/42053#issuecomment-763905683,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIFKDILDLNFNZVFWV6ROGU3S2422PANCNFSM4PVCRLKQ
.

I get a similar error, I have the latest version. Any help with this?

Try changing the loss in your model.fit from loss="categorical_crossentropy" to loss="binary_crossentropy"
I Had the SAME problem, but this worked for me. The thing is that I have a binary classification model, with only 1 output node, not a multi-classification model with multiple output node, so loss="binary_crossentropy" is the appropriate loss function in this case.
😊


Fix development steps in README

'symengine-0.0.0.gem' should be 'symengine-0.1.0.gem'

@codeworm96 Thanks for pointing it out.
@isuruf The one on RubyGems has version 0.0.2. Same version should be maintained in the repo.

@abinashmeher999, development version has to be ahead of the released version in RubyGems

0.0.3 then?

I think this is fine. When we want to make a new release we'll do it for 0.1.0 next


Dev: update link component to add visited state

Summary: Update the code of the link component to include a visited state, to align with Canada.ca.
Figma | Link component

[x] Sync with Design
[x] Update Requirements documentation
[x] Review of requirements doc
[x] Add new global token for visited link
[ ] Update link styling in utility framework
[x] Update storybook
[x] Update dev guidance if needed
[x] Develop the component according to the requirements doc and design specs
[x] Sync dev properties with Figma properties (design + dev)
[ ] Final sync with Design
[ ] Code review
[ ] Push to prod

Acceptance Criteria

[ ] The link component code includes a visited state.
[ ] Storybook and code editor reflect new state
[ ] New global token has been added
[ ]  New visited link styling has been added to the utility framework
[ ] Release notes are included

The new global token was added in this PR.

The visited state was added to the link component in this PR.

The visited link styling was added to the utility framework in this PR.


Large TLS segments cause crashes on Cray XE (exposed by AMPI using TLS for privatization)

Original issue: https://charm.cs.illinois.edu/redmine/issues/42

Memory incompatibility with large TLS > 64k.
Bug reported to Cray and being worked on by them.
From email:
"Updated    2/6/2013 3:46:40 PM
Comment    After a little more digging, I discovered this doesn't have anything to do with MPI at all. The following code fails on compute nodes, but not service nodes:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>
#define SIZE 800000
#define PATH_MAX 65536
__thread int globalbuf[SIZE] = {1} ;
void *work(void *thread_id)
{
char buf[PATH_MAX];
snprintf(buf, PATH_MAX, "hello, world!");
printf("%s\n", buf);
pthread_exit(NULL);
}
int main(void)
{
pthread_t thread;
long t = 0;
pthread_create(&thread, NULL, work, (void*)t);
sleep(2);
printf("exiting main\n");
pthread_exit(NULL);
}
mork tests/793540> cc -o test test.c
mork tests/793540> aprun -n 1 ./test
Application 10215948 exit signals: Segmentation fault
Application 10215948 resources: utime ~0s, stime ~0s
mork tests/793540> ./test
hello, world!
exiting main
mork tests/793540>
Reducing the value of SIZE allows it to execute successfully on compute nodes.
I used snprintf and a 64K buffer here because that's what was eventually getting tripped up in MPI. "

Original date: 2013-03-06 19:57:20

Clarify subject

Original date: 2013-03-06 19:57:58

As Celso reported, Cray has a workaround, setting stack size (not memory usage) ulimit to some large but finite value, and is working on a proper solution.

Original date: 2013-03-17 21:52:54

Celso is tracking this with Cray, so assign it to him.

Original date: 2013-03-29 22:26:58

Set as Upstream so it's clear that this isn't a problem to be fixed in our code.

Original date: 2013-05-03 03:30:09

Has Cray reported any progress on a fix for this issue?

Original author: Celso Mendes
Original date: 2013-05-03 04:30:07

Cray had responded that they are providing a new option (new env.var) to handle this, in a new CLE. So, they are keeping the bug as "Resolved" but still Open, such that it will be closed only after the new CLE is shipped. I suspect this will happen sometime in the Summer.

Original date: 2016-03-08 20:25:58

Does this issue still exist?

Original date: 2016-09-29 03:38:34

This seems to have been taken care of by Cray sometime in the past 3 years. It's not an issue on Edison or Blue Waters now.


fix: invalidate nominated pair if we invalidated one of its candidates

I didn't manage to write a test for this but I think it is an artifact of the test harness which "simply" flips source and destination and thus still allows traffic from <IP_ADDRESS> despite us invalidating that candidate.

@algesten I think there may still be a bug here that needs addressing at some point. It is not critical for us because the ICE timeout is good enough to detect failed connections.


Diff mode gets stuck

Unfortunately I haven't discovered an easy way to reproduce this.

1. open two similar files (call them A and B) in split windows
2. windo diffthis
3. navigate away from B using bufexplorer, netrw and other plugins, opening and
closing windows etc.
4. switch back to buffer B

It sometimes happens that in step 4 I get to see buffer B without diff
highlighting.  At that point I can switch to the window displaying buffer A,
:diffoff, close all other windows with :only, attempt a big hamer like  :bufdo
diffoff, and yet if I'll open an unrelated file C and try to :diffthis, vim
will begin diffing it with B.

The only way to make vim forget it was ever diffing with B that I discovered is
to open a new tab page and close the old tab page.

I'm using vim 7.4.617 on Linux.

Original issue reported on code.google.com by<EMAIL_ADDRESS>on 5 Mar 2015 at 11:14

Copying answer from https://code.google.com/p/vim/issues/detail?id=336
Well, it's a bit hard to understand the problem, but if I understand correctly, this can happen because if you open buffer a in a new window, 'diff' mode will be off, because it is a window-local option (this is explained briefly at :h option-summary).
So this works as expected, closing.


Skip whitespaces in preprocessor

This PR will skip any white spaces that occur in the lex::replace::replace method, to support macro calls where
the identifier and the arguments are separated by spaces (e.g. f ().
Resolves #457

r? @hdamron17
I'm worried this will lose whitespace again while printing with -E but I'll defer to Hunter's judgement.

As an alternative you could store the whitespace tokens and yield those tokens if no replacement was found.

That sounds like a good idea

As long as the macro is actually replaced then it is fine to consume the whitespace. The issue is if you have an id that is the name of a function macro but has is not used as one because it is not followed by a parenthesis. Can you peek at the token following the whitespace?

Make a test using assert_same_exact that has both cases of f(...) being replaced and f left as is.

Can you add the example I gave as a unit test (or runner test if that's easier)?
// should be 1
#define f(a) 1
#define h f (2)
h

The test is still failing with the following error:
left: `"\n\n1\n"`,
right: `"\n1\n"`',

I can't figure out why so you have to look over it.


NOISSUE - Add Things Migration

Signed-off-by: rodneyosodo<EMAIL_ADDRESS>KNOWN ISSUES

[x] We have an issue when importing version 0.14.0 - Standby mode was to create a branch on http://github.com/rodneyosodo/mainflux/pkg/sdk/go v0.0.0-20230321144058-d764206b0afa
[ ] When exporting connections of things and channels from 0.13.0 it takes a little bit longer time depending on the number of things and channels connected
[ ] When inserting the connections from 0.13.0 to 0.14.0 it takes some time depending on the number of connections. If is large it will timeout the token i.e takes more than 15mins provided

regarding the token timeout, you can add to documentation on how to change env  var to create a longer duration token for migration purposes


Improve Documentation

#CSS Notes

CSS Backgrounds and Gradients

 /* Using a background image */
 .container {
   background-image: url(bg.jpg);
 }
- > For a beginner, it's important to know that you can use different types of image files for background images. such as
- .PNG
- .SVG
- .GIF
but we specify only one there

can you assign this to me sir ?

Can you assign me to this issue?

Hey @Tanmay-Joshi73 go ahead


Cannot listen on team_join event

I want to set slack client to listen on team_join event.
slack.on('team_join', function(message) {
    var user = slack.getUserByID(message.user),
        channel = slack.getChannelByName('general');

    console.log('message:', message);
    channel.send('Hello ' + user.name);
});

However, nothing happens when a new member joins the team.

ugh, changing event name or combining events together without document is not fun :(

@khanh-nguyen I ran into this exact same problem yesterday. Were you able to solve it? I there a new naming convention for events?

Yeap, @jschuur, as you've seen in my code, they combined team_join and user_change into user_change. I don't know how different it is to use "hubot powered Slack bot" but if you want to another eye on your code, let me know.


Retrieving trust forest data

In an environment where DMZDC01.HOME.COM has a trust relationship like this:
dn: CN=dev.com,CN=System,DC=home,DC=com
cn: dev.com
securityIdentifier: S-1-5-21-1135011135-3178090508-3151492220
name: dev.com
trustDirection: bidirectional
trustPartner: dev.com
trustType: Windows domain running Active Directory
trustAttributes: FOREST_TRANSITIVE
flatName: DEV

It is possible to retrieve all dev.com data from the dc01 while running SharpHound:
sliver (LOVELY_HONESTY) > execute-assembly /home/kali/backpack/winaries/SharpHound-v1.1.1/SharpHound.exe '-c all,LoggedOn,GPOLocalGroup -d comply.com -v 1'

2024-04-28T09:21:44.4238613-07:00|INFORMATION|This version of SharpHound is compatible with the 4.3.1 Release of BloodHound
2024-04-28T09:21:44.5332372-07:00|INFORMATION|Resolved Collection Methods: Group, LocalAdmin, GPOLocalGroup, Session, LoggedOn, Trusts, ACL, Container, RDP, ObjectProps, DCOM, SPNTargets, PSRemote
2024-04-28T09:21:44.5488628-07:00|INFORMATION|Initializing SharpHound at 9:21 AM on 4/28/2024
2024-04-28T09:21:44.8270927-07:00|INFORMATION|[CommonLib LDAPUtils]Found usable Domain Controller for dev.com : rdc02.dev.com
2024-04-28T09:21:44.9962054-07:00|INFORMATION|Loaded cache with stats: 116 ID to type mappings.
 118 name to SID mappings.
 0 machine sid mappings.
 5 sid to domain mappings.
 0 global catalog mappings.
2024-04-28T09:21:44.9962054-07:00|INFORMATION|Flags: Group, LocalAdmin, GPOLocalGroup, Session, LoggedOn, Trusts, ACL, Container, RDP, ObjectProps, DCOM, SPNTargets, PSRemote
2024-04-28T09:21:45.0587119-07:00|INFORMATION|[CommonLib LDAPUtils]Found usable Domain Controller for home.com : dmzdc01.home.com
2024-04-28T09:21:45.2099560-07:00|INFORMATION|Beginning LDAP search for dev.com
2024-04-28T09:21:45.2461661-07:00|INFORMATION|Producer has finished, closing LDAP channel
2024-04-28T09:21:45.2461661-07:00|INFORMATION|LDAP channel closed, waiting for consumers
2024-04-28T09:21:45.2712512-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:45.2712512-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 4256AFBB-54AE-4C54-989B-CC602C85C08D: top, container
2024-04-28T09:21:45.2712512-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 4256AFBB-54AE-4C54-989B-CC602C85C08D: Container
2024-04-28T09:21:45.2788286-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:45.2788286-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 5382E506-20CC-4B00-96E8-4A680DAA98CA: top, container
2024-04-28T09:21:45.2788286-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 5382E506-20CC-4B00-96E8-4A680DAA98CA: Container
2024-04-28T09:21:45.7961899-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:45.7961899-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 040A1EBF-BED0-43C4-B339-43429FF9E931: top, container
2024-04-28T09:21:45.7961899-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 040A1EBF-BED0-43C4-B339-43429FF9E931: Container
2024-04-28T09:21:46.8162026-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:46.8162026-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 8F6E5AE9-8AA1-4ECA-80E5-1B8144EB30C8: top, container
2024-04-28T09:21:46.8162026-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 8F6E5AE9-8AA1-4ECA-80E5-1B8144EB30C8: Container
2024-04-28T09:21:47.3188991-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:47.3188991-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 9FB8A3B4-D9AD-49F8-BB83-610E545DA6E0: top, container
2024-04-28T09:21:47.3188991-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 9FB8A3B4-D9AD-49F8-BB83-610E545DA6E0: Container
2024-04-28T09:21:48.3063014-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:48.3063014-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 03F343C3-618E-4864-90F6-42DFAC63D4AF: top, container
2024-04-28T09:21:48.3063014-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 03F343C3-618E-4864-90F6-42DFAC63D4AF: Container
2024-04-28T09:21:49.3096264-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:49.3096264-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 850FB158-F1D0-4530-8ADE-7CF5454CDD8E: top, container
2024-04-28T09:21:49.3096264-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 850FB158-F1D0-4530-8ADE-7CF5454CDD8E: Container
2024-04-28T09:21:50.2972502-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:50.2972502-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 3CA382AB-EFAE-4D67-978C-A03F0AF6A972: top, container
2024-04-28T09:21:50.2972502-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 3CA382AB-EFAE-4D67-978C-A03F0AF6A972: Container
2024-04-28T09:21:51.3003128-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:51.3003128-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 0AD49009-FBB2-4E99-ACE8-B518A5DA0856: top, container, rpcContainer
2024-04-28T09:21:51.3003128-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 0AD49009-FBB2-4E99-ACE8-B518A5DA0856: Container
2024-04-28T09:21:51.8190313-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:51.8190313-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 3DB637EE-A74D-4E80-8AF2-B37039BEFF8B: top, container
2024-04-28T09:21:51.8190313-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 3DB637EE-A74D-4E80-8AF2-B37039BEFF8B: Container
2024-04-28T09:21:52.8067612-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:52.8067612-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for AA13F528-794C-479E-A7B6-D4390B5CE6D4: top, container
2024-04-28T09:21:52.8067612-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for AA13F528-794C-479E-A7B6-D4390B5CE6D4: Container
2024-04-28T09:21:53.8099255-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:53.8099255-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 5E7D650B-3DC3-4A16-89E6-C78FA9204D73: top, container, groupPolicyContainer
2024-04-28T09:21:53.8099255-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 5E7D650B-3DC3-4A16-89E6-C78FA9204D73: GPO
2024-04-28T09:21:53.8299817-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:53.8299817-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 53E2013D-6DBC-4AB1-9D9A-BDA92432C3D8: top, container, groupPolicyContainer
2024-04-28T09:21:53.8299817-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 53E2013D-6DBC-4AB1-9D9A-BDA92432C3D8: GPO
2024-04-28T09:21:53.8311220-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:53.8311220-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for AC4BCD10-8C71-453A-8824-4EEB28FF140E: top, container
2024-04-28T09:21:53.8311220-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for AC4BCD10-8C71-453A-8824-4EEB28FF140E: Container
2024-04-28T09:21:54.8013366-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:54.8013366-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 013E881A-0E09-4188-953F-F78CBF4A1C2F: top, container
2024-04-28T09:21:54.8013366-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 013E881A-0E09-4188-953F-F78CBF4A1C2F: Container
2024-04-28T09:21:55.3044308-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:55.3044308-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 08DA2244-B415-496F-82CD-5DAB8C7F944B: top, container
2024-04-28T09:21:55.3044308-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 08DA2244-B415-496F-82CD-5DAB8C7F944B: Container
2024-04-28T09:21:56.3079220-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:56.3079220-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for C71D8C5F-04C8-4C9C-AA66-11912A5E6D1F: top, container
2024-04-28T09:21:56.3079220-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for C71D8C5F-04C8-4C9C-AA66-11912A5E6D1F: Container
2024-04-28T09:21:57.3112956-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:57.3112956-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 11C95A66-997D-4FCD-BEFD-F06FEEC2648D: top, container
2024-04-28T09:21:57.3112956-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 11C95A66-997D-4FCD-BEFD-F06FEEC2648D: Container
2024-04-28T09:21:58.2988763-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:58.2988763-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 484416FB-E895-4707-8C98-8B1AAD570422: top, container
2024-04-28T09:21:58.2988763-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 484416FB-E895-4707-8C98-8B1AAD570422: Container
2024-04-28T09:21:59.3020219-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:59.3020219-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 57024E69-420F-4FA6-92ED-EE82F213BC67: top, container
2024-04-28T09:21:59.3020219-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 57024E69-420F-4FA6-92ED-EE82F213BC67: Container
2024-04-28T09:21:59.8050886-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:21:59.8050886-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for F7BD7E4E-89AE-4C7F-98C7-F247FC9DAEFF: top, container
2024-04-28T09:21:59.8050886-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for F7BD7E4E-89AE-4C7F-98C7-F247FC9DAEFF: Container
2024-04-28T09:22:00.8085852-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:22:00.8085852-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 6A14D697-A21B-435E-99DD-1E8FEC289EF6: top, container
2024-04-28T09:22:00.8085852-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 6A14D697-A21B-435E-99DD-1E8FEC289EF6: Container
2024-04-28T09:22:01.7966196-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:22:01.7966196-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 2FEE540A-1DC5-416B-AD89-1808DB4A46FF: top, container, msImaging-PSPs
2024-04-28T09:22:01.7966196-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 2FEE540A-1DC5-416B-AD89-1808DB4A46FF: Container
2024-04-28T09:22:15.8086213-07:00|INFORMATION|Status: 0 objects finished (+0 0)/s -- Using 45 MB RAM
2024-04-28T09:22:21.8110147-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:22.8132657-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:29.3190030-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3211661-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3211661-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3211661-07:00|DEBUG|[CommonLib ACLProc]Owner is null for<EMAIL_ADDRESS>2024-04-28T09:22:30.3211661-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3211661-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3211661-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3211661-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3211661-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3211661-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3211661-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3211661-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3211661-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3211661-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3211661-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3211661-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3528236-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3528236-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3528236-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Group
2024-04-28T09:22:30.3528236-07:00|DEBUG|[CommonLib ACLProc]Owner is null for<EMAIL_ADDRESS>2024-04-28T09:22:30.3528236-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned User
2024-04-28T09:22:30.3528236-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned User
2024-04-28T09:22:30.3528236-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned User
2024-04-28T09:22:30.3528236-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:22:30.3528236-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for S-1-5-21-1135011135-3178090508-3151492220-1104: top, person, organizationalPerson, user
2024-04-28T09:22:30.3528236-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for S-1-5-21-1135011135-3178090508-3151492220-1104: Base
2024-04-28T09:22:30.3528236-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:22:30.3528236-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for S-1-5-21-1135011135-3178090508-3151492220-1105: top, person, organizationalPerson, user
2024-04-28T09:22:30.3528236-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for S-1-5-21-1135011135-3178090508-3151492220-1105: Base
2024-04-28T09:22:30.3528236-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned User
2024-04-28T09:22:30.3684458-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Computer
2024-04-28T09:22:30.3840717-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:22:30.3840717-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 6A62EA2F-A1B2-4FC3-883E-E88FBAD2B8CE: top, organizationalUnit
2024-04-28T09:22:30.3840717-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 6A62EA2F-A1B2-4FC3-883E-E88FBAD2B8CE: OU
2024-04-28T09:22:30.3996953-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:22:30.3996953-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 211A9CE4-AB13-4693-8F1E-0E147B49A648: top, organizationalUnit
2024-04-28T09:22:30.3996953-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 211A9CE4-AB13-4693-8F1E-0E147B49A648: OU
2024-04-28T09:22:30.3996953-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:22:30.3996953-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for S-1-5-21-1135011135-3178090508-3151492220: top, domain, domainDNS
2024-04-28T09:22:30.3996953-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for S-1-5-21-1135011135-3178090508-3151492220: Domain
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 56 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 51 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 53 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 26 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 31 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 36 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 20 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 11 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 49 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 24 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 39 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 50 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 12 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 25 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 13 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 19 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 15 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 18 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 16 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 27 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:22:30.4778182-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for 547B369F-267C-4BBE-B986-ED49B30770BE: top, organizationalUnit
2024-04-28T09:22:30.4778182-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for 547B369F-267C-4BBE-B986-ED49B30770BE: OU
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 52 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 40 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 54 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 17 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 41 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 21 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 22 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 23 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 29 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 30 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 32 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 33 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 34 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 35 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 37 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 38 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 48 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 42 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 43 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 44 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 45 completed
2024-04-28T09:22:30.3996953-07:00|DEBUG|Consumer task on thread 46 completed
2024-04-28T09:22:30.4309442-07:00|DEBUG|[CommonLib Extensions]GetLabel - SamAccountTypeToType returned Base
2024-04-28T09:22:30.4788974-07:00|DEBUG|[CommonLib Extensions]GetLabel - ObjectClasses for F86CA649-3278-4DF3-A752-4EDE79446944: top, organizationalUnit
2024-04-28T09:22:30.4788974-07:00|DEBUG|[CommonLib Extensions]GetLabel - Final label for F86CA649-3278-4DF3-A752-4EDE79446944: OU
2024-04-28T09:22:30.4309442-07:00|DEBUG|Consumer task on thread 47 completed
2024-04-28T09:22:30.4794076-07:00|DEBUG|Consumer task on thread 28 completed
2024-04-28T09:22:30.4794076-07:00|DEBUG|Consumer task on thread 8 completed
2024-04-28T09:22:30.4794076-07:00|DEBUG|Consumer task on thread 4 completed
2024-04-28T09:22:30.4828717-07:00|DEBUG|Consumer task on thread 14 completed
2024-04-28T09:22:30.4828717-07:00|DEBUG|Consumer task on thread 9 completed
2024-04-28T09:22:30.4853092-07:00|DEBUG|[CommonLib CompSessions]NetSessionEnum failed on RDC02.dev.COM: ERROR_ACCESS_DENIED
2024-04-28T09:22:30.4931632-07:00|DEBUG|Consumer task on thread 55 completed
2024-04-28T09:22:30.5413931-07:00|DEBUG|Consumer task on thread 6 completed
2024-04-28T09:22:30.5413931-07:00|INFORMATION|Consumers finished, closing output channel
2024-04-28T09:22:30.5717691-07:00|INFORMATION|Output channel closed, waiting for output task to complete
Closing writers
2024-04-28T09:22:30.8065334-07:00|INFORMATION|Status: 94 objects finished (+94 2.088889)/s -- Using 49 MB RAM
2024-04-28T09:22:30.8065334-07:00|INFORMATION|Enumeration finished in 00:00:45.5871616
2024-04-28T09:22:30.8534113-07:00|INFORMATION|Saving cache with stats: 116 ID to type mappings.
 118 name to SID mappings.
 0 machine sid mappings.
 5 sid to domain mappings.
 0 global catalog mappings.
2024-04-28T09:22:30.8534113-07:00|INFORMATION|SharpHound Enumeration Completed at 9:22 AM on 4/28/2024! Happy Graphing!

But with bloodhoun-python it fails:
bloodhound-python -u 'Administrator' --hashes ':289136c329f3e42331048a0465b2290a' -ns '<IP_ADDRESS>' -c "all,LoggedOn" -d 'dev.com' --zip --dns-tcp -v

DEBUG: Authentication: NT hash
DEBUG: Resolved collection methods: rdp, objectprops, acl, session, container, loggedon, trusts, psremote, group, localadmin, dcom
DEBUG: Using DNS to retrieve domain information
DEBUG: Querying domain controller information from DNS
DEBUG: Using domain hint: dev.com
INFO: Found AD domain: dev.com
DEBUG: Found primary DC: rdc02.dev.com
DEBUG: Found Global Catalog server: rdc02.dev.com
DEBUG: Found Global Catalog server: cdc07.ops.dev.com
DEBUG: Found KDC for enumeration domain: rdc02.dev.com
INFO: Getting TGT for user
DEBUG: Trying to connect to KDC at rdc02.dev.com:88
DEBUG: Traceback (most recent call last):
  File "/home/kali/git/public/BloodHound.py/.venv/lib/python3.11/site-packages/impacket/krb5/kerberosv5.py", line 61, in sendReceive
    af, socktype, proto, canonname, sa = socket.getaddrinfo(targetHost, port, 0, socket.SOCK_STREAM)[0]
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/socket.py", line 962, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kali/git/public/BloodHound.py/bloodhound/ad/authentication.py", line 273, in get_tgt
    tgt, cipher, _, session_key = getKerberosTGT(
                                  ^^^^^^^^^^^^^^^
  File "/home/kali/git/public/BloodHound.py/.venv/lib/python3.11/site-packages/impacket/krb5/kerberosv5.py", line 185, in getKerberosTGT
    r = sendReceive(message, domain, kdcHost)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/git/public/BloodHound.py/.venv/lib/python3.11/site-packages/impacket/krb5/kerberosv5.py", line 65, in sendReceive
    raise socket.error("Connection error (%s:%s)" % (targetHost, port), e)
OSError: [Errno Connection error (rdc02.dev.com:88)] [Errno -2] Name or service not known

WARNING: Failed to get Kerberos TGT. Falling back to NTLM authentication. Error: [Errno Connection error (rdc02.dev.com:88)] [Errno -2] Name or service not known
DEBUG: Using LDAP server: rdc02.dev.com
DEBUG: Using base DN: DC=dev,DC=com
DEBUG: Using kerberos KDC: rdc02.dev.com
DEBUG: Using kerberos realm: dev.COM
INFO: Connecting to LDAP server: rdc02.dev.com
DEBUG: Using protocol ldap
DEBUG: Authenticating to LDAP server with NTLM
ERROR: Failure to authenticate with LDAP! Error 8009030C: LdapErr: DSID-0C0906C2, comment: AcceptSecurityContext error, data 52e, v4563
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/kali/git/public/BloodHound.py/bloodhound/__init__.py", line 343, in main
    bloodhound.run(collect=collect,
  File "/home/kali/git/public/BloodHound.py/bloodhound/__init__.py", line 78, in run
    self.pdc.prefetch_info('objectprops' in collect, 'acl' in collect, cache_computers=do_computer_enum)
  File "/home/kali/git/public/BloodHound.py/bloodhound/ad/domain.py", line 572, in prefetch_info
    self.get_objecttype()
  File "/home/kali/git/public/BloodHound.py/bloodhound/ad/domain.py", line 261, in get_objecttype
    self.ldap_connect()
  File "/home/kali/git/public/BloodHound.py/bloodhound/ad/domain.py", line 72, in ldap_connect
    ldap = self.ad.auth.getLDAPConnection(hostname=self.hostname, ip=ip,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/git/public/BloodHound.py/bloodhound/ad/authentication.py", line 175, in getLDAPConnection
    raise CollectionException(
bloodhound.ad.utils.CollectionException: Could not authenticate to LDAP. Check your credentials and LDAP server requirements.

 I need to put inside my /etc/resolv.conf at first the first dc as nameserver and append @home.com to the username and it will work.


Adding A Note Taking Magic Website In Java Script

Merge My small project for Practicing Java Script

@Asim-Amitav-Swain I respect your coding skills and your contribution to the open-source.
but it is a project and we need simple and easy beginner-friendly programs.
so try to make pr with a simple and unique program that helps beginners to understand basic concepts.
till then Happy Hacking 😊!!


Error: invalid value for api_key

Terraform version
Terraform v0.14.3

provider registry.terraform.io/cloudflare/cloudflare v2.14.0
provider registry.terraform.io/hashicorp/aws v3.22.0

Affected resource(s)

provider registry.terraform.io/cloudflare/cloudflare v2.14.0

Terraform configuration files
provider "cloudflare" {
  email   = "var.cloudflare_email"
  api_key = "var.cloudflare_api_key"
  account_id ="var.cloudflare_account_id"
}

cloudflare.tfvars
cloudflare_api_key starts with number and lowercase letters
// Cloudflare API KEY
variable "cloudflare_api_key" {
  default = "<SNIP>"
}

Debug output
Error: invalid value for api_key (API key must only contain characters 0-9 and a-f (all lowercased))
2020-12-20T02:38:12.244Z [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/cloudflare/cloudflare/2.14.0/linux_amd64/terraform-provider-cloudflare_v2.14.0 pid=55950
2020-12-20T02:38:12.244Z [DEBUG] plugin: plugin exited
2020-12-20T02:38:12.244Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020-12-20T02:38:12.245Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020-12-20T02:38:12.248Z [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/3.22.0/linux_amd64/terraform-provider-aws_v3.22.0_x5 pid=55943
2020-12-20T02:38:12.248Z [DEBUG] plugin: plugin exited

Panic output
n/a
Expected behavior
Cloudflare should accept API key generated in portal (I'm using the global API key from the portal).
This also fails when using api_token in like issue 842
Actual behavior
Fails with error Error: invalid value for api_key (API key must only contain characters 0-9 and a-f (all lowercased))
cloudflare_api_key starts with number and lowercase letters
Steps to reproduce

terraform apply

Important factoids
Using Terraform on Oracle Cloud VM (always free), running Ubuntu 20.04.
Removed snap Terraform  package and manually downloaded terraform_0.14.3_linux_amd64.zip and dropped into /bin/
References
https://github.com/cloudflare/terraform-provider-cloudflare/issues/842

In your test case, you're not referencing a variable; you're providing the literal string value "var.cloudflare_api_key". Remove the quotes and you will be referencing your variable value.


[Feature][connector-tidb][PIW] add tidb connector

add tidb connector which is referring jdbc connector and do some simplification such as removing postgre dialect etc.

Purpose of this pull request
refer to https://github.com/apache/incubator-seatunnel/issues/2173

Check list

[ ] Code changed are covered with tests, or it does not need tests for reason:
[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

@lhyundeadsoul  thanks for your contribution
Is there any difference with the JDBC connector currently, or is there any optimization point。Currently looks the same as the JDBC connector

We have discussed this in https://github.com/apache/incubator-seatunnel/issues/2173  ,but we have no further idea. TiDB obeys MySQL protocol, indeed it can handle by JDBC connector. So I did some simplification, such as removing MYSQL Dialect etc. If there are some special optimization for TiDB , we can add them based on this module.
Is there any good idea ,please let me know. Thx!

If it is only based on the jdbc connector, it is better to support TiDB directly in the JDBC connector. You now have a lot of the same code for the tidb connector and the JDBC connector,It's not good for maintenance.

If it is only based on the jdbc connector, it is better to support TiDB directly in the JDBC connector. You now have a lot of the same code for the tidb connector and the JDBC connector,It's not good for maintenance.

I think so.


chore(graph-gateway): rename network module entities to snapshot

Another network module API clean up PR to reduce noise in #750

Part of https://github.com/edgeandnode/gateway/issues/791


Decide finished goals

Acceptance Criteria

[ ] It's done when the / homepage route renders a list of all projects from the database.

[ ] It's done when the /project/:id route renders an individual project's details based on the route parameter id.

[ ] It's done when the /login route renders a form to log in and a form to create a new account.

[ ] It's done when an existing user can enter their credentials on the login page to create a session on the server.

[ ] It's done when a new user can create an account on the login page and then be immediately logged in with a session.

[ ] It's done when the /profile route renders the logged-in user's projects and a form to create a new project.

[ ] It's done when only a logged in user can visit the /profile route.

[ ] It's done when a logged in user is redirected to /profile when they try to visit /login again.

[ ] It's done when a user on the profile page can use the form to create a new project in the database.

[ ] It's done when a user on the profile page can select a "Delete" button to remove their project from the database.

[ ] It's done when a logged-in user can select a "Logout" button to remove their session.

[ ] It's done when the session for a logged-in user expires after a set time.

[ ] It's done when the API routes to create and delete posts are protected from non logged-in users.

[ ] It's done when the code is organized using MVC architecture.

[ ] It's done when the views are rendered with Handlebars.js templates.


chore(invariant): remove persist_state config, commit by default

Motivation
Closes #7268
Closes #4994

Solution

Remove preserve_state config and commit call by default - this will enable using vm.warp and vm.roll in invariants tests without any additional config (which is expected behavior)
From test result it can be seen there is a slightly penalty from persisting cheatcode state (not something to worry about IMO).
Tests
Results were collected by performing different number of runs and depth on a test contract with:

handler that advance block timestamp by 100 and increments block number on each call by using vm.warp and vm.roll  (worst case scenario as persisting cheatcode states is now done by default)
target contract with single selector that makes several state updates

Test contract
contract CommitState {
    uint256 state;
    mapping(address => uint256) public state1;
    mapping(address => mapping(address => bool)) public state2;

    function saveState(uint256 x, address y, address z) public {
        state = x;
        state1[y] = x;
        state2[y][z] = true;
    }
}

contract CommitStateHandler is Test {
    CommitState target;

    constructor() {
        target = new CommitState();
    }

    function fuzzTarget(uint256 x, address y, address z) public {
        vm.warp(block.timestamp + 100);
        vm.roll(block.number + 1);
        target.saveState(x, y, z);
    }
}

contract CommitStateTest is Test {
    function setUp() public {
        CommitStateHandler handler = new CommitStateHandler();
        targetContract(address(handler));
    }

    function invariant_commit_state() public view {}
}

Runs/Depth
time with Master
time with PR

100/1000
0m18.601s
0m19.587s

1000/1000
2m54.369s
3m12.411s

1000/5000
15m21.012s
15m55.039s

1/10000
0m10.528s
0m10.827s

10/10000
0m27.174s
0m29.075s

100/10000
3m18.966s
3m26.910s

10000/100
2m55.770s
3m1.623s

time contains test contract compile time too

hey @pcaversaccio wanted to let you know that this change breaks sneakmate invariant invariantExecutingNotReadyProposal because time is now preserved between calls
Let's say there's a first run calling invariantExecutingNotReadyProposal
https://github.com/pcaversaccio/snekmate/blob/e20185c1776256984d888d02dc243fe793119f2e/test/governance/TimelockController.t.sol#L4352-L4368 which calls TimelockControllerHandler.execute and properly reverts with "TimelockController: operation is not ready". However, TimelockControllerHandler.execute already does
        vm.warp(block.timestamp + minDelay);

before calling timelockController.execute (which reverts), see https://github.com/pcaversaccio/snekmate/blob/e20185c1776256984d888d02dc243fe793119f2e/test/governance/TimelockController.t.sol#L4424-L4433
Therefore  2nd time when  invariantExecutingNotReadyProposal is called time has properly passed (advanced from prev call) and proposal will pass / call won't revert (see logs here snekmate.txt). I think it's safe to assume a fix is to just remove the warp call or make sure you're not advancing time so much as the proposal becomes executable.

@grandizzy thanks for the heads-up; what's the time plan on the release on this breaking change?

@grandizzy thanks for the heads-up; what's the time plan on the release on this breaking change?

@pcaversaccio I am checking with other projects how this affects their usage, so probably 1 week or so

Addressed via https://github.com/pcaversaccio/snekmate/pull/241 in snekmate.

ping @PaulRBerg @lucas-manuel do you forsee any issue (like the one above) in your projects? Thank you

tagging @smol-ninja and @andreivladbrg

btw, if you want to give it a quick check you can set preserve_state = true as this will be made default with this PR

@grandizzy we are good with this PR. I have tested and there are no issues with the change.

tested ok ajna, morpho and maple


[To Be Confirmed] Improperly trigger "pressed' signal when dragging buttons in SmoothScrollContainer

As a contrast, the button in original ScrollContainer will not trigger "pressed' signal when being dragged

Here is my reproduction project. Thanks.
SmoothScrollContainer.zip

From the official godot documentation:

The input event propagates up through the node tree until a node consumes it.

This means that this is standard behaviour for any control element and Godot's native scroll containers are handled differently in the background. Due to this a workaround is needed like disabling all input of the children nodes (Node.set_process_input(false)) until the smooth scroll container handled them and propergates them to the children.
If someone has a different solution let me know.

I could work out a solution where every control node child's focus gets removed when dragging. This obviously removes any possibility for the inner controls to react to the input of this type (like selecting text). Feedback would be great.
https://github.com/SpyrexDE/SmoothScroll/assets/57133330/2bd91ade-89e3-4f8b-8d92-2c537a900119

Thanks for th help.
In my case, I don't need to select any Label in the ScrollContainer, but I need the buttons not to be pressed when being dragged.
If there is any testing version with the walkaround, I can test it.
Thank You.

Sure, here is the demo project:
demo.zip
I would agree that not having drag input for child controls is not as bad as false input when dragging so I will release a new version of the plugin soon that will work this way. Nonetheless, I will keep an eye out for a better solution. Thanks for reporting this issue!


Upgrade Blaze Persistence to Quarkus 2.6.1.Final and generate proper metadata

As seen in https://github.com/quarkusio/quarkus-platform/pull/417, the Blaze Persistence Quarkus extension does not provide all the necessary metadata for modern Quarkus.
Quarkus 2 has been around for a long while now so let's make the extension more modern.
I'm going to work on this.

Fixed via https://github.com/Blazebit/blaze-persistence/pull/1416


Random ShortURLs plugin doesn't show default value

Upon first activation it should show the default value instead of a blank input:

(Also, not sure the HTML control is what we need here, a simple input text would be simpler. Was I the one to come up with this idea in the original plugin? Don't remember)

What should be the default value?

I feel like I might have added what you were asking in this PR https://github.com/YOURLS/YOURLS/pull/2767 I don't know if 5 is a good enough default. But, the workflow works as requested.


more docs

@SUPERCILEX do you think this would be helpful for you?

Oh snap, you can run the JSON gen command within the test? Yes, that's super useful, thanks!


finetuning with flexible observations and flexible action spaces

Thank you for open-sourcing the VLA model. This initiative will significantly foster the advancement of foundational models for robotics.
I'm curious whether you have plans to support fine-tuning with customizable/flexible observations (e.g. add proprio info, left/right wrist images, depth images) and action spaces (e.g. diffusion action head, 14 dims actions, etc), similar to the capabilities of the Octo model.
Any guidance on this matter would be greatly appreciated. Thanks!

I've also experienced fluctuations in action accuracy when fine-tuning the model.

the same issue here #29

Hi everyone, thanks for your questions!
It seems that #29 is closed now -- @pxlong I think #29 was related to some misunderstanding of how our action tokenizer works.
With regards to your original question: we have not experimented with finetuning to new action or observation spaces for OpenVLA, but I would expect it to be possible. Adding proprio should be a small code change to the RLDSTransform. Adding another image input may require a bit more code change. If it's a possibility for you, you could also try "collaging" multiple images into OpenVLAs single-image input simply by concatenating them and resizing the concatenated image to 224x224 -- how reasonable that is depends on your particular setup.
Diffusion head etc is also possible, but not something we have experimented with / have code for.
@marinabar regarding your learning curve it's a bit hard to debug without knowing your data better.
One thing to note is that our loss mask specifically applies loss only on the last 7 action tokens, so if your action space has a different dimension it may not get fully trained?
A simple sanity check could be to train on a very small dataset (eg 10 transitions) and making sure that the model can fully overfit that. If it can't, it's most likely either an issue in the loss masking or your training data is corrupted / impossible to fit.

(feel free to reopen the issue, just closing to keep track what's answered)

@kpertsch Would you have an example of your accuracy curve during finetuning ? And isn't the mask already based on the taking into account the number of action tokens, as from what I understood it ignores the tokens that are not label predictions, but those labels are already only predicting tokens based on action dimensions and ignoring the rest ? with :  labels[: -(len(action) + 1)] = self.IGNORE_INDEX and then
action_gt = batch["labels"][:, 1:].to(action_preds.device)
mask = action_gt > action_tokenizer.action_token_begin_idx

I am finetuning it for the Aloha simulation environment, with about 400 frames per episode and 50 episodes.
I have also used the new learning rate you suggested in the updated finetune.py, but it did not help.
On 10 trajectories, the policy fully overfitted the data in 200 steps, so the problem does not lie there :/
Would you have some recommendations on what else could be wrong?
Thanks for the tips !

@kpertsch Would you have an example of your accuracy curve during finetuning ? And isn't the mask already taking into account the number of action tokens, as from what I understood it ignores the tokens that are not label predictions, but those labels are already only predicting tokens based on action dimensions and ignoring the rest ? with :  labels[: -(len(action) + 1)] = self.IGNORE_INDEX and then
action_gt = batch["labels"][:, 1:].to(action_preds.device)
mask = action_gt > action_tokenizer.action_token_begin_idx

I am finetuning it for the Aloha simulation environment, with about 400 frames per episode and 50 episodes. I have also used the new learning rate you suggested in the updated finetune.py, but it did not help.
On 10 trajectories, the policy fully overfitted the data in 200 steps, so the problem does not lie there :/
Would you have some recommendations on what else could be wrong? Thanks for the tips !

Have you solved this problem?


Envvar for setting default language

It would be nice if I could set the default language for new pastes via a docker env var. Something like this maybe:
PASTEY_NEW_LANGUAGE_DEFAULT
Either put it at the top of the list (just below AUTO, I think AUTO if enabled should always be priority) above Plaintext and the first selected item, or make that language the default selected in the list.
Could also be nice to add an array var for reducing the language set in the new page (though, this wouldn't disable the language entirely, such as if someone uses the API and knows the values):
PASTEY_NEW_LANGUAGE_ENABLED
Me personally, I would only want to use PASTEY_NEW_LANGUAGE_ENABLED to reduce down to my preferred language if you can't/won't add something for setting the default language in envvar.

Perhaps a better name would be
PASTEY_DEFAULT_LANGUAGE
And optionally also:
PASTEY_ALLOWED_LANGUAGE
Here is my first pass at what to add in the new.html template in place of the existing for loop.
{% for language in languages %}
{% if language == config.default_language %}
<option value="{{ language }}" selected>{{ language }}</option>
{% else %}
<option value="{{ language }}">{{ language }}</option>
{% endif %}
{% endfor %}

But I know that config.default_language is wrong. Can't figure out what it should be. But I know that swapping config.default_language out for "COBOL" makes the COBOL language be the select item on page load.


RabbitMQ Messaging support

This Epic will track the RabbitMQ messaging functionality.

Completed in 3.0.0 milestones 2 and 3 - #244, #273, #278 (80% completed)
3.0.0-rc1 will contain - #278 and #289

Closing Epic - Any other changes will be reflected in next release in #400.


cmd/go: suggest 'go get' instead of 'go mod tidy' when indirect dependencies are missing

(Noticed while testing #45979.)
If a lazy module (#36460) is missing explicit requirements on indirect dependencies, we currently emit the error message:
go: updates to go.mod needed; to update it:
	go mod tidy

This advice is correct, but it's too big a hammer — we should instead suggest go get on the specific packages within the main repo whose transitive dependencies are missing.
CC @jayconrod @matloob

@bcmills I'm encountering this issue. How do you recommend I approach this issue until it's fixed? how can I know what module is the problematic one?
Thank you

@dwertent, @meetme2meat: this issue is about changing which command we suggest when the go.mod file is inconsistent. (This is a UX issue, not a bug to be fixed or worked around.)

@bcmills Understood but now our Drone CI running under go 1.17 result in failure with the exact same message. Can you suggest what can be done here to have 1.17 (inside CI) not exit out with above error.

Run go mod tidy, as the error message says, and then commit the result. If that doesn't resolve the error message, please file a new issue from the issue template (https://golang.org/issue/new).

(CC @matloob @samthanawalla)


add x86 version

please added a x86 version support
thank you

With the same online waiting, urgent

sorry this took so long