#516 Breaks buildpack when parameters < 5

#516 buildpack/stage.py line 30 declares deps_dir only when parameters >=5
line 169 calls function set_up_launch_environment() (line 101) whether this variable is set or not. The functions then assumes that the variable would be declared (line 104)
util.set_up_launch_environment(DEPS_DIR, PROFILE_DIR)
at this point buildpack fails with NameError: name 'DEPS_DIR' is not defined
i'm sorry but I don't yet have the experience with the buildpack to know whether conditionally calling set_up_launch_environment would be a valid solution. For context I am attempting to call the cf-mendix-buildpack from the docker-mendix-buildpack

Hi @JWHolding - thanks for reporting this. As we can't support the Docker buildpack in this repository, please open an issue there.

Hi Its not an issue with the docker buildpack, just provided that for context, is the DEPS_DIR a required input? if not then the code in stage.py should deal with that


Acefit extension

@cortner This adds training support for AtomsBase input. It has only overloads the function in ACEfit. Later on we can support for ACEpotentials specific functions.
There was some issue with implementing this. I did stumble on a bug that caused assemble to give some random results (can be triggered by changing pmap in asseble to Folds.map). This is probably some bug in ACE data allocation, but it can also be in Julia general - I don't understand how the bug works. But I am very concerned of it, as appear somewhere else and possible changes in Julia can cause it to emerge in the existing "working" code.
Here is a example code and comparison to old way
using ACEmd
using ACEfit
using ACEpotentials

model = acemodel(
    elements = [:Ti, :Al],
	order = 3,
	totaldegree = 6,
	rcut = 5.5,
	Eref = [:Ti => -1586.0195, :Al => -105.5954]
)
basis = model.basis

data_j, _, meta = ACEpotentials.example_dataset("TiAl_tutorial")
weights = Dict( "default" => Dict("E" => 5.0, "F" => 1.0 , "V" => 1.0 ) )
datakeys = (energy_key = "energy", force_key = "force", virial_key = "virial")
train_old = [ACEpotentials.AtomsData(t; weights=weights, v_ref=model.Vref, datakeys...) for t in data_j[1:5:end]]
train_new = [  FlexibleSystem(x.atoms) for x in train_old  ]

# Old style
A, Y, W = ACEfit.assemble(train_old, basis);

# New style
a, y, w = ACEfit.assemble(train_new, basis; energy_default_weight=5, energy_ref=model.Vref)

# Compare results
maximum(abs2, A - a)
maximum(abs2, Y - y)
maximum(abs2, W - w)

The new way is way more faster than the old one, between several tens to several hundred (if you don't use parallel processing). The is also a way to make it even faster, but I will skip for now. The main point is that assembly is now faster than solving the linear equation, changing the dynamics of fitting potentials completely.

Hi Teemy - Thank you for exploring this.  I'll will need to find time to go through the code very carefully. In the meantime can you please explain a few things?

How does the fitting data enter the new assembly? I don't see this. Does FlexibleSystem keep the metadata like reference energies and forces?
Similarly, how do the fitting weights enter the new assembly?
Can you explain briefly where the speedup comes from?

Regarding the bug, I have some doubts about there being a bug in ACE1 allocations, it's been in use for many years now, but you never know - as you say changes in Julia or somewhere else might bring it out. But possibly also some incorrect assumptions made about how temporary storage is to be pre-allocated and how outputs are allowed to be used. Either way, I'd be grateful if you can open a separate issue or PR with a reproducable example so we can investigate it.
CC @wcwitt - would you mind following this discussion as well?

another question - why add the example training data to the repo when you can just get it via ACEpotentials.example_dataset?

How does the fitting data enter the new assembly? I don't see this. Does FlexibleSystem keep the metadata like reference energies and forces?

Yes, AtomsBase is defined so that you can store whatever data in the structures. The documentation page of this PR has the details. But in short, if data is AtomsBase structure is should have training data behind keys: data[:energy], data[:force] and data[:virial]. ExtXYZ does have a bug at the moment that makes it fail to read some of this data, like force, when used for AtomsBase structure (works for JuLIP), so I need to submit a PR for it, to get AtomsBase interface to work fully.
Fitting weights are also explained in the documentation page. In short, general weights are given as keywords, like in above example. In addition you can give each structure a separate weight parameter that is used only for that structure.

Can you explain briefly where the speedup comes from?

I am not completely sure.  Here is a benchmark comparison with the above posted example
# old impelentation
julia> @benchmark ACEfit.assemble($train_old, $basis)
BenchmarkTools.Trial: 1 sample with 1 evaluation.
 Single result which took 14.657 s (98.42% GC) to evaluate,
 with a memory estimate of 445.05 MiB, over 5253010 allocations.

# new implementation
julia> @benchmark ACEfit.assemble($train_new, $basis; energy_default_weight=$5, energy_ref=$model.Vref)
BenchmarkTools.Trial: 32 samples with 1 evaluation.
 Range (min … max):  136.020 ms … 180.292 ms  ┊ GC (min … max): 18.63% … 16.11%
 Time  (median):     156.335 ms               ┊ GC (median):    23.22%
 Time  (mean ± σ):   157.097 ms ±   9.635 ms  ┊ GC (mean ± σ):  22.56% ±  2.36%

                     █ ▃▃▃             █   █
  ▇▁▁▁▁▁▁▁▁▇▁▁▇▁▇▁▇▁▁█▇███▁▁▁▇▇▇▁▁▇▇▇▁▇█▁▁▁█▇▇▁▁▁▁▁▁▁▁▁▁▁▁▇▁▁▁▇ ▁
  136 ms           Histogram: frequency by time          180 ms <

 Memory estimate: 1.03 GiB, allocs estimate: 5453582.

Use of multiprocessing in the old implementation makes it hard to see, where the difference comes. But it is probably the garbage collection.

why add the example training data to the repo when you can just get it via ACEpotentials.example_dataset?

It is the first 3 structures from the ACEpotentials training data. Its function is to implement tests for correctness of assembly. I just thought that it is so little data that it was better to just add here is as a copy rather than load it over network every time you do tests.

Can someone explain the context here? Is the aim to replace JuLIP?

The new way is way more faster than the old one, between several tens to several hundred

I am fully prepared to believe this is possible, but we should agree on some larger benchmarks. This could become one: https://github.com/ACEsuit/ACEfit.jl/pull/54#issuecomment-1644079554.
The assembly, particularly in parallel, has felt confusingly sluggish to me for some time. I've been attributing it to something in ACE1/JuLIP (perhaps involving GC), where I've been reluctant to dig deeply, but not sure.

Yes we want to kill JuLIP by moving small pieces into ACE packages or into community packages.

I also had the impression that Teemu s weights are I compatible with ours so this may need to be fixed. But that’s a minor issue.

@tjjarvinen -- can you also please open a separate issue for the bug you found? If we trace it to ACE1 then I need to look into it.

One small remark, to make the tests entirely fair one should remove the cached neighbourlist after each assembly,
rm_nlist(at::Atoms) = delete!(at.data, "nlist:default")
rm_nlist.(data_j)

but this doesn't actually seem to make a measurable difference.

If this speedup holds, I'm excited, because it likely also means better strong scaling. Currently, we are at least a factor of 5 worse in that dimension than we should be, perhaps more.

The numbers change a bit when you make the model bigger
# length(basis) = 4676,
# length(data_j) = 329
 14.784368 seconds (14.64 M allocations: 8.669 GiB, 64.56% gc time)
  3.377460 seconds (18.59 M allocations: 30.495 GiB, 25.45% gc time)

# length(basis) = 12888
# length(data_j) = 329
 26.681512 seconds (25.48 M allocations: 23.483 GiB, 38.05% gc time)
  9.563755 seconds (37.73 M allocations: 83.549 GiB, 23.07% gc time)

Hm, am I correct that the new assembly routine does not force garbage collection after each structure, unlike the old one? That is likely a factor - perhaps the main factor - particularly for small tests.
We know forcing GC hurts performance, but it has been the only way to avoid crashes (for both distributed and threaded) during larger assemblies.

What if we add a counter and GC every once in a while?

Yeah, perhaps. Or I will merge in one of the batching ideas and we can GC after batches.
Still holding out hope this PR fixes the underlying issue somehow - let's see.

Just tried the small benchmark again (from the top of this thread):
# old (looks insane, but I triple checked)
20.635631 seconds (5.49 M allocations: 786.066 MiB, 97.01% gc time)
# old, without forcing GC
0.605159 seconds (5.49 M allocations: 786.014 MiB, 17.42% gc time)
# new
0.747427 seconds (5.61 M allocations: 1.094 GiB, 17.34% gc time)

So for the tiny model the forced GC explains everything.
Same trend, but less definitive for the bigger model:
length(basis) = 12888
length(train_old) = 66

# old
82.048485 seconds (31.27 M allocations: 41.684 GiB, 29.60% gc time)
# old, without forcing GC
62.509240 seconds (31.27 M allocations: 41.684 GiB, 3.69% gc time)
# new
49.417405 seconds (45.56 M allocations: 81.406 GiB, 2.45% gc time)

That points that the issue might be with the communication. For bigger system more time is spend on ACE evaluations, which will overshadow the other parts of the code. The new code has still unused optimizations that can be done, so we should get more out of it once they have been added.
The issue with the old version is that there is a Distributed array to which all processes write at the same time. This by definition needs that there has to be some kind of locking and broadcasting of changes made to the array. This is probably the point where the issue comes. However this would be the case only when there is more than 1 process, but the issue is present even with one process.
One point to note here is that you cannot implement the old asseble version with Rust, because in Rust only the owner can change the content and thus the compiler would not allow you to compile. This is in general a huge red fag that there are possible issues in the code and in general should be avoided.

I added support for customizing data keys
ACEfit.assemble(data, basis;
    energy_key=:energy,
    force_key=:force,
    virial_key=:virial
)

Changing those should allow you to use the other data.

I added a small performance improvement that is in use when both force and virial are calculated. The basis here is that both need evaluate_d call and instead to calling it two times, just collect force and virial from the same call. This nearly eliminate the time needed for virial calculation. It is about one third off from the time, so not much, but also it was very easy to do.

Yes - caching virials and forces is a long-standing issue :) - thanks for solving this here. Though we should really have this at a more general level. Until then this is a great improvement.

Regardless of understanding the origin of the improvement, can we please agree on a few tests that need to pass before we can make these changes the default for ACEfit and ACEpotentials

We should not hurry for the default yet. The best way is to add this as an alternative first, for some testing period. During this period we tell people to try it and once we are comfortable we can move it to be the default.
I also need to fix the issue with ExtXYZ until we can go for full AtomsBase workflow too. But I should be able to fix that this week, after which it needs for James to release a new version, but that should not be an issue.

I understand the reasoning, but I don't think it's the main bottleneck (yet). In the past, I've tested by eliminating the write step, such that the assembly blocks are created and then immediately discarded, but the times don't improve significantly. There is some discussion of that here (but in the threading context): ACEsuit/ACEfit.jl#49 (comment)

Yes I know, there is something that I don't understand either. But it is somehow related to the way data is moved/transformed...

We should not hurry for the default yet.

fair enough but it still needs equivalent functionality and identical results on a range of tests.


MNT: rerender

Hi! This is the friendly automated conda-forge-webservice.
I've rerendered the recipe as instructed in #24.
Here's a checklist to do before merging.

[ ] Bump the build number if needed.

Fixes #24

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Seen in https://github.com/conda-forge/tokenizers-feedstock/pull/48
The reported errors are:
- Encountered problems while solving:
-   - nothing provides zipfile-deflate64 needed by py7zr-0.18.0-pyhd8ed1ab_0

Hopefully a rerender helps

@conda-forge/py7zr this is green, PTAL! :)


Moving from redux-router to react-router-redux

I'm facing an issue with redux-router. I need to access the store's state in a route hook (onEnter).
It is not possible with redux-router as stated here : https://github.com/acdlite/redux-router/issues/32
So, i get the error about the fact that i'm trying to access the store state even if i don't.
Another interesting read is : https://github.com/acdlite/redux-router/issues/172
Maybe is it time to move on?

What exactly would you do in onEnter?

I fixed the issue in redux-router.
https://github.com/acdlite/redux-router/pull/272
You can leave a comment there if you want to speed up the merging process.

In the meanwhile I'll be migrating to react-router-redux just for fun

I investigated the possibility of moving to react-router-redux and it's currently not possible because it fires before route matching is performed where as redux-router does it after.
This is not critical, but the code style would change: instead of just @preload() it would be:
<Route component={Page} onEnter={Page.preload}/>

Which is more verbose.
While the aforementioned PR is being considered a workaround is to run the code inside onEnter in a setTimeout(..., 0) wrapper.

In version 5.0.2 I added onEnter substitute
https://github.com/halt-hammerzeit/react-isomorphic-render#onenter

Version 9.0.0 no longer uses redux-router and seems to work. See HISTORY for more details.


KAFKA-3265: Create Java Admin Client

Create a Java AdminClient as described in KAFKA-3265 and KIP-4.
Currently it supports getAllGroups, getAllBrokerVersions, deleteTopics, and createTopics.  For each API, it uses the associated Kafka requests rather than going through ZooKeeper.
It is called AdministrativeClient to avoid confusing with the scala.kafka.admin.AdminClient class.

Good to see some progress on KIP-4! One question - does this allow new topics to be created with the cluster defaults for number of partitions and replication factor? Having to have a priori knowledge of those configs on the client is a big pain point for us using AdminUtils and I thought the elimination of that requirement was a goal.

Hi @twbecker, thanks for taking a look!  Be sure to also check out the discussion thread here<EMAIL_ADDRESS>Unfortunately, as far as I know, there is no way to allow the server to use a default replication factor when making a CreateTopicsRequest.  The RPC must have either a manual partitioning assignment, or a number of replicas and number of partitions specified.
I think supporting server-side defaults would be a nice improvement, but it would require a separate KIP to implement.

3 test failures:
kafka.api.SaslSslAdminClientIntegrationTest.testGetAllBrokerVersions
kafka.api.SaslSslAdminClientIntegrationTest.testListNodes
kafka.api.SaslSslAdminClientIntegrationTest.testCreateDeleteTopics

I fixed the test failure and some trivial clean-ups:
https://github.com/cmccabe/kafka/pull/4
System tests passed:
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/866/
I think we can merge this after the PR above is merged. I will file a JIRA with additional clean-ups that I noticed, but those can be done later.

Thanks, @ijuma.  The cleanups look good to me.

retest this please

retest this please


Bombs Materialized perk does not sync

Ex: if the client picks up the perk, throws a bomb, the host cannot see the bomb nor the explosion.

disabled for now, i put all #116


demonstrate how to create and load custom plottables

It should be possible to create a new custom plottable, add it to the list, and have it plotted on ScottPlot without changing any ScottPlot code. Create a demo program which shows how to do this.
The Plot module lets me get the list of plottables. I should be able to just add my custom plottable to this list and it will appear.
https://github.com/swharden/ScottPlot/blob/aad44762d282d24d6d7e49e74ccf9c04c3f23f40/src/ScottPlot/Plot.cs#L708-L711

That is already working. I implemented a custom plot type (filled  multisignal envelope) exactly in this way.  Just implement Plottable , add it exactly to the named list and you're done.
Current issue though is, that ScottPlot.Plot.Clear() does not clear custom plottables.

... it was not totally obvious how to ...

Yeah, it's possible as-is but a little clunky. I think a concise quickstart guide to designing custom plottables would be nice!
Also a new function like Plot.AddPlottable(IPlottable customPlottable)

Here's an example. I create a new Plottable totally outside the ScottPlot namespace:
https://github.com/swharden/ScottPlot/blob/af33308f6f3a7f90079fe487e21f3de33a7b4f5a/tests/Experimental/BarGraphTests.cs
Then I plotted it with the new Plot.Add() function:
https://github.com/swharden/ScottPlot/blob/af33308f6f3a7f90079fe487e21f3de33a7b4f5a/tests/Experimental/BarGraphTests.cs#L95-L99


celix agent does not start

core@controller ~ $ kubectl logs inaetics-processor-celix-controller-ifq71
[DEBUG] -> etcd/values - args: /inaetics/node-provisioning-service
[DEBUG] -> etcd/keys - args: /inaetics/node-provisioning-service <IP_ADDRESS>:2379
[DEBUG] -> etcd/value - args: /inaetics/node-provisioning-service/6d72b9d586c2c7c0090dbe696fe4e123cd2d904f982f854731d677405350c618 <IP_ADDRESS>:2379
Provisioning service changed:  -> <IP_ADDRESS>:8080
Starting agent...
[INFO] CELIX Configuration
[INFO] =================================================
[INFO] RSA IP s : <IP_ADDRESS>
[INFO] DISCOVERY_ETCD_SERVER_IP : <IP_ADDRESS>
[INFO] DISCOVERY_ETCD_SERVER_PORT : 2379
[DEBUG] celix
agent running with provisioning <IP_ADDRESS>:8080
[DEBUG] -> etcd/putTtl - args: /inaetics/node-agent-service/celix_3 <IP_ADDRESS>:8080 75
[DEBUG] <IP_ADDRESS>:8080
Launcher: Framework Started
-> -> -> -> -> -> -> -> -> -> -> -> LogWriter: BUNDLE_EVENT_STARTED from apache_celix_log_service
LogWriter: BUNDLE_EVENT_RESOLVED from apache_celix_log_writer
LogWriter: BUNDLE_EVENT_STARTED from apache_celix_log_writer
LogWriter: BUNDLE_EVENT_RESOLVED from apache_celix_remote_shell
LogWriter: BUNDLE_EVENT_STARTED from apache_celix_remote_shell
LogWriter: BUNDLE_EVENT_RESOLVED from apache_celix_shell
LogWriter: BUNDLE_EVENT_STARTED from apache_celix_shell
LogWriter: BUNDLE_EVENT_RESOLVED from apache_celix_shell_tui
LogWriter: BUNDLE_EVENT_STARTED from apache_celix_shell_tui
LogWriter: Remote Shell accepting connections on port 6666 from apache_celix_remote_shell
[INFO] Pair </inaetics/node-agent-service/celix_3,<IP_ADDRESS>:8080> stored in etcd
-> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> Will update in 60 seconds...
-> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> ->
[...]
-> -> -> ERROR: File I/O exception [70008]: "Failed to delete tree"
	at deploymentAdmin_deleteTree(/usr/buildroot-2015.05/output/build/celix-Latest/deployment_admin/private/src/deployment_admin.c:450)
LogWriter: CONNECTION_LISTENER: Stopping thread
 from apache_celix_remote_shell

https://github.com/INAETICS/node-agent-c/commit/8eb09637e4b2ed699c8f8e148071f2997f179634 and https://github.com/INAETICS/node-agent-c/commit/a248a6334ca343d1837084c86661a0d9904c5d6b fixed the problem.


Lazy loading (loadChildren) modules with module mapping

Bug Report or Feature Request (mark with an x)
- [x] bug report -> please search issues before submitting
- [ ] feature request

Command (mark with an x)
- [ ] new
- [ ] build
- [x] serve
- [ ] test
- [ ] e2e
- [ ] generate
- [ ] add
- [ ] update
- [ ] lint
- [ ] xi18n
- [ ] run
- [ ] config
- [ ] help
- [ ] version
- [ ] doc

Versions

➜ node --version
v10.11.0
➜ npm --version
6.4.1
➜ ng --version

     _                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/

Angular CLI: 7.1.2
Node: 10.11.0
OS: linux x64
Angular: 7.1.2
... cli, common, compiler, compiler-cli, core, forms, http
... language-service, platform-browser, platform-browser-dynamic
... router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.11.2
@angular-devkit/build-angular     0.11.2
@angular-devkit/build-optimizer   0.11.2
@angular-devkit/build-webpack     0.11.2
@angular-devkit/core              7.1.2
@angular-devkit/schematics        7.1.2
@ngtools/webpack                  7.1.2
@schematics/angular               7.1.2
@schematics/update                0.11.2
rxjs                              6.3.3
typescript                        3.1.6
webpack                           4.23.1

Repro steps

Clone the sample repo: https://github.com/wescoder/angular-lazy-modules [feature/scaffold branch].
Run with npm start.
See the console logs.
The log given by the failure

ERROR Error: Uncaught (in promise): Error: Cannot find module<EMAIL_ADDRESS>Error: Cannot find module<EMAIL_ADDRESS>
Desired functionality

Using module mapping with typescript should map correctly in lazy loaded modules on angular.
Mention any other details that might be useful

https://github.com/wescoder/angular-lazy-modules

Hey @wescoder thanks for the example repo. It doesn't look like an Angular CLI project and it doesn't have npm start script in package.json. Do you have a CLI example where we can reproduce the problem?

@mgechev sorry, I hadn't made the push with the addition of angular to the project.
I have made it now, thanks for the heads up.

Would it be possible to bootstrap a fallback module on the .catch of the platformBrowserDynamic().bootstrapModule(), without running into this issue ?
eg:
platformBrowserDynamic().bootstrapModule(ModuleX).catch(()=>{
return platformBrowserDynamic().bootstrapModule(ModuleY);
})
I can make the app work , however when compiling I get the error on this thread and the dynamic module resolution won't work (from experience in another app, not this example )
Documentation is unhelpful and still trying to work my way on source code, so  Im reaching out for help. Any ideas are welcome


Add higher dimension matrix functions

See also secondary loss

Retiring as this can be dealt with via metamodels.


[ADD] 1inch and LiFi support

please add 1inch and LiFi support for swaps

duplicate of #465 #466


Fix get_ssh_key to use absolute path

Also a fix for copying delete_rancher_server test results
@sangeethah please review

LGTM


Magazine box deconstruct

About the PR

Add deconstructing magazine boxes into cardboard.
Why / Balance

Technical details

Fixes SpawnPrototype spawning items attached to the player.

This adds a new ConstructionGraphStep type: InteractionGraphStep that allows for advancing a construction graph without using a material or tool.
The construction step advances when the entity receives a ConstructionInteractionEvent. This code sends a ConstructionInteractionEvent when a verb is pressed in ConstructionInteractionVerbSystem.

Added a ConstructionGraphCondition RMCItemSlotsEmpty that checks whether all the entities ItemSlots are empty.
Media

[x] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

https://github.com/RMC-14/RMC-14/assets/10494922/2499038c-8311-465d-af04-104901daf81c
Breaking changes

Changelog

:cl:

add: Magazine boxes can now be deconstructed into cardboard using the fold box verb.

remove the doafter and then bingo
does this work if the box is not placed? and can you altclick it / activate in hand to fold it?

It works anywhere as long as the box is empty. When I asked in Discord, I was told to only make it a verb.
Currently:

ActivateInHand deploys the box on the ground.
Alt+Interact doesn't do anything when empty, but if there are magazines, it pulls one out.

I was told to only make it a verb.
a

Linter is failing
Error: /Prototypes/_RMC14/Entities/Objects/Weapons/Guns/Ammunition/Boxes/magazine_box_crafting.yml(296,9)  Field "interaction" not found in "Content.Shared.Construction.Steps.InteractionGraphStep".

Fixed, though kinda hacky since the value is unused.

Are merge conflicts a big problem? I made it in construction because i wanted to upstream it soon too.
Just want to know whether to do upstream merge first or here first.

Are merge conflicts a big problem? I made it in construction because i wanted to upstream it soon too.
Just want to know whether to do upstream merge first or here first.

Upstream first but if you want it to get merged here quickly as well you could do it with a separate component/system, unless you really want to keep it as part of construction system.

Closing until upstream stuff is merged. Until then #2855 implements similar functionality.


Reland Replace ideviceinfo and idevice_id with xcdevice

Description
Reland https://github.com/flutter/flutter/pull/49854.  Ignore false positive "Preparing debugger support for iPhone" -10 error message since usually the app launches fine after giving that error.
Xcode ships with a command line tool called xcdevice that produces a json array of all available devices, including iOS devices that can be deployed via Xcode over the network (Connect via network checkbox).  This PR filters out these network devices, but it's relevant to someday fixing https://github.com/flutter/flutter/issues/15072.

I checked to make sure xcdevice is available in Xcode 10.2 (minimum Xcode required by Flutter) and later.
Example output:
{
  "simulator" : false,
  "operatingSystemVersion" : "13.3 (17C54)",
  "interface" : "usb",
  "available" : true,
  "platform" : "com.apple.platform.iphoneos",
  "modelCode" : "iPhone8,1",
  "identifier" : "d83d5bc53967baa0ee18626ba87b6254b2ab5418",
  "architecture" : "arm64",
  "modelName" : "iPhone 6s",
  "name" : "iPhone"
}

Since this provides the identifier, name, operating system, and architecture, we can get rid of all usages of idevice_id and ideviceinfo (cleanup happening in the next PR since this one was already getting too big).
Also, show any connection errors in doctor instead of crashing.  Example:
[!] Connected device
    • Error: iPhone is not paired with your computer. To use iPhone with Xcode, unlock it and choose to trust this computer when prompted. (code -9)

Remove 2 libimobiledevice dependencies.
Remove all codepaths that resulted in cryptic lockdown errors and instead return Xcode's nice errors and recovery suggestions, like "To use iPhone with Xcode, unlock it and choose to trust this computer when prompted."

Related Issues
Part of https://github.com/flutter/flutter/issues/6118
Precursor to https://github.com/flutter/flutter/issues/15072 (will make networked devices discoverable when we stop filtering "network" interfaces).
Fixes https://github.com/flutter/flutter/issues/49375
Fixes https://github.com/flutter/flutter/issues/36524
Fixes https://github.com/flutter/flutter/issues/47727
Fixes https://github.com/flutter/flutter/issues/43398
Fixes https://github.com/flutter/flutter/issues/48057
Supersedes https://github.com/flutter/flutter/pull/49626
Supersedes https://github.com/flutter/flutter/pull/45657
Supersedes part of https://github.com/flutter/flutter/pull/36902?
Tests
xcode_test, devices_test
Checklist

[x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
[x] I signed the [CLA].
[x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement].
[x] I read the [Tree Hygiene] wiki page, which explains my responsibilities.
[x] I updated/added relevant documentation (doc comments with ///).
[x] All existing and new tests are passing.
[x] The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change

[x] No, no existing tests failed, so this is not a breaking change.
[ ] Yes, this is a breaking change. If not, delete the remainder of this section.

Landed on red, fuchsia infra failure.

running sudo pkill usbmuxd worked for me on Mac OSX


Update devcontainer dockerfile

Issue Link 🔗:
Issue: #606
Type of Change

[ ] Bug fix 🐞
[ ] New feature/page
[ ] Documentation update
[x] Other

Description 📋
Enhanced docker devcontainer to be a base installation of Python and Ruby and then run pip install and ruby bundle and playwright install with dependencies and also included more descriptive comment messages
Checklist ✅

[x] Followed the Code of Conduct and Contribution Guide
[x] Ran pre-commit run --all
[x] All tests pass locally
[ ] Added tests (if applicable)
[ ] Documentation updated (if applicable)

Additional Notes & Screenshots
Add any additional notes or comments that might be helpful for the reviewers.

Small nitpick...
It seems that you stacked your changes. This means that you need to merge only one and all the changes will apply.
Please only merge the last PR otherwise you may run into merge conflicts. Close the others


Flannel RBAC Fix

Fixes a bug that can occur if cni-flannel-rbac.yml was written but the playbook failed before it was applied. Uses the same approach as calico.

ci check this

Passed CI, merging.


[Bug]: Fibaro nodes do not sent button events

Contact Details (optional)
No response
What happened?
I cannot recall which version update this caused. But the controller node as the devices nodes do no report physical button presses anymore. So this means when I use my physical switch (for example the kitchen) it does not report this to nodered.
So the situation was, the UI dashboard (and homekit) got synchronised when using the physical button because it would trigger a value change. This is not happening anymore, cause no value change event is sent. I really cannot recall when this issue was introduced.
How to reproduce?
Use the physical switch and see that it does not create a value change event.
Version
6.1.1 and 6.2.0
Node-RED Version
2.0.6
What hardware are you seeing the problem on?
Raspberry Pi
Relevant Z-Wave-JS log output
None thats the problem :)

Alternative attempt:
@zwave-js-bot add compat flag to 0x010f:0x0402:0x1002:2.2
// The device sends reports from endpoint 1 un-encapsulated
mapRootReportsToEndpoint: 1

@dionmes We need to test which approach is better here.
File 1: https://raw.githubusercontent.com/zwave-js/node-zwave-js/2f9af2d27449a821cb0ee92aa73b446563c216b3/packages/config/config/devices/0x010f/fgs212.json
File 2: https://raw.githubusercontent.com/zwave-js/node-zwave-js/41bb6ea1df02b6b2be614f202e4a5ed9917122a7/packages/config/config/devices/0x010f/fgs212.json
The node red plugin allows you to specify a directory for custom config files (Custom CFG Dir). Please put file 1 there, re-interview the node and check if your issue is resolved (best make a log of the switching attempts and post it).
If not, put file 2 there and do the same.

Ok,
The event mechanism is a simple subscription to value updated + others - which is the same for all nodes.
and you're getting them, apart from these Fibaro devices.
I'll wait for AI, as I am at a loss at this stage.

The issue started after an update of node-red-contrib-zwave-js

the only note worthy change here was a bump to zwave-js 8.5.0


fix(README): closing tag on ecosystem

Issue #-
Fix closing tag
Changes

fix a closing tag <td> -> </td>
this was causing some weird rendering

Flags

Related Issues

Issue #
Pull Request #

@daniloff200 flagging this for you.

oh
my
Lord
@irmerk  :worried:
Yeah, okay, I'm gonna fix that during the next week


add cache members on start option

use case?

I thought you mentioned that we should add an option to cache members on start in case users don't want to. If thats not the case, I will revert it.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Codecov Report

Merging #878 (3736de3) into main (dfe822f) will increase coverage by 0.00%.
The diff coverage is 29.72%.

@@           Coverage Diff           @@
##             main     #878   +/-   ##
=======================================
  Coverage   66.72%   66.73%
=======================================
  Files         315      315
  Lines        9666     9671    +5
  Branches      439      438    -1
=======================================
+ Hits         6450     6454    +4
- Misses       3207     3208    +1
  Partials        9        9

Impacted Files
Coverage Δ

src/structures/message.ts
54.54% <ø> (ø)

src/handlers/misc/READY.ts
39.56% <23.07%> (+3.84%)
:arrow_up:

src/structures/guild.ts
31.39% <25.00%> (-0.37%)
:arrow_down:

src/bot.ts
44.73% <50.00%> (+1.87%)
:arrow_up:

src/structures/member.ts
47.91% <100.00%> (-1.14%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dfe822f...3736de3. Read the comment docs.

Ahh no this isn't exactly what i meant. I meant we should consider whether or not we should force all members to be cached from intial queue thing before ready event is run

Closing this as it needs to be rewroked at a later time as discussed in discord


wsl --mount not working on arm64

Windows Version
Microsoft Windows [Version 10.0.26100.2314]
WSL Version
<IP_ADDRESS>
Are you using WSL 1 or WSL 2?

[x] WSL 2
[ ] WSL 1

Kernel Version
<IP_ADDRESS>-1
Distro Version
Ubuntu 22.04
Other Software
Existing issue #12038 was closed after waiting for logs.
WSL2 is from Windows Store and is up-to-date (ran wsl --update to be sure).
Windows 11 is up-to-date (ran Windows Update check for updates to be sure).
$ wsl.exe --version
WSL version: <IP_ADDRESS>
Kernel version: <IP_ADDRESS>-1
WSLg version: 1.0.65
MSRDC version: 1.2.5620
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26100.1-240331-1435.ge-release
Windows version: 10.0.26100.2314
Repro Steps
With WSL2 IO performance (and the Linux file security model) work much better for the native filesystem (ext4) though there is very much appreciated access to the Windows NTFS filesystems through the drvfs filesystem driver albeit at a slower IO rate.
To reproduce:

WIndows 11 on ARM64 machine (Microsoft Surface Pro 11 in my case).
WSL2/Ubuntu 22.04 installed from Microsoft Store.
Use Windows Disk Management to Create VHD outside of WSL2, in my case on external D: drive connected via USB.
In Powershell get the physical disk number by running
Get-Disk
Then try to mount the VHD by running
wsl.exe --mount \\.\PhysicalDrive# --bare

In my case the drive number of the new VHD was 2 so I tried running
wsl.exe --mount \\.\PhysicalDrive2 --bare
in a variety of contexts:

from my normal user in Ubuntu/WSL2
from the root user in Ubuntu/WSL2
from an Administrator elevated Powershell

I am generally trying to follow the steps outlined in https://joeferguson.me/adding-another-disk-to-wsl2/, though some of those steps are apparently outdated.  Like the process to get the VHD drive number, so I eventually figured out that (these days) a simple Get-Disk in Powershell will interactively provide the number.
Expected Behavior
New VHD will mount into Ubuntu where an ext4 filesystem can then be created and the resultant ext4 filesystem can then be mounted and used for normal file handling from that point forward.
So basically now a df command in Linux would show the usual / (root ext4) filesystem, all the drvfs filesystems exposing the various NTFS Windows "drives", plus this new ext4 filesystem which happens to be on a different physical disk than the root filesystem.  Expectation is that the IO performance of the new VHD would be similar to the root filesystem (after taking into account any physical limitations due to the disk characteristics and interfaces, e.g. SSD vs spinning, USB, etc).
Actual Behavior
PS C:\> wsl --mount \\.\PhysicalDrive2 --bare
wsl.exe --mount is not supported for ARM64 on this version of Windows.
Error code: Wsl/Service/WSL_E_WSL_MOUNT_NOT_SUPPORTED
also tried
PS C:\> wsl --mount --vhd D:\media\extdisk.vhdx --bare
wsl.exe --mount is not supported for ARM64 on this version of Windows.
Error code: Wsl/Service/WSL_E_WSL_MOUNT_NOT_SUPPORTED
Diagnostic Logs
Since this is a WSL --mount issue, tried to follow the guidance to run the wmic command but looks like that has been deprecated.
Running wmic diskdrive get Availability,Capabilities,CapabilityDescriptions,DeviceID,InterfaceType,MediaLoaded,MediaType,Model,Name,Partitions from an elevated command prompt resulted in -
'wmic' is not recognized as an internal or external command, operable program or batch file.
Attached logs from collect-wsl-logs.ps1.
WslLogs-2024-12-09_19-33-42.zip

Powershell Get-Disk output

Disk 0 is the internal 1TB SSD,
Disk 1 is an external 4TB USB disk, and
Disk 2 is the VHD created using Windows DIsk Management > Action > Create VHD.

The file path for the VHD is on Disk 1 (D:\media\extdisk.vhdx).
PS C:\> Get-Disk
Number Friendly Name             Serial Number                    HealthStatus         OperationalStatus      Total Size Partition
                                                                                Style
------ -------------             -------------                    ------------         -----------------      ---------- ----------
2      Msft Virtual Disk                                          Healthy              Online                     512 GB RAW
0      MZ9L41T0HBLB-00BMV-SAM... 0025_38E2_4100_A58A.             Healthy              Online                  953.87 GB GPT
1      Seagate Expansion                     NA8P86N1             Healthy              Online                    3.64 TB GPT

wsl.exe --mount on ARM64 requires Windows platform support which was added in version 27653.

Hmm, sounds like maybe a WIndows OS update is needed then?  And I'm guessing that that version of Windows 11 has not been released yet.  I am on the Microsoft Store version of WSL2 and thought that it was decoupled from Windows ... sounds like it was mostly decoupled but maybe not entirely (but as much as technically possible).

@benhillis so we just need to wait until version 27653 is available? Do you know if enabling the Windows Update option "Get the latest updates as soon as they're available" will install this version?

@benhillis, if you see this and have pity on a poor WSL hacker, any help/suggestions would be appreciated.
I believe I have Windows version: 10.0.26100.2314 so I need that to update to 10.0.27653.xxxx then?  (Sorry, my Windows background is a bit sketchy).
In my Windows Update I see that "2024-12 Cumulative Update for Windows 11 Version 24H2 for arm64-based Systems (KB5048667)" is pending and I'm guessing that is probably the 27653 update that I need.  Unfortunately every time I try to install it, it fails with "Install error - 0x800f0983".  In reading about that error it sounds VERY painful as it sounds like my Windows installation has become "corrupted" and I will need to wipe out my machine and re-install Windows and then everything I have...  pretty depressing.  I'm guessing that the "corruption" was caused when I created the VHD in Disk Management and then tried to mount it in WSL2.  I have since deleted the VHD but it looks like the damage has already been done.  :(  Does that sound about right?


Handle spaces in path when shortening

Spaces in directory names currently will cause truncation

Thanks, @andjscott! Nice quick fix :)


Deleted r-value constructor overloads for 'expression' object

I have removed the r-value reference overload for the expression objects. As it caused a bug as mentioned here:
https://github.com/rapidsai/cudf/pull/5494#discussion_r474983646

Hey @lamarrr! Were you able to confirm if this fixed the problem? I thought I had tried this locally and I don’t think it worked, but I might have made a mistake (I was in a hurry when I tried it). Also, it would be better to make this PR to https://github.com/rapidsai/cudf instead of my fork of the repository. I’m not working on the project actively right now. Thank you!

Yeah, I tested it locally and it worked. I suppose you only deleted the  overloads for the unary op and tested with the binary op.

Awesome!


Update SUPPORT.md

Added section to SUPPORT.md on our auto issue close process.

/check-enforcer override


No notifications for new incoming messages.

No notifications show when someone sends me a new message, or even any indication of a new message when I open the app. I think the notifications are a bug, but at least make it do something when you open the app, like bolded chat names for new messages.

You’ve configured something wrong yourself to not get notifications. The bit about not flagging conversations is a known limitation.
Are you sure your api key is correct for IFTTT?

I copy and pasted the IFTTT key from the Documentation page under Webhooks into the second field in the app. Yes, I launched it with the script.

Actually I typed it because I couldn't copy and paste.

You can right click to paste. The shortcut AFAIK doesnt work.

The notifications for the text I sent myself through another iPad 7 hours ago finally came... 7 hours after the message was received on the Mac.

Notifications aren’t my issue. The app polls the DB every five seconds and sends the notifications. The issue is in IFTTT or Pushbullet if you don’t have a log to show me.

Also clicking on notifications from a group chat sends me to the private chat of the person who sent the message in the group chat instead of the actual group chat.


Ensure substrate-node-template keeps up with API changes in substrate

Currently, substrate-node-template can easily get out of date. We probably want to amalgamate it into the substrate repo but without compromising the its raison d'etre which is to be an isolated piece of code that can be used to jumpstart a new Substrate chain, as used by substrate-up.

Fixed by: https://github.com/paritytech/substrate/pull/1637


Remove arm from Drone pipeline

Removes arm from Drone pipeline, do we need it?

@a-blender This repository does not build in Drone at the moment (https://drone-pr.rancher.io/rancher/channelserver/53/3/1) because the arm pipeline needs changes, but the question remains if it is still needed as it's better to just remove it if not (like done in other repositories).


[WIP] [Gluon] Add VAE demo

Description
Implement VAE with gluon.probability
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ ] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

Should we use https://pypi.org/project/notedown/ instead of committing the raw notebook? Would you recommend integrating the example into the website as well?

Yes. I agree that using markdown is better. It is a good example to showcase the usuage of probability module so I think we can put it into the website.

Should we use https://pypi.org/project/notedown/ instead of committing the raw notebook? Would you recommend integrating the example into the website as well?

@leezu That's sounds great, are there any existing examples that I can follow?

@szha @leezu
I am wondering if MXNet's website currently support automatically converting markdown demos to notebook via notedown, like GluonNLP: https://gluon-nlp.mxnet.io/examples/sentiment_analysis/sentiment_analysis.html ? Or I believe it could be little bit weird to provide the example in a markdown form.

not yet. I think we should provide the automation in CI first, which may take some time. in the meantime, I think we can check in the jupyter notebook for now with the expectation that it will be replaced by notedown in the future.

@xidulu @szha it's supported via https://github.com/apache/incubator-mxnet/blob/243ade93bcb8b7962d1faeb89c98409e3ae0d7a4/docs/python_docs/python/Makefile#L31-L33

@leezu
Thx for pointing that out, I will take a look at it.

@szha @leezu
Anything else to polish?

I'm ok with merging.

Thanks for the demo, @xidulu!


define mandatory and optional fields in the schema

the webapp can break in unexpected ways if it depends on fields which are not always there (eg. some fields are not there for every gene).
The difference should be coded in the schema, so the @opentargets/frontend team can know what to rely on

@ElaineMcA part of the refactoring process.

Issue moved to opentargets/platform #35 via ZenHub


Add profile for Paul DeCarlo

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

advocates/media/profiles/paul-decarlo.png
:white_check_mark:Succeeded
View

advocates/paul-decarlo.yml
:white_check_mark:Succeeded
View

advocates/toc.yml
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

advocates/media/profiles/paul-decarlo.png
:white_check_mark:Succeeded
View

advocates/paul-decarlo.yml
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


[Client]Allow to override PULSAR_MEM settings via PULSAR_EXTRA_OPS

Motivation
pulsar-client runs conf/pulsar_tools_env.sh to set PULSAR_EXTRA_OPTS. If PULSAR_EXTRA_OPTS are given in the command line and contain -Xmx, it will be overridden by PULSAR_MEM.
Modifications
Exchange the order of applying env variables. First go the hardcoded PULSAR_MEM settings, then potentially the user settings.
Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?

[ x] no-need-doc

Change in a shell script, which is not documented.

Why we need this feature, you can config both PULSAR_MEM and PULSAR_EXTRA_OPS
cc @lhotari @michaeljmarshall

Why we need this feature, you can config both PULSAR_MEM and PULSAR_EXTRA_OPS cc @lhotari @michaeljmarshall

I believe that this change improves consistency. The assumption is that you can override other options by setting them in PULSAR_EXTRA_OPTS. I made a similar improvement to consistency for bin/pulsar as part of PR #13025 .
Another point is that this change isn't harmful or risky.


Result of the query with multiple carriage return line feed  is truncated

Hello, i'm having this issue with SQLKATA with Orcale :
In my database, i have this kind of column --> Varchar2(4000)
with data like this
"TEST[CR][LF]TEST1[CR][LF]TEST2"
But in my query result, from a select instruction, i have this :
"TEST[CR][LF]"
So my data seems to be truncate, but i don't see where i can change this.
Is it in the complier ?
Thanks for your help!

Hi Ahmad, Thanks for your help here my sample code :
QueryResult result = new QueryResult();
query = select.GetQuery();
SqlResult sqlResult = QFContexte.Compiler.Compile(query);
IEnumerable<dynamic> v = QFContexte.FromQuery(query).Get(Transaction);

From the given code, it's not clear to me, what part is causing this issue, but I suggest running a raw string SQL with dapper, and see if the issue persist


Use bounded queue for metadata batch threadpool

When unboundedQueue became an alias for SynchronousQueue, it
improved performance in asyncChain, and almost everywhere in blueflood.
Except for the metadata batch threadpool. This is one of those areas
where a large queue is required in order for work to build up. The more
work that builds up, the bigger the batches, the more efficient the meta
cache batch operations get.
This should speed up blueflood ramp up during cache loads.

When unboundedQueue became an alias for SynchronousQueue

@dlobue unboundedQueue should never have become an alias for SynchronousQueue, it is a huge renaming overlook. We should refactor the method to explicitly have a method called syncQueue or something like that.

@chinmay-gupte I don't disagree. That is out of the scope of this PR though.

Changes Unknown when pulling 471e0c8f718a2e9bcd95af35db2607f3158d4835 on dlobue:dominic/meta_cache_batch_use_bounded_queue into ** on rackerlabs:master**.

+1
I don't know why coveralls failed to figure out the coverage change but I verified that it does know the lines you added were covered so shrug.


[Question]Autoware deployed in Autosar on yocto or QNX?

Autoware source can deployed in Autosar(like Adaptive Autosar;https://www.autosar.org/standards/adaptive-platform/) build on Yocto or QNX build-run on Hardware on QUALCOMM boards .to test ADAS features like object detection?

I'm afraid you should ask in the autoware community


chore: move ISSUE_TEMPLATE to root of .github

PR Details

Moved ISSUE_TEMPLATE to .github

Description

Previous PR created the folder ISSUE_TEMPLATE in the wrong place, it should be directly under .github
Motivation and Context

It needs to be under .github
Types of changes

[ ] Docs change / refactoring / dependency upgrade
[x] Chore (workflows / linting / tooling)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist

[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

Disclaimer
By sending us your contributions, you are agreeing that your contribution is made subject to the terms of our Contributor Ownership Statement

:tada: This PR is included in version 1.0.3 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


[eda] improve stability of flaky platform tests

Description of changes:

[mac] mypy and coverage have conflicts with lightgbm when running coverage stage during the build process; switching to use Random Forest for SHAP tests.
[win] np.int64 conversion is required for test on Windows box (Windows unlike Linux is frequently using int32 even on 64-bit machines).

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/platform_tests

/platform_tests

/platform_tests

EDA platform tests are passing now


HTTPServerRequest::Body() does not return the body of a request

    BeastErrorCode get_req(EaseTCPStream &stream) {
        BeastErrorCode ec{};
        BeastBuffer buffer;
        char buf[4096];
        BeastRequestParser rp(req);
        rp.get().body().data = buf;
        rp.get().body().size = sizeof(buf);
        rp.get().body().more = true;
        boost::beast::http::read_header(stream, buffer, rp, ec);
        while (!rp.is_done()) {
            rp.get().body().data = buf;
            rp.get().body().size = sizeof(buf);
            rp.get().body().more = true;
            boost::beast::http::read(stream, buffer, rp, ec);
            if (ec == BeastError::end_of_chunk) ec = {};
            if (ec) break;
        }
        if (!ec) {
            req = rp.release();
            m_origin_host = to_std_sv(req["Host"]);
        }
        return ec;
    }

Obviously Body() returns a pointer to a stack variable which makes no sense

Why stack variable makes no sense? Could you elaborate?

Why stack variable makes no sense? Could you elaborate?

Stack variables are destroyed once the function exits. The correct behaviour should be to read the following TCP stream bytes according to Content-Length header, and stores the request body in some buffer that has the same lifetime of the request object.

So I guess your were writing your own HTTP server handler and found that the http request buffer was corrupted?

Yes, I am handling a POST request and the Body() returns garbage data

Feel free to fix this if you are convenient. And we will follow up this bug next week as well.

@howardlau1999 Thank you for reporting bugs. We are actually rewriting HTTP server completely, which will be no longer based on boost.beast. I believe this bug will solved by the new one, and I expect it will be released in the next version.

This bug has been fixed by the release of version 0.5
https://github.com/alibaba/PhotonLibOS/releases/tag/v0.5.0


cmake, lint: Adjust lint_includes_build_config

The linter has been adjusted to consider CMake-specific bitcoin-config.h.in file.
Linter warnings have been fixed.
Can be tested by running linters locally.

Friendly ping @maflcko :)

Rebased.

Rebased.


Fix misspelling

Changed Imgage to Image

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

xml/Microsoft.Azure.Management.Compute.Models/ImageDiskReference.xml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


How to limit the generated text token to a maximum of 77?

When I show Video-LLava a short video, given inp = 'Could you please provide a detailed description for this video? Your comprehensive video caption should allow listeners to visualize the scene without actually watching the video. Note that the generated text tokens should not exceed 77!'
But I found that the length of the text tokens it generated was always greater than 77. How should I input inp or adjust the model to make its output meet my requirements?
(Because I want to use CLIP to process the generated text tokens later, I want to limit the length to within 77.)

Sorry, this is a known issue. It may not be able to follow instructions well due to too little video fine tuning data.


[Frontend hbbtv] Cannot start a Broadcast related parallel application

Hi,
I wish to start a "Broadcast related" application (with <HowRelated href="urn:dvb:metadata:cs:LinkedApplicationCS:2019:1.1" /> tag) with your DVB-I hbbtv implementation.
When I make a try, my application is succesfully launched, but I lose my broadcast.
By analyzing your code, parallel "Broadcast related" application seems to be supported :
      if(serviceInstance.parallelApps) {
        for(var i = 0;i< serviceInstance.parallelApps.length;i++ ) {
          if(serviceInstance.parallelApps[i].contentType == "application/vnd.dvb.ait+xml") {
            serviceApp = _application_.createApplication(serviceInstance.parallelApps[i].url,true);
            return;
          }
        }
      }
      if(selectedService.parallelApps) {
        for(var i = 0;i< selectedService.parallelApps.length;i++ ) {
          if(selectedService.parallelApps[i].contentType == "application/vnd.dvb.ait+xml") {
            serviceApp = _application_.createApplication(selectedService.parallelApps[i].url,true);
            return;
          }
        }
      }

I note you used createApplication HbbTV API function.
But from HbbTV specification, createApplication function is dedicated to "broadcast independent" application.
Did i misunderstand something ?
Is there a limit due to dvb-i implementation in a HbbTV context ?
Below an example of my service list files.
<?xml version="1.0" encoding="UTF-8"?>
<ServiceList xmlns="urn:dvb:metadata:servicediscovery:2020" xmlns:mpeg7="urn:tva:mpeg7:2008" xmlns:tva="urn:tva:metadata:2019" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1" xsi:schemaLocation="urn:dvb:metadata:servicediscovery:2020 dvbi_v2.0.xsd">
  <Name>DVB-I Demo</Name>
  <ProviderName>Demo</ProviderName>
  <LCNTableList>
    <LCNTable>
      <LCN channelNumber="1" serviceRef="tag:dvb.org,2020:sid1"></LCN>
    </LCNTable>
  </LCNTableList>
  <Service version="1">
    <UniqueIdentifier>tag:dvb.org,2020:sid1</UniqueIdentifier>
    <ServiceInstance priority="1">
      <DVBTDeliveryParameters>
        <DVBTriplet origNetId="65281" tsId="1" serviceId="1"/>
        <TargetCountry>FRA</TargetCountry>
      </DVBTDeliveryParameters>
    </ServiceInstance>
    <ServiceInstance priority="2">
      <DASHDeliveryParameters>
        <UriBasedLocation contentType="application/dash+xml">
          <URI>http://my-demo/dash/bbb1/bbb.mpd</URI>
        </UriBasedLocation>
      </DASHDeliveryParameters>
            </ServiceInstance>
    <ServiceName>Channel1</ServiceName>
    <ProviderName>Demo</ProviderName>
    <RelatedMaterial>
      <HowRelated href="urn:dvb:metadata:cs:LinkedApplicationCS:2019:1.1"></HowRelated>
      <MediaLocator>
      <tva:MediaUri contentType="application/vnd.dvb.ait+xml">http://my-demo/dvbi/ait.xml</tva:MediaUri>
      </MediaLocator>
    </RelatedMaterial>
    <ServiceType href="urn:dvb:metadata:cs:ServiceTypeCS:2019:linear"></ServiceType>
    </Service>
</ServiceList>

Thanks in advance for your response.
Benjamin

I'm don't think it is even possible to start a broadcast related application without having the application signaled in the broadcast.
DVB-I specification section <IP_ADDRESS> states:
In the present document, the semantics of applications signalled using a RelatedMaterial element with a
HowRelated@href attribute set to urn:dvb:metadata:cs:LinkedApplicationCS:2019:1.1 are not defined for
broadcast service instances using DVBTDeliveryParameters, DVBSDeliveryParameters or DVBCDeliveryParameters. Such elements should not be used. The DVB-I client shall ignore any application
signalling using a RelatedMaterial element with this HowRelated@href attribute when selecting such a broadcast
service instance and act solely on the broadcast AIT signalling.
Perhaps the reference application should not start the application signaled in RelatedMaterial-element the at all with a DVB-T service.

Ideally, and I presume for compatibility with an app environment, an app for a broadcast instance should be signalled in the AIT and launched from there.
The DVB-I specification could be more descriptive in regards to app signalling at the service or instance level, especially of there is application invoked switching of instances.

The reference app has limitations when run as a regular HbbTV app and lack of support for broadcast-related apps on DVB-I services is one of them. If you run it as an OpApp then these restrictions should go away.

@bpeexc , do you have access to an environment where you could try running the Reference client in OpApp mode? Exact instructions differ from platform to platform and depending on what kind of equipment you have available.

Hi,
I'm not sure but I don't think I have an access to an OpApp mode. I use the DVB-I client on mainstream TVs and smartphones.
I finally found a workaround for this issue : I created a broadcast-independant application which embed a DASH player (DashJs).
Like that, if an hybrid TV wants to play a channel in DASH, it will start application defined in AIT with my specifics background services and play dash video on foreground.
Otherwise, others devices just start the DASH video from your DVB-I client with the manifest url defined in the services list.

ok, good job! To enable OpApp mode even for debugging, you'd need to contact the manufacturer directly. But your solution seems to fit the purpose for now.
Do you want to share some source code to the project repository here perhaps?

Below an example of HbbTV application that automatically starts a dash video and does other things
index.html
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//HbbTV//1.1.1//EN" "http://www.hbbtv.org/dtd/HbbTV-1.1.1.dtd">

    <title>Launcher</title>
    <meta http-equiv="content-type" content="application/vnd.hbbtv.xhtml+xml; charset=utf-8" />
    <link rel="stylesheet" href="/hbbtv/styles.css" />

        <object type="application/oipfApplicationManager">
            application/oipfApplicationManager
        </object>
        <object type="application/oipfConfiguration">
            application/oipfConfiguration
        </object>
        <object type="application/oipfCapabilities">
            application/oipfCapabilities
        </object>

        <object type="video/broadcast" aria-label="video/broadcast"></object>
        <video id="video"></video>

styles.css
#oipf {
    visibility: hidden;
    display: none;
}

#video-container object {
    position: absolute;
    top: 0;
    left: 0;
    width: 1280px;
    height: 720px;
    display: block;
    z-index: 0;
}

#video-container video {
    position: absolute;
    top: 0;
    left: 0;
    width: 1280px;
    height: 720px;
    display: block;
    z-index: 0;
}

main.js (using jquery, transpiling with webpack)

import jquery from "jquery"

(function() {

  async function initHbbTV() {
    console.log("Init HbbTV")
    const applicationManager = jquery("#oipf object[type='application/oipfApplicationManager']")[0]

    try {
      const oipfApplication = applicationManager.getOwnerApplication(window.document)
      oipfApplication.show();
    } catch (e) {
      console.log("Non TV Mode")
    }

    const broadcast = jquery("#video-container object[type='video/broadcast']")[0]
    const broadband = jquery("#video")[0]
    return {
      broadcast,
      broadband,
    }
  }

  async function play(broadcast, broadband, url) {
    console.log("prepare Dash")
    let timer
    broadband.setAttribute("src", url)
    const canPlayPromise = new Promise(resolve => {
      broadband.oncanplay = () => {
        console.log("Broadband video can play")
        timer = setTimeout(() => {
          console.log("Force to play broadband")
          resolve()
        }, 1000)
      }
      broadband.oncanplaythrough = () => {
        clearTimeout(timer)
        console.log("Broadband video can play through !")
        resolve()
      }
    })

    return canPlayPromise.then(() => {
      this.__broadcast.stop()
      this.__broadband.play()
    }).catch(e => {
      console.log("Broadcast stop error: " + e.message)
    })
  }

  async function doOtherThings() {
    //TODO
  }

  jquery(async () => {
    try {
      const {broadcast, broadband} = await initHbbTV()
      await play(broadcast, broadband, "http://my.dash.example/manifest.mpd")
      await doOtherThings()
    } catch (e) {
      console.log("Error during process: ", e.message)
    }
  })
})()

thanks a lot for this. Just a minor detail, display: none should be avoided. :)


dataset annotations

Professor，can you give a brief introduction of annotations like(test_hico.json,trainval_hico.json,corre_hico.npy)?In test_hico.json,there are no introductions about subject_id、category_id and object_id.This confuses me.No map between .id and name was introduced

Thx for your interest in our repo. The answers to your mentioned problems can be found in similar issues #17 and #12.

Thank you，I have finished transforming this.


fixed using evolution script containing non ascii characters

Thank you for your contribution. I released the new version 0.3.2 containing your change.


Handle delivered parcels

Parcels are updated once reported as being delivered because they are no longer returned from the server as the integration only queries "active" parcels.
So either we also query historic parcels or we set the status manually.

I think i need to check for historic parcels if active parcels returns nothing. Then remove parcel 24 hours after delivery date

This should be fixed in 2024.10.3


Incompatible with Sodium

Is it a mod compatibility issue?
Yes
If it's a mod compatibility issue, please provide the link(s) to the incompatible mod(s)
Immersive Portals Continued
Sodium
Minecraft version
1.20.1
Describe the issue

https://mclo.gs/15ob3R7
immersive portals continued version: 3.4.0-iris1.8
sodium version: 0.5.11 + mc1.20.1
minecraft version: 1.20.1
fabric loader: 0.16.0
java version: 17.0.12

Duplicate to https://github.com/iPortalTeam/ImmersivePortalsMod/issues/1536


feat: Added APIs for adding removing namespace maintainers

[x] Don't allow package maintainers to remove other maintainers. Only namespace admins should be able to do that.
[x] Added API to add namespace maintainers.
[x] Added API to remove namespace maintainers.
[x] Remove duplicate conditions from the remove package maintainers API

cc: @minhqdao, @federicoperini

I am going to merge this branch.
As said, I have been trying to test up docker containers & test environment for the project. I will create a separate PR for that and add some testcases to that as well. After that we can create small PRs to test all the APIs @henilp105


⚠ Fortnite has degraded performance

In 16c623f, Fortnite (https://fortnite.com) experienced degraded performance:

HTTP code: 200
Response time: 1635 ms

Resolved: Fortnite performance has improved in c6ddfa3.


Suggestion: Password input icon

Suggestion

In the latest version of iTerm, they have included an icon that is a replacement for the current cursor icon when you have to type a password, so it looks like a key. I think this is very interesting.
iTerm:

Thank you for your suggestion.

FWIW, this has been in terminal.app for a release or three as well. I prefer the implementation too:


Implement EnvironmentVariablePasswordProvider

Adds EnvironmentVariablePasswordProvider which is capable of pulling metadata store password from environment variable.

@nishantmonu51 looks useful - could you add docs please?

added docs.

👍

I really like this. It also allows custom password providers to grab passwords in complex ways.
@gianm  / @nishantmonu51 how would you feel about just making this an environment variable provider? it uses passwordprovider so the name of EnvironmentVariablePasswordProvider is accurate, but can we change the field to just "variable" instead of "passwordVariable"?
I can see a scenario where you have an internal PasswordProvider that can take sub-PasswordProviders as parameters...  for example:
{
  "type":"companyPassFetcher",
  "key": "s0m3_k34",
  "secret":{
    "type":"environment",
    "variable":"SOME_DRUID_SECRET"
  }
}

I'm on board with naming it "variable"

@gianm @drcrallen: sounds good, renamed to variable.

👍 after jenkins

👍  after TRAVIS
DOWN WITH JENKINS


Adjust target frameworks

@HEskandari I revised the content of this PR since this repo has some special things going on that make it different from the others.


button added

fixes : #141
@Spyware007 beautiful and amazing button added

Thank You for your contribution. @ANMOLYADAV1402  🤩
Leave a star if you like it. 🌟


Instance and hosting organization name should be HTML encoded in the footer

If you make the name <A>wesome Hotel, for example, then the A becomes an HTML tag :/

Closing in favour of #4012


Pgx compatible?

Hi thanks for this repo.
It seems like the query interface is meant for libpq driver. Any chance this is compatible with pgx?

Statement query methods can be used with any driver, as long as driver implements standard *sql.DB interface - https://github.com/go-jet/jet/blob/master/qrm/db.go.
Main pgx Conn does not implement this interface so it is not possible to use it.
But, as I can see pgx also has stdlib package, which implements standard DB interface, and DB and Tx from this package you should be able to use with jet.
https://github.com/jackc/pgx/tree/master/stdlib

Thank you!


[SPARK-6440][CORE]Handle IPv6 addresses properly when constructing URI

Can one of the admins verify this patch?

ok to test

Test build #29936 has started for   PR 5424 at commit 84763d7.

Test build #29936 has finished for   PR 5424 at commit 84763d7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29936/
Test FAILed.

Use getHostAddress instead of getHostName, because of getHostName may return additional information like <IP_ADDRESS>-red.dhcp.yndx.net.

Test build #29940 has started for   PR 5424 at commit 2098081.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29940/
Test FAILed.

jenkins, test this please

Test build #29943 has started for   PR 5424 at commit 2098081.

Test build #29943 has finished for   PR 5424 at commit 2098081.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29943/
Test FAILed.

Seems like jenkins was killed again.

whoops, i missed retriggering this build.  :)
jenkins, test this please

Jenkins, retest this please.

Test build #29962 has started for   PR 5424 at commit 2098081.

Test build #29962 has finished for   PR 5424 at commit 2098081.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29962/
Test FAILed.

Jenkins, retest this please.

Test build #29974 has started for   PR 5424 at commit 2098081.

Test build #29974 has finished for   PR 5424 at commit 2098081.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29974/
Test PASSed.

I think I've looked at 10 different versions of 2 PRs for this JIRA now, and all seem to be changing or adding something slightly different than what's essential here. Let me describe what I think needs to be done and why and we can settle that, then implement.
The code needs a local IP address in many cases. That's localIpAddress. (Separately, your update to find an IPv6 address sounds OK.)
localIpAddressHostname and the getAddressHostName() method actually look pointless. They make a String into an InetAddress and then ask for the exact same text representation back as a String. Personally I'd make localIpAddress an InetAddress, as you've done. I'd remove localIpAddressHostname and replace with localHostName() where a String is needed.
... because then customHostname is consistently applied, right? I think that must be the intent?
But then, localIpAddress itself doesn't have much value. It is always used for its string rep anyway. So, localHostName() as well? it's the only thing that needs to be non-private.
Back to the original issue: where localHostName() would then be used in a URI, it can be run through the utility method in InetAddresses. I don't think that we should instead have more methods in Utils just for adding a call to that method.
Finally, there are instances in SparkSQLEnv, KinesisReceiver and TestUtils that use InetAddress.getLocalHost directly. These should also use localHostName().
Maybe there's a reason that doesn't 100% work but it seems a lot less messy than the 3-5 ways code is reasoning about the local IP address string now.
What say, or, should I try it in my own PR?

Ok, i'll try to:

replace localIpAddressHostname in Utils.localHostName with localIpAddress.getHostAddress
replace localIpAddressHostname usage with localHostName
remove localIpAddressHostname and getAddressHostName
make localIpAddress private
fix SparkSQLEnv, KinesisReceiver and TestUtils

today/tomorrow.

Back to the original issue: where localHostName() would then be used in a URI, it can be run through the utility method in InetAddresses.
I don't think that we should instead have more methods in Utils just for adding a call to that method.

I didn't find any method in InetAddresses like escapeHostname(String) => String. InetAddresses.toUriString(InetAddress.getByName(Utils.localHostName())), but this looks pointless like localIpAddressHostname = InetAddress.getByName(localIpAddress).getHostName.
And there we have an option:

make localIpAddress private, remove localIpAddressURI variable and add def localHostURI() = customHostname.getOrElse(InetAddresses.toUriString(localIpAddress))
ask everyone to use InetAddresses.toUriString(InetAddress.getByName(Utils.localHostName()))

imho additional method with private variable is better. Also i don't like conversion from String to InetAddress

InetAddresses.toUriString accepts a String and returns an InetAddress, so it does not need its input changed. Yes you have to call getHostName on the output.

InetAddresses.toUriString is (InetAddress) => String, maybe you meant forUriString(String) => InetAddress or forString(String) => InetAddress ?
for forUriString
scala> val ipv6 = InetAddress.getByAddress(NetworkInterface.getNetworkInterfaces.toList.apply(0).getInetAddresses.toList.apply(0).getAddress).getHostAddress
ipv6: String = fe80:0:0:0:3623:87ff:fe6a:423c

scala> InetAddresses.forUriString(ipv6)
java.lang.IllegalArgumentException: Not a valid URI IP literal: 'fe80:0:0:0:3623:87ff:fe6a:423c'
  at com.google.common.net.InetAddresses.forUriString(InetAddresses.java:501)
  ... 33 elided

And for forString i still need InetAddresses.toUriString
scala> val ipv6 = InetAddress.getByAddress(NetworkInterface.getNetworkInterfaces.toList.apply(0).getInetAddresses.toList.apply(0).getAddress).getHostAddress
ipv6: String = fe80:0:0:0:3623:87ff:fe6a:423c

scala> InetAddresses.forString(ipv6).getHostName
res35: String = fe80:0:0:0:3623:87ff:fe6a:423c

scala> InetAddresses.forString(ipv6).getHostAddress
res36: String = fe80:0:0:0:3623:87ff:fe6a:423c

scala> InetAddresses.toUriString(InetAddresses.forString(ipv6))
res37: String = [fe80::3623:87ff:fe6a:423c]

Oh gosh yeah looking at this the wrong way around of course. Aren't there just two places where the URI logic is needed? I think calling InetAddresses.toUriString(InetAddress.getByName(Utils.localHostName())) in two places isn't so bad. A new method is OK too if you rprefer; I suppose ... localHostNameForURI() or something to make the relation clear? Then we have just two methods for accessing 'the local host name' with a clear role.

Ok, localHostNameForURI sounds reasonable.
About network/common/src/test/java/org/apache/spark/network/TestUtils.java: there is just only one method -- getLocalHost() => String, i can delete the whole class and use Utils.localHostname instead, but maybe using scala from java it's not a good practice?

I'd just leave TestUtils for purposes here, but can make it call through to Utils.

Test build #30091 has started for   PR 5424 at commit 6b717aa.

Test build #30091 has finished for   PR 5424 at commit 6b717aa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30091/
Test PASSed.

LGTM. This simplifies and standardizes the local host name logic across the code, fixes the original problem of IPv6 addresses in URIs, chooses a better local address in some IPv6 cases. Of course, making things consistent means slight behavior changes in some cases -- for example, customHostName now applies in cases it didn't before, but this actually seems more correct. I've eyeballed every change and it looks logical, and the tests pass.


Encode XML entities rather than wrapping in CDATA

There is a number of good reasons not to use CDATA for entity encoding. Because of that we should consider encoding entities rather than wrapping text in CDATA section:
Input:
<doc xmlns="http://foo">
<specialChar>
</doc>

Output:
<doc xmlns="http://foo"><![CDATA[
<specialChar>
]]></doc>

Expected:

preserve encoding or switch to encoding per default

Closed via https://github.com/bpmn-io/moddle-xml/commit/5645b5822644a461eba9f3da481362475f040984.

@nikku Thank you for the quick fix!


Fix #54

Please describe the changes this PR makes and why it should be merged:
Fixes #54
Downgrades ow to v0.27.0 to fix ESM issues.
Status and versioning classification:

Codecov Report

Merging #55 (bc961e4) into main (65467ec) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##              main       #55   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           21        21
  Lines          264       264
  Branches        33        33
=========================================
  Hits           264       264

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 65467ec...bc961e4. Read the comment docs.


How to forward and store trace files to a specified machine through the network？

hello~
When you use the perfetto command to capture trace files, they are stored on the local machine by default. So how do you store the trace files on a remote designated machine?
my platform：linux
thank you~

I am not fully sure I understand what you want to do. I can offer two options

Capture the trace locally, maybe on tmpfs, and then use ssh/scp/sftp to move it to another machine.
Capture the trace remotely forwarding the cmdline client via a SSH tunnel. Something like the below:

Start perfetto services (traced, traced_probes) on the target machine, forwarding the consumer socket over ssh: ssh -L/tmp/perfetto-consumer:/tmp/perfetto-consumer lon /s/perfetto/tools/tracebox traced  (remember also to start traced_probes and other damons as needed)
Start the cmdline client on your local machine, using the forwarded socket path:  PERFETTO_CONSUMER_SOCK_NAME=/tmp/perfetto-consumer tracebox perfetto -o out.pftrace --txt -c config_file

I am not fully sure I understand what you want to do. I can offer two options

Capture the trace locally, maybe on tmpfs, and then use ssh/scp/sftp to move it to another machine.
Capture the trace remotely forwarding the cmdline client via a SSH tunnel. Something like the below:

Start perfetto services (traced, traced_probes) on the target machine, forwarding the consumer socket over ssh: ssh -L/tmp/perfetto-consumer:/tmp/perfetto-consumer lon /s/perfetto/tools/tracebox traced  (remember also to start traced_probes and other damons as needed)
Start the cmdline client on your local machine, using the forwarded socket path:  PERFETTO_CONSUMER_SOCK_NAME=/tmp/perfetto-consumer tracebox perfetto -o out.pftrace --txt -c config_file

I tried the method you mentioned, however, the collected data is local, not remote. It feels like this PERFETTO_CONSUMER_SOCK_NAME does not take effect. May I ask if you have any other methods?

I am sorry but if you want to get some help you need to be more specific in your requests.
Please word your bugs in the form of:

The exact commands you are running and where
What you expect to happen
What happens instead

I am sorry but if you want to get some help you need to be more specific in your requests. Please word your bugs in the form of:

The exact commands you are running and where
What you expect to happen
What happens instead

machine A:  <IP_ADDRESS>   arch: arm   os: linux
machine B:  <IP_ADDRESS>   arch: x86    os: linux

Purpose: I want the data captured on machine A to be stored in machine B in real time and generate a trace file.
I used method 2.
machine A:

./traced --background
./traced_probes --background --reset-ftrace
ssh -L <IP_ADDRESS>:/tmp/perfetto-consumer:<IP_ADDRESS>:/tmp/perfetto-consumer<EMAIL_ADDRESS>bash /machineB/run.sh

The content of file run.sh is:

#!/bin/bash
./traced --background
./traced_probes --background

Since ftrace is not enabled on machine B, there is no --reset-ftrace parameter.
machine B:
PERFETTO_CONSUMER_SOCK_NAME=/tmp/perfetto-consumer ./perfetto --txt -c demo.cfg -o out.ftrace
Problem: The captured out.ftrace file does not store the data of machine A, but of machine B, which is contrary to my purpose.
Please tell me where I wrote it wrong? What should I do to achieve this purpose?
I'm looking forward to your reply, thank you so much.

./traced --background
./traced_probes --background --reset-ftrace

Wait, you are running traced and traced_probes both on Machine A and Machine B (via run.sh). DO you really want both? I thought you wanted only Machine B's tracing, not Machine A.
In all the instructions below I'm going to assume that you want only Machine B's trace data, and no tracing at all from Machine A (if that's the case we need to tweak things a little bit)

ssh -L <IP_ADDRESS>:/tmp/perfetto-consumer:<IP_ADDRESS>:/tmp/perfetto-consumer<EMAIL_ADDRESS>bash /machineB/run.sh

Why the forwarding is not just -L/tmp/perfetto-consumer:/tmp/perfetto-consumer ? You are forwarding a unix socket, not a TCP connection, there should be no IP addresses in the fwd config.

machine B:
PERFETTO_CONSUMER_SOCK_NAME=/tmp/perfetto-consumer ./perfetto --txt -c demo.cfg -o out.ftrace

This is wrong, this command should be run still from Machine A.
What's going to happen is the following

Machine B will start traced and traced_probes
Machine B will by default listen on /tmp/perfetto-consumer
The SSH cmdline will forward connections for MachineA:/tmp/perfetto-consumer to MachineB:/tmp/perfetto-consumer (This will be used by the next command)
Then you run perfetto's cmdline client on Machine A (This is your mistake)
Perfetto cmdline tries to connect to /tmp/perfetto-consumer
Because of the forwarding, the perfetto cmdline client running on machine A will be talking to the traced server on machine B

Let me change the names a little bit so you might be able to tell more what's happening and we can avoid confusion:
Machine A:

ssh -L /tmp/forwarded-socket:/tmp/perfetto-consumer<EMAIL_ADDRESS>bash /machineB/run.sh
PERFETTO_CONSUMER_SOCK_NAME=/tmp/forwarded-socket ./perfetto --txt -c demo.cfg -o out.ftrace

That's it
This will generate an out.trace on Machine A. The trace will be obtained by the perfetto cmdline client (which will run on Machine A) talking to the traced service running into Machine B, tunneling the socket via SSH forwarding.
Does it make sense?

hello~
I tried it as you suggested, but when I executed this command(PERFETTO_CONSUMER_SOCK_NAME=/tmp/forwarded-socket ./perfetto --txt -c demo.cfg -o out.ftrace) on machine A, I did not run traced and traced_probes separately on machine A.
I got an error:
[162.402]    perfetto_cmd.cc:1040 Disconnected from the Perfetto traced service
The perfetto cmdline client running on machine A will not be talking to the traced server on machine B.
I saw the forwarded /tmp/forwarded-socket on machine A.
What is the reason for this? I'm looking forward to your reply, thank you so much.

hello, first of all thank you~
Now the perfetto cmdline client will always be stuck,  the error message is channel 2: open failed: unknown channel type:

Is this because the ssh versions of machine A and machine B are different?
machine A:  <IP_ADDRESS>   arch: arm   os: linux  ssh version: Dropbear v2019.78
machine B:  <IP_ADDRESS>   arch: x86   os: linux  ssh version: OpenSSH_8.2p1 Ubuntu-4ubuntu0.9, OpenSSL 1.1.1f  31 Mar 2020

machine A's /etc/ssh/sshd_config:
AllowTcpForwarding yes
#PermitTunnel no

machine B's /etc/ssh/sshd_config:
#AllowTcpForwarding yes
#PermitTunnel no

I'm looking forward to your reply, thank you so much.

Try setting AllowStreamLocalForwarding yes as that is not a TCP connection
Unfortunately this seems a SSH-tunneling issue now. Probably dropbear doesn't support something there.
I don't know how to help on this sorry.

hi~
Both machines now use the same version of openssh and no longer use dropbear.
write_into_file=true is set in my configuration file all.cfg, I got an error:

If write_into_file is not set, the trace data can be captured correctly.
However, I need to set write_into_file=true, how should I solve this problem?
I'm looking forward to your reply, thank you so much.

write_into_file=true cannot work across forwarding because relies on the cmdline client to forward a file descriptor to traced, which can be done only on the same machine.
Aahhh I see you want long traces.
Okay if you want long traces, then we need to change strategy completely.
Let me propose something completely different. Let's run all 3 (traced, traced_probes, and perfetto_cmd) onto MachineB and let's forward only the output.
To be honest this is even easier, I am not sure why I didnt' think about this earlier. Let's start everything from scratch:
From Machine A:
ssh MachineB './tracebox --txt -c /tmp/cfg.txt -o /dev/stdout' > /tmp/trace
The beauty of tracebox is that takes care of running traced, traced_probes and perfetto all together without having you do deal with 3 processes.
That simple. I tried it with this cfg.txt and it works
write_into_file: true
duration_ms: 30000

buffers {
  size_kb: 32768
  fill_policy: RING_BUFFER
}

# Enable various data sources as usual.
data_sources {
  config {
    name: "linux.ftrace"
    target_buffer: 0
    ftrace_config {
      ftrace_events: "sched_switch"
      ftrace_events: "sched_wakeup_new"
      ftrace_events: "sched_wakeup"
      ftrace_events: "sched_waking"
      ftrace_events: "task_newtask"
    }
  }
}

data_sources {
  config {
    name: "linux.process_stats"
    target_buffer: 0
  }
}

hi~
The new method you mentioned can correctly generate the trace file. However, the file cannot be opened using perfetto ui. It should be that the file format is wrong.

Is it possible that the/dev/stdout output content is then written to the file, causing a format error? How should I solve it?
I'm looking forward to your reply, thank you so much.

hi~ The new method you mentioned can correctly generate the trace file. However, the file cannot be opened using perfetto ui. It should be that the file format is wrong.

Is it possible that the/dev/stdout output content is then written to the file, causing a format error? How should I solve it? I'm looking forward to your reply, thank you so much.

I solved this problem, thanks you~
For some reason I'm using the perfetto command instead of tracebox.
Wrong command:
ssh MachineB './traced --background; ./traced_probes --background --reset-ftrace; ./perfetto --txt -c /tmp/cfg.txt -o /dev/stdout' > /tmp/trace
When I split the above command into 3 commands it works correctly.
Correct command:
ssh MachineB './traced --background
ssh MachineB './traced_probes --background --reset-ftrace
ssh MachineB './perfetto --txt -c /tmp/cfg.txt -o /dev/stdout' > /tmp/trace
Thank you very much for your patient guidance these days. See you soon.☺

what's happenig is that you are forwarding to /tmp/trace the stdout of the other 2 commands (Traced and traced_probeS) that emit some PIDs in stdout.
Why are you not using tracebox btw?
The right way to combine them is something like
ssh MachineB './traced --background > /dev/null; ./traced_probes --background --reset-ftrace > /dev/null; ./perfetto --txt -c /tmp/cfg.txt -o /dev/stdout' > /tmp/trace


The extension 's version display in marketplace is not same as after installed it

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version: 1.70.1 6d9b74a70ca9c7733b29f0456fd8195364076dda x64
OS Version:  Microsoft Windows 11 21H2 (OS 22000.2538)

Steps to Reproduce:
I found that the version number displayed after downloading my extentions is inconsistent with the version number in the marketplace.
And I try many times to re-installed it.
Here are some pics about this bug.
Before I installed it in marketplace:

After I installed, it changed immediately to the old version number, and the functions of this extension also rollback to the old one.

I have no idea what happened, is this all about my proxy by any chance?

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.84.0. Please try upgrading to the latest version and checking whether this issue remains.
Happy Coding!