Decide on mitigation of missed leadership checks due to ledger snapshots

Problem description
John Lotoski informed us that currently on Cardano mainnet, adequately resourced nodes (well above minimum specs) are missing lots of leadership checks during ledger snapshots.
Concretely, during every ledger snapshot (performed every 2k seconds = 72min by default), which takes about ~2min, the node misses ~30 leadership checks with 32GB RAM, and ~100 with 16GB RAM. This means that the node is missing ~0.7-2.3% of its leadership opportunities, and without mitigations, this number will likely grow as the size of the ledger state increases over time.
This problem is not a new one, it has existed since at least node 8.0.0 (and likely even before).
Analysis
Various experiments (credits to John Lotoski) indicate that this problem is due to high GC load/long GC pauses while taking a ledger snapshot (current mainnet size is ~2.6GB serialized). The main reasons for this belief are:

Using --nonmoving-gc fixes the problem for some time.[^1]

Judging from a 6h log excerpt, both GC time and missed slots increase greatly during a ledger snapshot:

(GC time comes from gc_cpu_ns)

Changing other aspects of the machine running the node (compute, IOPS) has no effect.

[^1]: Quoting from John Lotoski:
> Trying the non-moving GC out did in fact resolve the missed slots, at least for about 5 days, after which missed slots started happening again, and eventually they surpassed the default copying GC missed slots in quantity of about 30 per hour and looked to continue increasing further over time.
> I suspect this is due to increasing fragmentation which the default copying GC is better at minimizing.  There have been several improvements and bug fixes in the non-moving garbage collector through GHC 9.4.X, including improving the non-moving GC's ability to return allocated memory back to the OS which doesn't seem to happen at all on 8.10.7, so perhaps there might be better results with the non-moving GC once node is compiled on GHC >= ~9.4.X.  In any case, when using the non-moving GC on 8.10.7, there are no new observed segfaults or other immediately obvious problem related to the use of the non-moving GC.
Potential mitigations
Several orthogonal mitigation options have been raised:

Try to find a set of RTS options that fixes the problem without any code changes. In particular, it might be the case that --nonmoving-gc on a more recent GHC is enough, see [^1].
Incrementalising the creation of ledger snapshots. Specifically, this could be achieved by introducing a delay between writing individual chunks of the snapshot to disk. One hope here is that by spreading out the allocation work over a longer period of time (default GC interval is 72min), the total time spent GCing will be less, but a priori, it might just be the case that we miss as many slots as before, just spread out more evenly over a longer period of time.
Don't take ledger snapshots around the time the node is elected. Since we know in advance when the node will be elected, we can take this into account when deciding whether to take a ledger snapshot, and choose not to do so if an election opportunity is imminent (eg <5min). This way, we are guaranteed not to miss slots due to this problem when it is actually important.
Tro to optimize allocation behavior of ledger snapshot serialization There might be some opportunities to improve the existing code to not allocate as much or in a gentler way, eg maybe by using unpinned instead of pinned (ByteString) chunks?

Note that UTxO HD will also help, but it will likely not be used for some time by block producers (where this is issue is actually important).

The goal of this ticket is to interact with other teams/stakeholders to identify the best way forward here.

Incrementalising the creation of ledger snapshots. Specifically, this could be achieved by introducing a delay between writing individual chunks of the snapshot to disk. One hope here is that by spreading out the allocation work over a longer period of time (default GC interval is 72min), the total time spent GCing will be less, but a priori, it might just be the case that we miss as many slots as before, just spread out more evenly over a longer period of time.

It seems fair to assumed the leadership check thread is starved of CPU while the snapshot is being taken. I think there are two a priori suspects who might be occupying the CPU during that time:

long GC pauses
the snapshotting thread (threads?)

If the GC pauses are greater than a second, then it's entirely possible the leadership check doesn't have CPU time during the elected slot. If we incrementalize the snapshotting work, then perhaps the lower allocation rate will allow for smaller GC pauses, ideally sub-second (ie sub-slot).
Generally the RTS scheduler is fair, so I wouldn't expect the snapshotting threads (busy while taking the snapshot) would be able to starve the leadership check thread. But I list them anyway because maybe something about the computation is not playing nice with the RTS's preemption. (Examples include a long-running pure calculation that doesn't allocate, but I don't anticipate that in this particular case. Maybe some "FFI" stuff? I'm unsure how the file is actually written.)
The lack of fairness in STM shouldn't matter here. The leadership check thread only reads the current chain, which shouldn't be changing so rapidly to incur contention among STM transactions that depend on it.

The heap profile shows that the spikes are dominated by pinned memory (ARR_WORDS):

However, we also see allocations of Data.Map and Data.Set:

We will discuss with the Ledger team possible ways to avoid these allocations.

There are possible culprits for the increase in CPU and memory usage, and GC time:

The encoding is not consumed as a stream, which requires a large amount of data to be encoded as bytstring at once before being written to disk.
We are forcing the Ledger's pulser, which requires a time consuming computation (as suggested by @jasagredo).
We are converting the compact representation of multi assets, which allocates a Data.Map.

@lehins will look into 1 and 2, to see how much work they would require. As for 0 this is something we could do at the Consensus side.
@coot suggested we could also use info tables profiling to extract more information.

I tried using hPutBuilder (using

A second of db-analyser built without the profiling flag does not seem to exhibit the spikes observed above

We also observe a very low productivity during the run above:
[426.09964258s] Started StoreLedgerStateAt (SlotNo 72436908)
[442.253250968s] BlockNo 7793000	SlotNo 72343251
[473.668029643s] BlockNo 7794000	SlotNo 72364115
[504.554850119s] BlockNo 7795000	SlotNo 72384162
[544.457781846s] BlockNo 7796000	SlotNo 72405218
[568.979508903s] BlockNo 7797000	SlotNo 72425439
....
[933.002690111s] Snapshot stored at SlotNo 72436985
[933.00286017s] Snapshot was created at SlotNo 72436985 because there was no block forged at requested SlotNo 72436908
[933.00319006s] Done
ImmutableDB tip: At (Block {blockPointSlot = SlotNo 109730906, blockPointHash = c51616e6e4eefa40d13f5d0f9a47371f50d01ce760e3b8f0d0add212449e9b12})
 216,427,440,776 bytes allocated in the heap
1,487,832,561,768 bytes copied during GC
   3,344,035,160 bytes maximum residency (684 sample(s))
     122,786,736 bytes maximum slop
            6465 MiB total memory in use (0 MiB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0      6211 colls,  6211 par   22.448s  22.395s     0.0036s    0.0370s
  Gen  1       684 colls,   683 par   1208.587s  852.713s     1.2467s    1.8203s

  Parallel GC work balance: 90.70% (serial 0%, perfect 100%)

  TASKS: 7 (1 bound, 6 peak workers (6 total), using -N2)

  SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)

  INIT    time    0.263s  (  0.263s elapsed)
  MUT     time   58.518s  ( 58.299s elapsed)
  GC      time  1231.035s  (875.108s elapsed)
  EXIT    time    0.000s  (  0.010s elapsed)
  Total   time  1289.816s  (933.680s elapsed)

  Alloc rate    3,698,500,665 bytes per MUT second

  Productivity   4.5% of total user, 6.2% of total elapsed

Processing 20K blocks and storing 3 snapshots corresponding to the last 3 blocks that were applied results in the following heap profile:

The "Detailed" tab seems to show PulsingReward as allocating a large portion of data:

The branch used to obtain these results can be found here.
The profile can be produced by running:
cabal run exe:db-analyser -- cardano   --config $NODE_HOME/configuration/cardano/mainnet-config.json   --db $NODE_HOME/mainnet/db   --analyse-from 72316896   --only-immutable-db --store-ledger 72336896 +RTS -hi -s -l-agu

NODE_HOME points to the directory where the node was run (which contains the chain database).
db-analyser.eventlog.zip

@dnadales could you run it once more, with the https://hackage.haskell.org/package/base-<IP_ADDRESS>/docs/GHC-Stats.html#v:getRTSStats dump before and after each of the ledger snapshots being written?

I've created:

https://github.com/IntersectMBO/cardano-ledger/issues/4191
https://github.com/IntersectMBO/cardano-ledger/issues/4192

CC: @teodanciu @lehins @TimSheard

FTR, Ledger experiment on avoiding forcing the pulser when serializing a ledger state https://github.com/IntersectMBO/cardano-ledger/pull/4196

I'm surprised that it "only" took 70 seconds to write out the snapshot file, since I recall reports that the node was hampered for several minutes when writing a snapshot.

In the original report by John Lotoski above, it took ~2min, so it isn't off by that much.

There is also a chance that creation of a snapshot by a running node can actually trigger GC to run at the same time, which would have a significant impact on a snapshot creation time. There is a good chance GC is not triggered in db-anaylzer, while it would be for a running node, due to lower memory overhead.
Naturally this is just a speculation that is not backed by any real data. :grin:

I've also embellished my run with stats similar to what Damian did in https://github.com/IntersectMBO/ouroboros-consensus/issues/868#issuecomment-1983836481
My results (which look similar-ish to his, but I haven't doubled checked). These numbers are all observed via GHC.Stats.getRTSStats immediately before and immediately after writing the snapshot file in db-analyser.

~3000 minor gcs
~5 major gcs
~118 gigabytes allocated
~40 gigabytes copied
~27 seconds elapsed in the mutator
~41 seconds elapsed in the GC

I did some experimentation with an ad-hoc db-analyser pass that merely loads the snapshot file and then (tail recursion) forces the spine of the CBOR Encoding. There's no writing to disk, or even conversion to a bytestring.

With the code as-is, that calculation incurs 4 majors, 3000 minors, spends 25 seconds in the mutator and 41.5 seconds in the GC. There are 4.5 GB live bytes. (on my laptop, see above for specs)
With the UTxO immediately relocated into a compact region, everything is similar except there's only 35 seconds in the GC. I was expecting that'd make a bigger difference. There are still 4.5 GB live bytes, but the UTxO need not be traversed for pointers.
With the UTxO immediately replaced by the empty map, the numbers are drastically different: 7 major, 500 minor, 3.8 seconds mutator, 14 seconds in the GC. There is only 1 GB live bytes.

I had a call with @lehins. We both have an idea, and they both seem plausible and orthogonal. And moreover orthogonal to UTxO HD.

For the disk-based serialization, some of the motivations to use CBOR as much as possible don't apply as much as they do for on-chain and on-the-wire codecs. So Alexey was thinking to skip a few expensive steps when serializing each individual UTxO --- they'd just be one short ByteString CBOR token instead of the fully expanded "official" CBOR representation of each UTxO. We're hoping this avoids a non-negligible amount of allocation, and hence would improve the mutator&GC time. (This has a secondary benefit of requiring less work when writing to and querying from the UTxO HD backend.)

The other idea is my "pulser" in the previous comment. We can spread out the serialization work (and its allocation and hence its GC pressure) over tens of minutes instead of what is currently 70 seconds.

The wrinkle here is that there's no fundamental way to anticipate how many total tokens. It's likely to stay around 220 million until UTxO HD, and then it'll drop significantly (but also in proportion to its GC pressure). And then it's likely to stay around the same again. So there would be some "magic numbers" to choose, but it's plausible they could be related to some metrics that are relatively stable (eg number of stake pools?).
Also, overestimating the total token count would be harmless: the serialization would finish sooner than we anticipated, but the GC rate will still have been bounded to the extend intended.
220 million tokens in 70 seconds is ~3.15 million tokens per second. For example, 315k tokens per second ought to take around 700 seconds, which would mean the GC pressure would be 10x less concentrated.
Perhaps the node could calibrate itself: count the tokens in the snapshot it read upon initialization and use that. That doesn't work nearly as well for a syncing node, but big GC pauses are also less disruptive for a syncing node, since they're not "properly online" yet.

The other idea is my "pulser" in the previous comment. [...] The wrinkle here is that there's no fundamental way to anticipate how many total tokens.

What about separating the work based on the chunks of the resulting lazy ByteString, as suggested in the issue description?

Incrementalising the creation of ledger snapshots. Specifically, this could be achieved by introducing a delay between writing individual chunks of the snapshot to disk. One hope here is that by spreading out the allocation work over a longer period of time (default GC interval is 72min), the total time spent GCing will be less, but a priori, it might just be the case that we miss as many slots as before, just spread out more evenly over a longer period of time.

Yeah, byte size/chunk count/etc could replace token count.
But the same problem remains: we don't a priori know the total byte size, the total chunk count, etc.


Create pull_request_template.md to resolve #211

With a default checklist and an item to resolve the linked issue automatically.

@mrkara looks good, yet I would like to also add this:
Build artifact for QA: <build_artifact_file>

Thanks for the review. Sounds sensible. I've just updated the PR to include that.


appcache nanny update found

The site uses app cache to ensure the site works offline.
At the moment AppCache nanny checks for updates, when it finds an update it just refreshes the whole page. This experience is not ideal.
We should look at how this can be improved.
Initially, I would suggest showing the user a notification of some kind and asking if they would like to refresh the page to see new content.

solved with service workers


Add missing bower.json.

Hey, maintainer(s) of Norkart/Leaflet-MiniMap!
We at VersionEye are working hard to keep up the quality of the bower's registry.
We just finished our initial analysis of the quality of the Bower.io registry:
7530 - registered packages, 224 of them doesnt exists anymore;
We analysed 7306  existing packages and  1070 of them don't have bower.json on the master branch ( that's where a Bower client pulls a data ).
Sadly, your library Norkart/Leaflet-MiniMap is one of them.
Can you spare 15 minutes to help us to make Bower better?
Just add a new file bower.json and  change attributes.
{
  "name": "Norkart/Leaflet-MiniMap",
  "version": "1.0.0",
  "main": "path/to/main.css",
  "description": "please add it",
  "license": "Eclipse",
  "ignore": [
    ".jshintrc",
    "**/*.txt"
  ],
  "dependencies": {
    "<dependency_name>": "<semantic_version>",
    "<dependency_name>": "<Local_folder>",
    "<dependency_name>": "<package>"
  },
  "devDependencies": {
    "<test-framework-name>": "<version>"
  }
}

Read more about bower.json on the official spefication and nodejs semver library has great examples of proper versioning.
NB! Please validate your bower.json with jsonlint before commiting your updates.
Thank you!
Timo,
twitter: @versioneye
email<EMAIL_ADDRESS>VersionEye - no more legacy software!

I see from the 0.7.3 changelog for Leaflet that they've included bower support a while now. Might be time to revisit this issue now.

I just checked out the changes, but there is still no bower.json file in the root directory. We track the project anyway, but as long there is now bower.json file we have no information about dependencies, licenses, description, authors and so on.

+1 for a bower.json

+1 for bower support!

Is there anything we can do to help with bower support?

I've added a bower.json, so I think things should be available now soon. (Someone had already registered this repo in the bower registry with the correct git address, so I suppose things should be showing up correctly now that the bower.json is in place.)
@deeg: If you want to help, try using bower to install leaflet-minimap and see if it all works properly. IF something is wrong with the bower.json, pull requests are welcome.

Cool! Many THX for the bower.json :-)

It is working now, thanks. I do suggest adding to the readme about how to run the command to install it, since it is all lower case while this repo as caps.
Also I think it is pretty good practice to have a version number in your bower.json file if you want to add one.

The spec actually says that a version number is deprecated and ignored, so I'm happy to not have to mess about with updating that manually when I npm version major/minor/patch.
I can put somtehing in the readme!

Ah, thanks for letting me know!


[Question] EOS in reward model dataset

Hi,
first of all thank you very much for the repo!
I would like to ask, is the EOS token in the reward model dataset necessary for the model? I'm using a gpt2 model where EOS is used for BOS and EOS is not usually used. That's why I was wondering if maybe this token is needed for reward modeling or is it model specific?
https://github.com/OpenLLMAI/OpenRLHF/blob/072e286a5c5f3cd6acf2c9ad7e4ef727a8dedb83/openrlhf/datasets/reward_dataset.py#L148
Thank you!

Usually, we use an EOS token to output the reward value.
So we did not consider the situation of GPT-2

Hi, thank you for the reply. What is a possible way to fix this? Should I use a new special token?

Hi, thank you for the reply. What is a possible way to fix this? Should I use a new special token?

You can try to allocate an unused special token instead.

Usually, we use an EOS token to output the reward value. So we did not consider the situation of GPT-2

Why use this EOS? Can we not add EOS? Can we directly input the sentence into RM to output a scalar value? EOS is also not added to the training set when training RM.
If adding EOS, the final token needs to be replaced with EOS when using it.


Turborepo using npm

I'm following your monorepo template for using shadcn-ui but using npm (workspaces). However, the application has continuous import errors, in specific for the component, as well as not responding the default shadcn styling after export. It is possible to provide a template for such a package manager or other alternative?
Ps. I wanted to use npm because my team uses such a manager and it would be easier to standardize the resources.

Hi @matheralvs, I created a branch that uses npm as package manager. Hope it works as intended.
Check it out here.

Perfect @dan5py, everything worked fine here, however it got to the point I wanted to discuss, I usually use a src folder to allocate my development files (contexts, functions, constants and etc) in next projects, however I saw that in the next projects package tsconfig next, you passed a path like:
 "baseUrl": ".",
 "paths": {
   "@/*": ["../../packages/ui/*"]
 }

What happens is that in my web app, I always try to import using an alias like @/, it always references the ui package and not the src of the folder I'm using, send a screenshot to illustrate:

However, placing an alias in the tsconfig of the web app, referencing the web folder, as in:
 "paths": {
   "@/*": ["./src/*"]
 }

The ui package is no longer able to read your index.tsx imports. I think this was the error that popped up for me every time I tried to adapt your template using npm.

Thank you for the descriptive explanation! I just updated the template.
Now by default you can use the "@/" that references the relative app folder. Remember to add the baseUrl: "." to the tsconfig.json if you add any other app.
p.s. the ui package now uses the `@ui/" alias.

Perfect @dan5py, it worked perfectly fine here. However I can't apply tailwind styles itself, as in:

That's really strange. Have you removed the import for globals.css from layout.tsx by any chance?

That's really strange. Have you removed the import for globals.css from layout.tsx by any chance?

No no, I just pulled the updates you made and tried to apply those styles, however it didn't apply.

Have you tried restarting the server?

Have you tried restarting the server?

Hey bro, I got it here, restarted the app, cleared the cache and it's working again. Before hand @dan5py thank you very much for your attention and feedback, this repo will be very useful for me and my team, improvements or suggestions, I'll come back here and open an issue.
Ps.: What a great job you are doing on shadcn-ui, I'm a huge fan of the library, thanks for making it the perfection it is today. 😁

Thank you so much, I really appreciate it.


readme: fix badge urls

Two of the badges were pointing to a different project by mistake; fixed the CodeQL URL and removed the codecov one

Thanks!


Please add Roll forcing.


Wrong label on for select box on load

When you have separate values and labels for a select box in the metadata, it fails to load the correct label for the box from the value already in the metadata.
Edit: Actually it doesn't matter whether you use a label or not, it just flat out doesn't load the current value in the metadata at all.

Confirmed this behavior. The docs don't say anything about setting a default value which makes me think this was never a feature anyway, but I think it'd be a nice to have.

Nope, I was wrong, this is a bug in which the default select value doesn't get chosen.


Cannot change preferredStatusBarStyle on ComposeUIViewController

The usual way to change the status bar style on iOS is to override the var preferredStatusBarStyle:
override var preferredStatusBarStyle: UIStatusBarStyle {
     return .lightContent
}

override func viewDidLoad() {
    super.viewDidLoad()

    setNeedsStatusBarAppearanceUpdate()
}

However, ComposeUIViewController() returns a final UIViewController and there is no way to change this value. Maybe ComposeUIViewControllerConfiguration could grow extra nullable blocks that could be implemented to provide extra customization.

I'll forward this API, as a temporary workaround, you can wrap your ComposeUIViewController into UIViewController with overriden properties

Please check the following ticket on YouTrack for follow-ups to this issue. GitHub issues will be closed in the coming weeks.


Update the fortran code to match JHUGen updates

@hroskes @gritsan https://github.com/JHUGen/JHUGen/pull/177 requires this PR to be merged. Would you please add your review so that we can merge this PR?


Fix mute function in FF and Safari

Hi Guys,
video.setAttribute('muted', 'true'); doesn't work well on Firefox and Safari.
So instead to setAttribute(), I changed to set the property directly by video.muted = true;.
I tested on Chrome 59, Firefox 54, IE Edge, Safari (iPad and iMac) and it is working as expected.

Oh sorry! I re-included because the examples are using the javascript that is hosted on build folder.
The index.html gets the vrview js from "build/vrview.js".
I think that http://googlevr.github.io/vrview/examples/video/index.html stopped to work because "build" folder was removed.
What do you think about it? Anyways, I'll remove this from the .gitignore. It doesn't make sense to be on this PR.

No, http://googlevr.github.io/vrview/examples/video/index.html works fine (have you tried the link?). I have a gh-pages branch that does have the build folder for that exact reason. The googlevr.github.io/vrview is hosted from gh-pages branch.
Why did you close the PR? Seemed like a good fix...

Ohh the 'gh-pages' thing is awesome!
I closed cause I sent additional commits. I'll create a new branch and pull request again. I think it is better.


Updates to latest LTS Java 21 and recent maven

This updates to the latest LTS version of Java 21.
This also updates maven as it has drifted a while, and removes a jlink workaround that seems no longer necessary.
from my mac silicon after running build per README
$ docker run --rm openzipkin/java:test -version
openjdk version "21.0.1" 2023-10-17
OpenJDK Runtime Environment (build 21.0.1+12-alpine-r0)
OpenJDK 64-Bit Server VM (build 21.0.1+12-alpine-r0, mixed mode, sharing)
$ docker run --rm --entrypoint /bin/sh openzipkin/java:test -c 'mvn -version'
Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /java/maven
Java version: 21.0.1, vendor: Alpine, runtime: /usr/lib/jvm/java-21-openjdk
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "6.5.0-10-generic", arch: "aarch64", family: "unix"

cc @NishikantThorat we can try your ppc64le after this, though there may be other problems in the downstream images (zipkin uses the base to test elasticsearch etc, and also we may be moving off alpine anyway). In other words, I'm not sure if folks besides you will be able to support it.
Marking draft until https://github.com/openzipkin/docker-alpine/pull/35

https://github.com/openzipkin/zipkin/pull/3606 matches this version of maven


[Bug] sanic-ext seems to be instantiated twice when running in debug mode

Describe the bug
When the Sanic app is ran with debug=True, it seems like sanic-ext is being instantiated twice (maybe due to the reloader?).
Screenshots
Debug mode:

Non-debug mode:

To Reproduce
Create a Sanic app, load sanic-ext into it and run it using debug mode.
Expected behavior
Only one instance of sanic-ext is instantiated, or at least it's only logged once
Environment (please complete the following information):

OS: Windows 10 Home 21H1
Browser: N/A
Version: Sanic-Ext 21.9.3; Sanic 21.9.3; Routing 0.7.2

Additional context
N/A

Not a bug. It is logging twice because you are extending in the global scope, which is executed once in the main process and once in the worker process. This will behave differently starting in the next release based upon https://github.com/sanic-org/sanic/pull/2308


Alternative to splat argument insert/update/delete in RDF::Mutable and RDF::Writable

This PR introduces new methods in RDF::Writable and RDF::Mutable to complement the existing RDF::Writable#insert, RDF::Mutable#insert, RDF::Mutable#update and RDF::Mutable#delete methods with a non-splat argument alternative.
The problems with splat argument syntax:

it is limited to 130000 statements or so, as seen here
it is extremely inefficient when passing a computed Array (or other Enumerable) of statements (converting a 10000 element Array into splat argument can be 100 times slower as just passing the Array directly), as seen in JuanitoFatas/fast-ruby#76.

With the new methods RDF::Writable#insertn, RDF::Mutable#insertn, RDF::Mutable#updaten and RDF::Mutable#deleten, one can safely refactor code like this:
repository.insert *update_repository
repository.delete *(old_subject_statements + old_object_statements)

To this:
repository.insertn update_repository
repository.deleten(old_subject_statements + old_object_statements)

This is a big enough change that it is likely something we'd consider for the upcoming 2.0 release. My prefer nice would be to accept an enumerator as an argument to insert/delete rather than create a new method signature. We're taking another look at making the use of enumerators (or at least enumerables) for results in 2.0, so ready-considering their use as arguments makes sense.

Hi @gkellogg this is also my preference, since it pollutes the API. I tried changing the existing API to be backward compatible but that was not pretty (some statements are already Array of triples or quads), that's why I suggested duplicating the affected methods.

These changes aren't necessary, as all methods taking splat arguments can already take a single RDF::Enumerable argument. Documentation is updated in a branch related to issue #192 to reflect this, to be included in RDF.rb 2.0.

Thank you very much @gkellogg , looking forward to RDF.rb 2.0!


[WIP] Newest PR/issue

Still needs work.

This is massive and I broke it somewhere, so I'm redoing it as smaller PRs. So much wasted effort. ;~;

Handled in #83, #85.


ci: asynchronous + only modified files

Issue: closes #95
Description
This PR introduces optimizations to the CI by modifying the bash verifier script. Key changes include:

Selective Testing: The script now identifies and tests only the Cairo programs that have been modified.

Asynchronous Execution: The updated script executes tests in parallel.

Error Tracking Across Processes: A temporary file-based system is used to track errors across these asynchronous processes.

Refined Logging and Error Handling.

Struggled a bit, sorry for the flood of commits but should be good to merge now @julio4

@julio4 changes requested done!

@julienbrs Thank you for addressing the earlier changes.
I modified it a bit:

Configure upstream when not set (for fork and clones)
Fetch listings instead of changed files
Only execute format and test if build step is passing
Removed parallel executions: there's less listing to test so it's not too long to execute, this make error output better.

Also some changes in the github actions:

PR: execute verify script on changes
push to main: deploy mdbook without executing verify script

@julienbrs Thank you for addressing the changes. I modified it a bit:
* Configure upstream when not set (for fork and clones)

* Fetch listings instead of modified files

* Only execute format and test if build step is passing

* Removed parallel executions: there's less listings to test so it's not too long to execute, this make error output better.

Also some changes in the github actions:
* PR: execute verify script (only on changed listings)

* push to main: deploy mdbook without executing verify script

Let me know if you agree with these changes

Your changes make sense, I agree with everything


Bump @ethersproject/signing-key dependency from 5.7.0 to 6.x

Dependabot complains that:
<EMAIL_ADDRESS>requires<EMAIL_ADDRESS>via a transitive dependency on<EMAIL_ADDRESS>
If I'm reading it right, the CVE severity is set as critical.
This is fixed in elliptic 6.5.7.
I'm uncertain if the ethersproject has combined all their packages into one, or dropped their dependency on elliptic, but it doesn't appear in their 6.x package declaration anymore.
The latest @ethersproject/signing-key in their 5.7.x series, 5.7.2, still depends on the same elliptic version.
The latest<EMAIL_ADDRESS>(and the master branch) depend on ethersproject dependencies 5.7.
How much of a stretch would be to bump ethersproject's dependencies to 6.x?

Thanks @pirj. We're removing this dependency in https://github.com/OpenZeppelin/merkle-tree/pull/56.


Show all negatives as red doesn't show negatives in the pacing column as red.

Would it be possible to allow the "Square Negative Numbers" feature be enabled for all numbers in nYNAB that have a background fill color.  Right now, only RED negative numbers get the square background fill - it would be nice to use the same square look for any number that has a colored background fill ie: amber, green.

If you had the show all negatives as red option turned on as well, would that solve the problem for you?
If not, I can think of where negative numbers are amber (credit cards) but are they ever green? Where does that happen?

Negative numbers are amber in the PACING column.  So if you are pacing towards an over budgeted category at the end of the month, the pacing number has an amber background fill but with rounded edges. It would be nice if it too was Square instead of Rounded.

In that case, it sounds like we should make the "show all negatives as red" actually show negatives as red in the pacing column as well and that'd solve this.
Does that sound like a good approach to you too?

This one is kind of tricky, 'cause @iamnafets once set the feature up to show negative pacing as 'hot' (red) and toned it down after feedback. Then, for consistency (and interoperability with colourblind mode) we actually take and use YNAB's available category balance CSS selectors. Choosing to use 'negative' instead of 'cautious' as the CSS selector would make them square when that setting is enabled, but we are back with 'hot'.
Maybe @iamnafets has an idea on how to tackle this. The other thing we might do is use JS to scan for any negative balance and the override border-radius to 0 anytime we see the - symbol.

Whichever way would make it easiest for implementation but allow a consistent look with squared edges instead of rounded edges, that would be great!

Yeah, @Niictar, I think if they've specifically chosen the "show all negatives as red" option, then negatives in the pacing column should turn red. Maybe the best implementation is to stop using YNAB's classes and just doctor up the pacing column with .negative-value or something, then have the show negatives as red feature can target both classes so we don't have to change what's already working.

I'm thinking of implementing an update this way:

Update description for 'highlight negatives red' to better describe it as 'highlight negative balances red'
Add rule that will apply square styling to negative values in the pacing column
Look at adding a toggle to the Pacing column that will let users choose a red 'hot' highlight in addition to the existing available grey/amber toggle for negative values.

I think that that would provide the most consistency and flexibility when all of the three options are enabled at the same time.

Yeah, I think the interaction between these features is tricky. Your solution sounds as consistent as we can make it.

I realize this was requested forever ago... But I've added a bugfix that will appropriately highlight the Pacing column value in red or with square edges if the "Highlight all Negative Category Balances Red" or "Square Negatives Mode" features are enabled.
This change should be included in the next version of the Toolkit.


Failed Update AutoRest C# version to 3.0.0-beta.20240112.2

Update AutoRest C# version to 3.0.0-beta.20240112.2

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.ResourceManager


Fixing up IRGen tests that use attribute no-frame-pointer-elim.

Changes I made to swift (PR #26263) for llvm cfe r366645 have caused
these tests to fail because the string attributes have changes. This
patch fixes them to use the new frame pointer elision attributes.

@swift-ci please test and merge

@vedantk - the tests are still broken on master-next, you can actually run the normal pre-commit tests when committing to master-next.  This improves the state of the tests, but its not completely fixed yet.


Compatibility with 'vanguard' data format including negative DST offsets.

In Fedora, there was recently announced this change proposal [1]:

As of tzdata-2018e, the upstream will now default to using the 'vanguard' data format including negative DST offsets.

Because we are using the system tzdata to provide the time zone information, I wonder if the tzinfo is compatible with this updated format. Since the tzinfo-data was already updated to tzdata-2018e, I assume that the answer is "yes, tzinfo supports the new format just fine", but I'd like to double check.
Thx

Yes, tzinfo supports the vanguard format introduced in tzdata 2018d that became the main format in 2018e.
Conversions to and from local time will be accurate with any release supporting system zoneinfo files.
I would recommend using the latest v1.2.5 release though. This includes a fix to allow negative daylight savings time offsets to be derived from zoneinfo files. This sets the TimezonePeriod#std_offset and TimezonePeriod#utc_offset attributes correctly when loading zoneinfo time zones that contain negative DST offsets.

@philr Thanks a lot for details


Allow to use setIn for deep-nested values

Hi!
When using Documents, sometimes I need to set a value which parent node doesn't exists (yet) but that ends up in an error:
> d = yaml.parseDocument("foo: bar")
Document {
...
> d.setIn(["a", "b"], 1)
Error: Expected YAML collection at a. Remaining path: b

The above can be done executing d.setIn(["a"], {b: 1}) but I was thinking that it would be helpful for the method setIn to do that for you, similar to how lodash works:
> a = {b: 1}
{ b: 1 }
> lodash.set(a, "c.d", 2)
{ b: 1, c: { d: 2 } }

Do you think this is something doable? Thanks in advance!

That could work. There's a corner case that'll need to be settled with a path like ['foo', 3]: Should the created object be a map or a sequence?

Finally got around to implementing this. Went with non-negative integer keys creating sequences + everything else creating maps, as that's what Lodash also does.


Number of GPUs not automatically detected on a single node instance

Testing the codebase on an AWS instance and, if /job/hostfile is not present, you need to add num_gpus to the config to get training working.
Could (should?) we autodetect the number of GPUs if nothing is specified?
if not, we should add a more informative error message. This is the current traceback if num_gpus isn't specified and /job/hostfile isn't present:
Traceback (most recent call last):
  File "./deepy.py", line 67, in <module>
    old_style_args, conf = ConfigMonster().consume_args(extra_conf=extra_conf)
  File "/home/ubuntu/gpt-neox/megatron/config_monster.py", line 379, in consume_args
    ds_runner_conf, megatron_conf, ds_config_conf = self.derive_params_and_split(conf)
  File "/home/ubuntu/gpt-neox/megatron/config_monster.py", line 318, in derive_params_and_split
    world_size = ((num_gpus / pp_size) / mp_size)
TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'

fixed


Update build tools and dependencies

Update to the SDK 24 and also corresponding libraries.

Sorry for the large number of commits.

Should I close this PR and open a fresh one as this one is already too cluttered?

When we merge the PR I'll be squashing the commits so they won't show up in the master branch like that, up to you if you want to clean it up before hand.

I'm not surely how I can squash the commits. So it would be great if you do it. Thanks

Closing for cleaned up version #255


Flattened Lists

Currently, the flattenTranslations function also flattens lists. For example:
"some.key" : ["some", "list", "of", "values"]

is flattened to
"some.key.0" : "some",
"some.key.1" : "list",
"some.key.2" : "of",
"some.key.3" : "values"

When translating longer paragraphs (Terms&Conditions for example) it is nice to just have a list of them without having to know how many entries that list has. I also don't see any other reason why a list should be flattened.
I vote for keeping lists in. I can also create a PR for this.

Hi @nkoehring,
Thank you for raising this issue. We actually never thought about this use case. I will adapt the flattenTranslations function to not flatten arrays and see where this will take us.
We should probably also think about how objects in an array should be handled.
"some.key" : [
    {title: "title one", content: "some content"},
    {title: "title two", content: {name: "myname"}}
]

// should probably be transformed to
"some.key": [
    {title: "title one", content: "some content"},
    {title: "title two", "content.name": "myname"}}
]

I see where you want to go but I wonder if this is really worth much additional work. I can't imagine a reason why people should create such a data structure for their translations. Do you know any example? Would be interesting to see.

Cannot really think of an example. I just added the changes and pushed it to npm as version 1.3.0. We assume, that lists would usually be composed only of strings. Therefore we added a console warning if this is not the case. For list of strings variable replacements are supported as usually.


Needs proper working examples for Java Tool Installer (for Windows/Mac/Linux)

This task is impossible to use without a working example for all three operating systems. The documentation does not cover the intrinsic details of where the JDK archive (ZIP/TAR.GZ) will be placed, where it will be extracted, how JAVA_HOME will be set, and how it can be used later.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 182d85e3-f124-b440-78cb-2b1c52617ae0
Version Independent ID: 26372d15-3e1a-0e8b-f5bb-abf0165f6480
Content: Java Tool Installer task - Azure Pipelines & TFS
Content Source: docs/pipelines/tasks/tool/java-tool-installer.md
Product: devops
GitHub Login: @andyjlewis
Microsoft Alias: alewis

Pinging @DavidStaheli

Keep-alive

This worked for me...
  - task: JavaToolInstaller@0
    inputs:
      versionSpec: "11"
      jdkArchitectureOption: x64
      jdkSourceOption: localDirectory
      jdkFile: "/builds/openjdk-11.0.2_linux-x64_bin.tar.gz"
      jdkDestinationDirectory: "/builds/binaries/externals"
      cleanDestinationDirectory: true

@darrenfurr The installer does not describe how to drop the file (the .tar.gz for example) in a location one can reference from the yaml configuration.
It would also be helpful to reference a publicly hosted file (https URL) instead of a storage endpoint or a local file.
Lastly, versionSpec is tricky these days. Is it 1.11? 1.8? 8.0? 11.0? What's the standard and how is this version going to be used?

@brunoborges - Valid points & it would be nice if the pipeline task supported that. I was just trying to provide an example that worked for me using a local file.

So there's a type in the documentation - "azureStorage" should actually be "AzureStorage". This is an example of a working JavaToolInstaller task that pulls the JDK from Azure Storage. The file in Azure Storage should be an archive (.zip, .gz, etc) of the JAVA_HOME directory (so it includes the bin, lib, include, jre, etc directories).
- task: JavaToolInstaller@0
  inputs:
    versionSpec: '6'
    jdkArchitectureOption: 'x64'
    jdkSourceOption: AzureStorage
    azureResourceManagerEndpoint: gss-build-artifacts
    azureStorageAccountName: gssbuildassets
    azureContainerName: jdk
    azureCommonVirtualFile: 'jdk1.6.0_45.zip'
    jdkDestinationDirectory: '$(Agent.ToolsDirectory)/jdk6'
    cleanDestinationDirectory: false

@brunoborges tried for many an hour to get this tool to work. Keeps saying "Directory does not exist". Tried using your jdkDestinationDirectory with no luck.
##[error]Directory does not exist: C:\builds\binaries\externals\com\bin
##[debug]Processed: ##vso[task.issue type=error;]Directory does not exist: C:\builds\binaries\externals\com\bin
##[debug]task result: Failed
##[error]Directory does not exist: C:\builds\binaries\externals\com\bin
##[debug]Processed: ##vso[task.issue type=error;]Directory does not exist: C:\builds\binaries\externals\com\bin
##[debug]Processed: ##vso[task.complete result=Failed;]Directory does not exist: C:\builds\binaries\externals\com\bin
If I manually add com\bin then I don't have the next sub-folder then the next after that. cleanDestinationDirectory false or true has no bearing on the result-- still the same outcome.
Another question regarding the jdkFile-- is this the src.zip inside the jdk location when it is installed? Currently trying to use jdk8 x86 (latest install from Oracle is jdk1.8.0_212). What in the world are we to do here?

Sorry for all the trouble with this.  We fixed issues in the docs and added examples based on the above (thanks!).  The updated docs should be live within a couple of days.
@jvhgamer if you are still encountering the problems, can you please post a snippet of your YAML here so that we can help diagnose?

@davidstaheli could you please link the commit/PR to this issue so I can take a look?

Hi @brunoborges.  Here is the PR.  Any additions/suggestions are welcome.

Hi, quick question, where or how can I get the azureResourceManagerEndpoint?


added calling expressions on closures

This change is

crates/cairo-lang-semantic/src/expr/compute.rs line 769 at r3 (raw file):

        if let Some(var) = get_variable_by_name(ctx, &variable_name, path.stable_ptr().into()) {
            is_shadowed_by_variable = true;
            if ctx.are_closures_in_context {

why is this needed? Is this an optimization?
Code quote:
if ctx.are_closures_in_context {

crates/cairo-lang-semantic/src/expr/compute.rs line 812 at r3 (raw file):

                let args_expr = Expr::Tuple(ExprTuple {
                    items: args,
                    ty: TypeLongId::Tuple(arg_types.clone()).intern(db),

why is this cloned?
Code quote:
arg_types.clone()


Dota2 trace playback doesn't render with Release validation layers

Playing back the Dota2 trace from our trace archive with Release validation layers enabled, almost nothing is rendered.  Playback without validation renders as expected.  Depending on the LVL commit used, random splotches of landscape are sometimes rendered but never a whole frame. Playback with Debug validation layers seems to render as expected, but the slowness of playback make this hard to be sure of.  This is on Windows, I haven't tried Linux. This started happening with commit b06851c3 "layers: Update cmake for safe_struct codegen" and it doesn't happen using the commit previous to that one.  Removing the unique_objects layer made playback work as expected.  It's understood that the Dota2 trace is old and may not reflect current Dota2 Vulkan usage, and that the trace gets validation errors as a matter of course, but it has always rendered with validation layers enabled until the above commit.
As another data point, I traced today's Dota2 with the end of VulkanTools master and tried to play it back with validation layers and it would not play back with either debug or release layers.  It plays back fine without validation enabled. I didn't do anything to be sure it's the same issue as seen with the old trace.
The old Dota trace is available on lunarg-backup3:~dropbox/Tony/dota2.vktrace

Fix pushed to Master and SDK branches.


⚠ 366.ru has degraded performance

In 894da52, 366.ru (https://366.ru) experienced degraded performance:

HTTP code: 200
Response time: 1599 ms

Resolved: 366.ru performance has improved in 93e0a73.


open .mat with scipy.io.loadmat

Description
What steps will reproduce the problem?

after loading in to the variable explorer .mat file, I was clicking in the feature with type void2624, size (1,1), and value ndarray object of numpy module
the date is within this structure.
thanks
Traceback
  File "D:\Users\lapi2\Anaconda3\lib\site-packages\spyder\widgets\variableexplorer\collectionseditor.py", line 476, in createEditor
    if not editor.setup_and_check(value, title=key, readonly=readonly):
  File "D:\Users\lapi2\Anaconda3\lib\site-packages\spyder\widgets\variableexplorer\arrayeditor.py", line 661, in setup_and_check
    readonly, xlabels, ylabels))
  File "D:\Users\lapi2\Anaconda3\lib\site-packages\spyder\widgets\variableexplorer\arrayeditor.py", line 540, in __init__
    ylabels=ylabels, readonly=readonly, parent=self)
  File "D:\Users\lapi2\Anaconda3\lib\site-packages\spyder\widgets\variableexplorer\arrayeditor.py", line 154, in __init__
    self.vmin = np.nanmin(self.color_func(data))
  File "D:\Users\lapi2\Anaconda3\lib\site-packages\numpy\lib\nanfunctions.py", line 294, in nanmin
    res = _copyto(res, np.nan, mask)
  File "D:\Users\lapi2\Anaconda3\lib\site-packages\numpy\lib\nanfunctions.py", line 106, in _copyto
    a = a.dtype.type(val)
AttributeError: 'float' object has no attribute 'dtype'
WARNING:spyder.widgets.github.backend:failed to send bug report on github. response={'code': 401, 'json': {'message': 'Bad credentials', 'documentation_url': 'https://developer.github.com/v3'}}

Versions

Spyder version: 3.3.1
Python version: 3.7.0
Qt version: 5.9.6
PyQt5 version: 5.9.2
Operating System: Windows 10

Dependencies
pyflakes >=0.6.0  :  2.0.0 (OK)
pycodestyle >=2.3 :  2.4.0 (OK)
pygments >=2.0    :  2.2.0 (OK)
sphinx >=0.6.6    :  1.7.9 (OK)
rope >=0.9.4      :  0.11.0 (OK)
jedi >=0.9.0      :  0.12.1 (OK)
nbconvert >=4.0   :  5.4.0 (OK)
pandas >=0.13.1   :  0.23.4 (OK)
numpy >=1.7       :  1.15.1 (OK)
sympy >=0.7.3     :  1.1.1 (OK)
cython >=0.21     :  0.28.5 (OK)
qtconsole >=4.2.0 :  4.4.1 (OK)
IPython >=4.0     :  6.5.0 (OK)
matplotlib >=2.0.0:  2.2.3 (OK)
pylint >=0.25     :  2.1.1 (OK)

I cannot reproduce this behaviour with 3.3.3. Could you include MWE with sample data? Or try to update to the latest Spyder 3.3.3, Numpy 1.16.1and Scipy 1.2.1.
It is also possible that there is something wrong with your input mat file.

@elap2, this looks like an installation problem so you'd probably be better by reinstalling Anaconda and trying again.

I cannot reproduce this behaviour with 3.3.3. Could you include MWE with sample data? Or try to update to the latest Spyder 3.3.3, Numpy 1.16.1and Scipy 1.2.1.
It is also possible that there is something wrong with your input mat file.

import scipy.io
import os
os.chdir('directory_to_be_imported')
mat = scipy.io.loadmat('name_of_file.mat')
does spyder accept the objects from .mat workspaces?
thanks

Please upload a mat file that it's generating this error so we can test it in our side.

Thanks Daniel, Thanks Cordoba
Please close the case, the scipy.io function loads the .mat variables correctly as mapping types => type(var) output: dict
I solved the problem as follow:
mport scipy.io
import os
os.chdir('directory_to_be_imported')
mat = scipy.io.loadmat('name_of_file.mat')
type(mat) # == dict
var = mat['key'].item(i)[j] #i and j are indexes of the variable
and so on with the code to get the values


Better way to mark defunct organizations

Currently you mark some orgs as "DISSOLVED" in the legal field but maybe that should be a separate field.
You can also add:

Mambo Foundation
LogiLogi Foundation
Open Health Tools
Open Media Now! Foundation

Good point, we'll review them, thks!


[Help with implementation]

Hello,
I am trying to implement a set of custom keymappings to use with buffers that have neovimLSP running on them. My setup script for the LSP calls this file to setup the key mapping for the buffers I open. However these key mappings are not being registered
Could someone help me understand whats wrong in my code?
local M = {}

local whichkey = require "which-key"

local keymap = vim.api.nvim_set_keymap
local buf_keymap = vim.api.nvim_buf_set_keymap

local function keymappings(client, bufnr)
  local opts = { noremap = true, silent = true }

  -- Key mappings
  buf_keymap(bufnr, "n", "K", "<cmd>lua vim.lsp.buf.hover()<CR>", opts)
  keymap("n", "[d", "<cmd>lua vim.diagnostic.goto_prev()<CR>", opts)
  keymap("n", "]d", "<cmd>lua vim.diagnostic.goto_next()<CR>", opts)
  keymap("n", "[e", "<cmd>lua vim.diagnostic.goto_prev({severity = vim.diagnostic.severity.ERROR})<CR>", opts)
  keymap("n", "]e", "<cmd>lua vim.diagnostic.goto_next({severity = vim.diagnostic.severity.ERROR})<CR>", opts)

  -- Whichkey
  local keymap_t = {
    t = {
      name = "Code",
      r = { "<cmd>lua vim.lsp.buf.rename()<CR>", "Rename" },
      a = { "<cmd>lua vim.lsp.buf.code_action()<CR>", "Code Action" },
      d = { "<cmd>lua vim.diagnostic.open_float()<CR>", "Line Diagnostics" },
      i = { "<cmd>LspInfo<CR>", "Lsp Info" },
    },
  }
  if client.resolved_capabilities.document_formatting then
    keymap_t.l.f = { "<cmd>lua vim.lsp.buf.formatting()<CR>", "Format Document" }
  end

  whichkey.register(keymap_t, { buffer = bufnr, prefix = "<leader>" })
end

function M.setup(client, bufnr)
  keymappings(client, bufnr)
end

return M

Are you calling require("<this_script>").setup{<client>, <buffer>} anywhere? In some BufEnter autocmd perhaps as you scope to buffers?
You don't seem to be passing bufnr to the which-key -wrapped vim.*.buf.* commands, not an expert here but I think those are scoped to specific buffers (so the LSP knows the language I assume)

Look here for examples: https://github.com/neovim/nvim-lspconfig#suggested-configuration


borrowed ast and default builtin types to immutable - String -> Box etc

An alternative to this is that we could try using compact_str to avoid allocating for strings shorter than 24 bytes.


Escape HTML characters in list of suggestions

fix #16932
Generate manual DOM subtree for each marked instance of the queried string instead of generating an unsafe HTML string

I have hopefully adjusted the code to the coding style.
I may be missing something for the replace function - while it would work fine to generate the mark elements, I don't see how to use it succinctly to escape the HTML characters that would be outside of the matched parts of the string.

I have hopefully adjusted the code to the coding style.

Thanks!

I may be missing something for the replace function - while it would work fine to generate the mark elements, I don't see how to use it succinctly to escape the HTML characters that would be outside of the matched parts of the string.

That made me realize that as currently written, this is backwards incompatible: HTML in options works right now, whereas if this is merged, it would stop working and possibly break numerous use cases. Ideally we want something that keeps HTML that was there from the beginning, but doesn't convert special characters to HTML when highlighting them. I.e. something that's XSS-safe, but doesn't restrict what you can do in the initial options. Does that make sense?

re backwards-compatibility, I don't think it can be preserved:

right now, the API is ambiguous about the type of content in label - when rendered in the list options, it is rendered as HTML, but when rendered in the <input> field, it is rendered as plain text (since it goes through the value attribute).
it's not really a matter of highlighting the content of the label with  - the XSS risks exists even when no highlighting happens - if the list of label options contains a malicious


Build on Windows

Gtk3 is available for Windows.
The problem is figuring out how to install Gtk3 dev and gintrospection, then get to a point where the gobject shard is running (you can compile and run samples).
Once Crystal and Gtk3 are setup, you should be able to build an executable for Windows?

Probably blocked by https://github.com/crystal-lang/crystal/pull/13419 since the app relies on MT, or we shall use another hack (call Fiber.yield in GLib.timeout_milliseconds callback) on Windows.


Fix All Current Codacy Issues

We still have 29 issues on Codacy. These should be fixed.

Fixed in #9


Implement "log out everywhere" functionality

This requires changes to the Redis session store, as right now we're unable to determine which tickets are for which user.
Note that just replacing the security stamp won't work as the tickets have somewhat long lifespan.
The proper way would:

change the security stamp,
remove all tickets for the user from Redis,

thus forcing an immediate invalidation of all sessions.

This has been split from #6 as that issue was already too large for Redis changes.


Crash When Clustering Pins on MapBox 6.2.0

I upgraded from 6.1.3 to 6.2.0 and I now get this crash when loading my map with a cluster manager. Also all of my map related code is executed on the main UI thread.
    com.mapbox.mapboxsdk.exceptions.CalledFromWorkerThreadException: Map interactions should happen on the UI thread. Method invoked from wrong thread is .
        at com.mapbox.mapboxsdk.maps.NativeMapView.checkState(NativeMapView.java:104)
        at com.mapbox.mapboxsdk.maps.NativeMapView.getWidth(NativeMapView.java:1094)
        at com.mapbox.mapboxsdk.maps.Projection.getVisibleRegion(Projection.java:96)
        at com.mapbox.mapboxsdk.plugins.cluster.clustering.view.DefaultClusterRenderer$RenderTask.run(DefaultClusterRenderer.java:379)

I updated the plugin to 0.4 but that made no difference.

@tobrun I'd love to get some feed back on this. I would really like to update to 6.2.0 because of some of the issues on 6.1.3. However all maps that have clusters crash on 6.2.0.

Wow that's very unfortunate. Seems like clustering would be a common desired feature.

@simon-the-canadian , did you got any workaround solution?

Yeah, I dropped MapBox and went back to Google Maps. Google just changed its pricing policy on maps so that its free on mobile. Pretty good timing.

I wanted to support both Google Maps and Mapbox because some of the users do not have Google Play services installed on their devices by default.
So I used the changes in https://github.com/mapbox/mapbox-plugins-android/pull/49 to add clustering support for mapbox.
I also faced this error when calling getVisibleRegion.
So I made a workaround:
In class ViewModifier

Modify the constructor to use the main looper

public ViewModifier() {
super(Looper.getMainLooper());
}

In class RenderTask:

Add a property mVisibleBounds and create setter:

private LatLngBounds mVisibleBounds;
public void setVisibleBounds(LatLngBounds visibleBounds) {
this.mVisibleBounds = visibleBounds;
}

Replace the call
final LatLngBounds visibleBounds = mProjection.getVisibleRegion().latLngBounds; with
final LatLngBounds visibleBounds = mVisibleBounds;
Replace the call
renderTask.setProjection(projection) with
renderTask.setVisibleBounds(projection.getVisibleRegion().latLngBounds);
Delete the now useless property mProjection and its setter setProjection from RenderTask

This workaround works because renderTask.setVisibleBounds is called by the main looper using the UI thread but everything inside the RenderTask is executed by a worker thread.

@DavisDevasia I tried with fix. It working fine.


Tracking issue: Release primitives v0.100.0

This is the game plan I'm working off, please consider it theoretical unless there is a linked PR

[x] Literally empty crate: #2950
[x] Move opcodes module into primitives: #2989
[x] Move scriptpubkey stuff out of Script into an extension trait: #2955
[x] Move locktimes and Sequence to primitives: #2991
[x] Decide on and implement re-export policy: #3037
[x] Move script module into primitives: #3431
[ ] Put trait AsRawTransaction into internals and use it to implement push_decode::Encoder
[ ] Move Transaction and friends into primitives
[ ] Move Block into primitives with optional Block::compute_blockhash

The primitives crate is our top priority, please consider prioritising review of PRs related to this work, I'll use the milestone to help filter. Also I'll use P-High label to indicate the next step required.

Idea for the text above, credit to @apoelstra (normal caveat, all mistakes are my own).

Also, my normal caveat about not being that experienced with project management, holla at me if you want to see something different that is more helpful.

(instead of wrapping eg hashes::sha256::Hash)

What does it mean and why do it that way?

What does it mean and why do it that way?

Our intention is to make hashes optional. I think we can do this and only lose a few things (e.g. Transaction::compute_txid) if the hash types themselves are just dumb 32-byte array wrappers.

Our intention is to make hashes optional.

:open_mouth: isn't it too much? Although it could help @benma and we wouldn't have to create funny hash_newtype_api_stable_v1 macro. :thinking:

I had initially thought it was too much, but then we realized how easy it would be :). And true! If @benma was willing to implement his own compute_txid method (not hard, our impl is 4 LOC and only calls public methods) then he'd be able to use his own hashes impl without any crazy configure flags or anything.

Thanks for thinking of me :smile: I don't need compute_txid though at all, and dead code elimination means it's not a real concern it's optional or not (it just helps a bit to avoid accidentally pulling it in).
Even if I used compute_txid and had a re-implementation for it, the hashing functions of bitcoin-hashes still end up in my binary because they are used in many other places where re-implementations would not be worth it. https://github.com/rust-bitcoin/rust-bitcoin/issues/647#issuecomment-2226969049 would work a lot better as a general solution for me.

I'm lost as to what is the next step for primitives. Are we prioritising push_decode and creating a consensus-encoding crate or are we moving consensus stuff to primitives first?

Personally I think we should copy the existing encoding traits into primitives as private traits so that we can use them internally to implement compute_txid etc.
It's ugly but it'll get us moving forward.

Aight, thanks, I remember now you saying this before. I'm likely to be a bit sketchy this week with the jet lag and all :)

I don't think I can manage to do it until the end of this week but hopefully I can the next one.
Also maybe rather than copying, just don't move compute_txid yet. Let primitives be just PODs for now.

@Kixunil it's not just compute_txid, it's also several other methods, and most importantly, fmt::Display, which can't be added via extension trait. So I think maybe it's worth the ugliness to sneak the encoding stuff in. And the main cost, code size, is not really too important here because our target use case is demoing API usage, not releasing a production-grade verson of the library.
OTOH, I took a quick glance at miniscript and we only use compute_txid in the PSBT module, which we could feature-gate or disable in a primitives demo.
So maybe we should try to move forward without, and before release, assess how much extra functionality would be needed for rust-miniscript (as a prototypical primitives user). And if we're pretty-much there we can go ahead and just release 0.1 -- semver-tricking 0.1.1 to use a later version that adds fmt::Display and other missing functionality -- but if we find that we need a dozen extra methods and they all depend on encoding, that's a sign that we should just put the hidden encoding traits in.

Do we really have to have Display on Transaction from the start? It's not currently in the released API anyway. We can have a method in extension trait for now or just let people call consensus_encoding::serialize_hex.
Also I was unsure what the fuss with miniscript is about but now I've remembered you want to break the dependency loop and export miniscript from bitcoin. Honestly I think this is really not super important and it can stay as-is for one more release.

semver-tricking

No semver-tricking should be needed and if we're in situation where it would be it probably won't be possible/usable at all.
Disclaimer: I also want primitives to be reasonably useful. E.g. if we can use it together with p2p in electrs that could nicely improve compile time. However given we have developer time constraints I think a more realistic approach is to get something usable out and then improve it progressively rather than wait an release a huge chunk of work.
Also, I've now realized that bitcoincore-rpc might be a better prototypical primitives user.

We can have a method in extension trait for now or just let people call consensus_encoding::serialize_hex.

Ok, sure, as long as consensus_encoding exists and its traits are implemented for everything in primitives as soon as we're ready to release. We absolutely cannot have a "lol we removed hex but don't worry we'll replace it in 6 months" situation like we did with rust-bitcoin 0.30.

Also I was unsure what the fuss with miniscript is about but now I've remembered you want to break the dependency loop and export miniscript from bitcoin. Honestly I think this is really not super important and it can stay as-is for one more release.

If we want this to be usable and get testing, then we should take miniscript seriously.

No semver-tricking should be needed and if we're in situation where it would be it probably won't be possible/usable at all.

It's never needed. But I think it would be possible, and useful, if it turned out that primitives 0.2 had a Transaction type which was the same as the one in primitives 0.1 except that it also had fmt::Display.

Also, I've now realized that bitcoincore-rpc might be a better prototypical primitives user.

I guess you mean bitcoind-json-types and bitcoind-json-client, since bitcoincore-rpc is basically unmaintained. But yes, getting the RPC crates to use a stable version of primitives would be a godsend.

Ok, sure, as long as consensus_encoding exists and its traits are implemented for everything in primitives as soon as we're ready to release.

Yeah, I was always planning to do that.

We absolutely cannot have a "lol we removed hex but don't worry we'll replace it in 6 months" situation like we did with rust-bitcoin 0.30.

We did not remove hex formatting, just made Display the primary API. Big difference. The real problem was that (a) people were depending on bitcoin_hashes::hex for their arbitrary code or to cover our API holes (I think, I'm not sure) and didn't want to change it and (b) didn't want to change to_hex to to_string for some reason.

If we want this to be usable and get testing, then we should take miniscript seriously.

I mean it felt to me like minisript is some "king crate" that determines which API is good while there are multiple other projects that have both similar and different needs. Thus any reasonable set of crates should work and doesn't require miniscript if the entire API is used.

if it turned out that primitives 0.2 had a Transaction type which was the same as the one in primitives 0.1 except that it also had fmt::Display.

Then we would do primitives 0.1.1 rather than 0.2.

I guess you mean bitcoind-json-types and bitcoind-json-client, since bitcoincore-rpc is basically unmaintained.

Haven't noticed it was forked already. Nice.

Ok, great, agreed on all counts (other than the history of hex, but that's water under the bridge).

There are a lot of moving parts here to remember, if I push PRs that contradict any "current" ideas you hold please take it simply as I forgot not that I'm trying to push a different way. I'll try to keep pushing discreet PRs that are reviewable and mergable and that move us somewhat in the direction we are going.

If you want to use an issue this one is as good as any I'd say. I'm totally open to learning better ways/tools to do project management. I don't think I use github that well for more complex projects like this primitives work.

I just wanted to track every single step required to stabilize primitives. Including things like tagged script. But I'd change the title to say "stabilize".

I renamed this issue and added https://github.com/rust-bitcoin/rust-bitcoin/issues/3133

Relevant comment about release strategy primitives v0.100.0 and v1.0.0: https://github.com/rust-bitcoin/rust-bitcoin/pull/3110#issuecomment-2258471476

what is the status of this issue?

I edited the main text then realized doing so makes the thread stale.

Can you remind me or link to where we decided to release primitives starting from 0.100.0?

100 is actually released already, I updated the title. We did it because we took over the crate from the C++ transpiling project, remember the lad doing that kindly gave it to us.

Ahh that's right, thanks!

I removed the AsRawTransaction thing from the list, that is @Kixunil's along with push decode. We might be ready to release before he is back.

Hey @apoelstra I went through the 2 pages of issues that show up if one filters by 'primitives' and I don't see anything urgent for v0.101.0 - do you want to take a quick look sometime please?

We are getting close, closing this in favour of the release tracking PR.
I believe the AsRawTransaction stuff may not be needed because we manually implemented compute_txid but I may be missing something.

LGTM. And yes, I view compute_txid as critical functionality, but pretty-much nothing else is critical.


Wsgiapp: Only raise exception if an error occurred.

Currently, mitmproxy always raises an exception when serving a wsgiapp even if everything worked as expected. This patch causes mitmproxy to only raise an exception if an error actually occurred.

Good catch, thanks.


As a FAPO, I want to filter, sort, and easily digest the CBG data, so that I can see the information most relevant to my programs (design CBG .xlsx Excel styling)

Description
We want to support federal agency program officers in being able to filter, sort, and easily digest the data shown for census block groups in the community list.
Solution
Design for styling of the .xlsx, potentially including columns to distinguish location information (state, county, CBG ID) from score information, from "bonus" indicators
Describe alternatives you've considered
Links to user research or other resources
Related card with task for column headers https://github.com/usds/justice40-tool/issues/303
Tasks

[ ] Create .xlsx mockup showing desired column and column header styling

Definition of "Done"

[ ] .xlsx for packet is styled consistent with design mockup

Test comment

test comment

Adding Glennette as design assignee

Beth has shared document plus descriptions of indicators for Glennette to mockup what Excel styling could look like

Here is the mockup for the .xls file.
There are three tabs:

Dataset definitions
Score data
Non-score data

Screening Tool dataset download for .xls

Updated mockup for the .xls file. Passed Excel's accessibility checker.
Four tabs:

About the score
Dataset definitions
Score data
Non-score data

Screening Tool Datasets download

Duplicate of #303

this ticket is specifically around styling, .xlsx file that is now part of the download packet has readable column names in bold, but still has more formatting and explanatory tabs to be added. Adding Blocked label as this is blocked until indicators to be included are finalized. We can format and add explanatory tabs to meet Design spec as a manual step for launch.


buffer: disconnect clients that commit an unknown buffer type

If this happens, something went really wrong and we should do more
than just logging an error. See 1.

Thanks!


Add yahoo support

Sorry, but I do not accept pull requests on the master branch:
https://github.com/nraboy/ng-cordova-oauth#contribution-rules
Please clone the repository, checkout the development branch, and then do your changes there.  The development branch is always a few commits ahead, so you must checkout rather than creating a new development branch from the master branch.
Best,


Set Content-Type to application/json.

As of 2019/11/24 at around 8:30 PM EST, Discord started requiring this for webhooks. Otherwise, it thinks you aren't sending a message.
Tested the cURL path. file_get_contents() should work, but is untested.

Looks like my local branch didn't rebase properly; there was already a fix here.


KAFKA-4083: prevent changing replication factor via partition reassignment

Summary
ReassignPartitionsCommand does not check whether the replication factor of a partition is the same as the existing replication factor, which may result that different partitions of a topic have different numbers of replicas.
A few years ago, I attempted a fix to ReassignPartitionsCommand within https://github.com/apache/kafka/pull/1779, but now I think it should be fixed on the broker side.
Tests
Besides the integration test I have changed, I built a 3-node Kafka cluster on my laptop with Docker containers and ran a few shell scripts to test.
Create a topic
bin/kafka-topics.sh --bootstrap-server localhost:29092 --create --topic test-reassign-partitions --partitions 3 --replication-factor 3

reassignment-plan.json
{
   "version":1,
   "partitions":[
      {
         "topic": "test-reassign-partitions",
         "partition": 0,
         "replicas" :[1, 2]
      }
   ]
}

Execute a partition reassignment
bin/kafka-reassign-partitions.sh --bootstrap-server localhost:29092 --reassignment-json-file reassignment-plan.json --execute

The output is
Current partition replica assignment

{"version":1,"partitions":[{"topic":"test-reassign-partitions","partition":0,"replicas":[2,1,0],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Error reassigning partition(s):
test-reassign-partitions-0: Replica assignment on test-reassign-partitions-0: [1, 2] violates its replication factor

The fix seems still relevant to me, @chia7712 wdyt? Could you review it?

While I agree that all partitions ought to have the same number of replicas, I think this fix is enforcing something different: That you can't change the RF for a partition. While that prevents the partitions getting different RFs from each other if they had the same RF originally it also prevents changing the RF for all the partitions of a topic, which we should support.
There's also the use case where you want to change the RF for a topic, but because it has a large number of partitions you want to do it in batches, using multiple reassignments. I think Cruise Control does this kind of batching, for example. Supporting that requires that, for a time, the topic has partitions with different numbers of replicas.

While I agree that all partitions ought to have the same number of replicas, I think this fix is enforcing something different: That you can't change the RF for a partition. While that prevents the partitions getting different RFs from each other if they had the same RF originally it also prevents changing the RF for all the partitions of a topic, which we should support.
There's also the use case where you want to change the RF for a topic, but because it has a large number of partitions you want to do it in batches, using multiple reassignments. I think Cruise Control does this kind of batching, for example. Supporting that requires that, for a time, the topic has partitions with different numbers of replicas.

Thanks, @tombentley. I totally agree that changing RF via partition-reassignment API is a very valid use case. May I add this piece of logic as a solution:
If a partition reassignment attempts to change RF, its plan must change the RF of all partitions of a topic to the same number. what do you think?
One more thing to confirm with you: partitionAssignments in KafkaController contains the all topic partitions in the entire cluster, right? I see this line that all topic partitions are loaded into the controller upon initiation.


[5.3] Fix an issue that doctrine/inflector requires PHP 7

There is an issue related to doctrine/inflector as after updating packages the doctrine/inflector is upgraded to version 1.2.0 which requires PHP 7.0, however Laravel 5.3 should support PHP 5.6

5.3 isn't supported any more, this "issue" has been fixed in newer versions of Laravel

composer should find the best match based on other packages being included and your php version
to encounter this issue i would guess you have a different version of php on your development machine compared to your server


http: superfluous ...WriteHeader call from ...Render

2021/03/01 16:12:19 http: superfluous response.WriteHeader call from go.mindeco.de/http/render.(*Renderer).Render (templating.go:164)
This is the HTTP server code saying "hey! somebody tried to set the response code twice!" The problem is in the http/render package. This issue is here to track that.

We also need a way to customize the error handler to show error pages in the expected design and not as plain text without any HTML.


feat: add CLI keyboard shortcuts

Description
This is a continuation of #3182 with the following changes:

Rebased onto main branch
Add bindShortcuts method to ViteDevServer
Added h shortcut for toggling HMR updates (useful when debugging)

⚠ Do not squash!

What is the purpose of this pull request?

[ ] Bug fix
[x] New Feature
[ ] Documentation update
[ ] Other

Hi @aleclarson, is this still on your radar?

@joelmukuthu I'm using it in a fork without issue. Would you like to rebase this and open a new PR?

@aleclarson done via https://github.com/vitejs/vite/pull/9673. If you'd prefer, I can close that PR and create one targetting your feat/shortcuts instead so your PR here is updated. Just let me know!


PHA4GE template to GISIAD export: change separators

I think that a | as a separator may cause problems during upload to GISAID.
Under "Specimen source" in the export, can we have a comma separate the parts?

Under "Assembly method" in the export, can we just have a space between the consensus software name and version number?

done!


"Usage Example" isn't an example.

Refer http://todobackend.com/ for an easy to fork technology that would allow you to pass 'specs' 9in page, JavaScript) in a build for a virtualizaed backend.
The example should perhaps showcase recording/capturing, and how that recorded data could be used programmatically to mock out the backend for the sake of passing the JS specs.  Note: recording/capture can't just keep a map of responses, as there's statefulness in the interactions, thus weld having an (re)programmable mode is key.

Thanks for your comment, some interesting ideas in there, as you might be able to tell I didn't ever develop weld beyond it's initial proof of concept. But maybe it's worth revisiting and turning in to something useful.

OK - well opensource can be a huge commitment. Others in the field include Mountebank, Hoverfly, Bokor, WireMock, machzqcq/service_virtualization, epam/Wilma, and BaiduQA/terminator


Better KLayout TilingProcessor scaling

PR #1535 added tiled fill, thanks to @sebastian-goeldi and @SkandanC
Using default parameters and simply changing n_threads results in poor scaling:
Sample cell:

n_threads
Script time (s)

1
real    2m54.031s

user    2m55.305s

sys     0m13.799s

5
real    1m27.213s

user    2m55.673s

sys     0m13.970s

10
real    1m16.487s

user    2m56.166s

sys     0m13.860s

4x larger cell:

n_threads
Script time (s)

10
real    4m1.849s

user    10m41.205s

sys     0m13.980s

20
real 3m37.944s

user 10m39.971s

sys 0m14.395s

I've noticed only a small part of the overall script actually launches n_threads.
Is there any advice on how to better improve the parallelization? Are there optimal ways to define fill_layers vs fill_regions and tile_size?

How much it scales with number of threads entirely depends on your machine and the problems. Tl;dr I can only think of one problem and I am not even sure I left that in the code (the fill cell layers might be in the build part of the exclusion, that might not be smart).

Your machine has more cores available than threads you are using
You aren't running out of memory (swap doesn't count)
You are using relatively few layers to calculate the area to fill
The resulting areas are almost exclusively complex (many points, non-trivial shapes, like doesn't result in rectangles or similar)
You tried multiple sizes for the tiles.

If any of this is true, I am almost certain that your filling is spending most of the time in the TileOutputProcessor part. That part is not (safely) parallelizable, see https://www.klayout.de/forum/discussion/550/using-the-fill-tool
If you want to dive deeper into this issue, I suggest to use a proper profiler and see where python spends the most time.

If you can, avoid fill regions and just use layer indexes

Thanks Sebastian, this is great advice.

Ok, in that case your main benefit of the tiling processor comes from splitting the region to be filled. For large regions it can take ages to allocate memory and also to make the region object due to the sheer amount of polygons.
You could also do this manually and it might be faster, but I doubt that tbh.
But as you have seen, such a case doesn't benefit from multi-threading a lot.  This is a classic problem in parallel-computing if you have a work load that is largely dominated by one function that is not further parallelizable (this is also the reason why nowadays a lot of programs and games don't take advantage of the number of cores available in machines).

moved to gplugins


Allow wildcard dependencies for compilation

If mixins or colors (in less, scss or similar) are splitted in multiple files it would be nice to address them in the config.json with the wildcard sign.
Example:
"app.css": [
			"+assets/css/variables.scss",
			"+assets/css/mixins.scss",
			"+assets/css/variables/*.scss",
			"+assets/css/mixins/*.scss",
			"assets/css/reset.css",
			"assets/css/basic.scss",
			"components/**/css/*.scss",
			"components/**/css/skins/*.scss"
		],

instead of
"app.css": [
			"+assets/css/variables.scss",
			"+assets/css/mixins.scss",
			"+assets/css/variables/_baukasten.scss",
			"+assets/css/variables/_colors.scss",
			"+assets/css/variables/_constants.scss",
			"+assets/css/variables/_utils.scss",
			"+assets/css/mixins/_arrows.scss",
			"+assets/css/mixins/_border-dotted.scss",
			"assets/css/reset.css",
			"assets/css/basic.scss",
			"components/**/css/*.scss",
			"components/**/css/skins/*.scss"
		],

@beatgeb can you verify that 4ff7b83 fixes this?
The issue's not completely solved for me, though. If you're already watching and add new dependencies to config.json, gulp will not import those.
I will look into this tomorrow and am hopeful to be able to provide a fix soon.

@beatgeb config is now reloaded when changes.
Fixed by 035886e.


Windows GUI: Basic dark mode

A basic dark mode for Windows GUI that can be toggled in the Options menu. It also follows the current Windows dark mode on application start as well as when Windows dark mode is changed while the application is running.
Issue #672
It has the following limitations:

Some pages might still be displayed in light mode.
It is only tested on Windows 11 and it is not known how it looks on other Windows versions or how it behaves on Windows versions that do not have dark mode settings.

Depends on Windows11 Dark style by Embarcadero Technologies from Getit Package Manager.

Windows11_Modern_Dark.vsf is not on our build machine (also not on my home machine), maybe installed only on Win11 machines? I don't want Win11 to be mandatory for building the tool.
Please use e.g. Windows10Dark.vsf.

Windows11_Modern_Dark.vsf is not on our build machine (also not on my home machine), maybe installed only on Win11 machines? I don't want Win11 to be mandatory for building the tool. Please use e.g. Windows10Dark.vsf.

I got that from the Getit Package Manager as mentioned in first post.

If it is not possible to add this then I can switch to Windows10Dark.vsf. However, Windows10Dark.vsf has two issues:

It is totally black so MediaInfo logo is not visible in ABout window and maybe also the title bar.
It does not have rounded corners on the windows when run on Windows 11.

If it is not possible to add this then I can switch to Windows10Dark.vsf.

Oops, I missed that, thanks for the reminder.

Windows10Dark.vsf has two issues:
It is totally black so MediaInfo logo is not visible in About window and maybe also the title bar.

Annoying.
Let's try with Windows11_Modern_Dark.vsf, @g-maxime please install it on the build machine.

this build has this patch.
@cjee21 please rebase from our main branch.

I'm working on getting it to follow Windows dark mode settings. Will re-base and add one more commit if successful.

I'm working on getting it to follow Windows dark mode settings.

Appreciated!

I'm working on getting it to follow Windows dark mode settings.

Please keep the manual change option, at least in preferences if not in the menu directly.

No idea how to pass the current mode over to the other file. I'll get it pushed here first and maybe you can have a look.

It is more classic stuff on our side, something we could manage.

It is more classic stuff on our side, something we could manage.

The problem is that the code to set dark mode on application start is in MediaInfo_GUI.cpp while the manual toggle is in GUI_Main.cpp. Can't find a way to get current mode. It's my first time using C++Builder and doing VCL. I tried putting the code to set dark mode on application start in GUI_Main.cpp at first but can't find a place for it that works properly.

It's my first time using C++Builder and doing VCL

Kudos for having taken the time to learn it and check doc about the different features.

I tried putting the code to set dark mode on application start in GUI_Main.cpp at first but can't find a place for it that works properly.

I didn't touch this code in the last several years, I don't remind but I'll give it a look. Maybe not this week, however.

Kudos for having taken the time to learn it and check doc about the different features.

Glad to be able to contribute to MediaInfo while learning useful things along the way and getting more familiar with Git/GitHub.

I didn't touch this code in the last several years, I don't remind but I'll give it a look. Maybe not this week, however.

Take your time. I am not in a hurry to get this merged. I don't really need dark mode, just helping implement it since it seems doable for me.

Good news! I managed to solve the issues with Windows dark mode integration. I will update this pull request once I finish integrating it into MediaInfo.

Basic dark mode implementation is completed. I have updated the first post to reflect the current state of this PR so please take a look.
One possible improvement is to enable the user to choose if they want to follow Windows dark mode setting or set MediaInfo to always be in light/dark mode. I will not be implementing this as I am not familiar with how MediaInfo handles preferences.

One possible improvement is to enable the user to choose if they want to follow Windows dark mode setting or set MediaInfo to always be in light/dark mode. I will not be implementing this as I am not familiar with how MediaInfo handles preferences.

Fair enough, we'll manage that (should be for other things too, we'll implement it when we have time, except if you are motivated for that ;-) ).

I also do not know if this is good enough to consider issue https://github.com/MediaArea/MediaInfo/issues/672 resolved so I did not mark this as closing that issue.

It is, flagged.
build.

The file open dialog is not high DPI.

Why? Not handled by a component?
And it is not the only issue, the dark mode is buggy there:

But I guess this is not in your control. Or maybe you can manage to have another open windows (maybe hight DPI aware)?
Not blocking.
But (see next post)...

I guess both blurry open window in high DPI mode and no drag and drop in dark mode is caused by the VCL style. Drag/drop in light mode still works.

Drag/drop in light mode still works.

Not on my side, so it seems to be due to a part of the code which breaks that, there is a need to add code step by step and see what breaks the drag and drop.

🤔 Is this the reason/solution? https://stackoverflow.com/a/10337744

Please try :)

I guess both blurry open window in high DPI mode [...] is caused by the VCL style.

I think that it is due to an open box not supported by the style manager, but it is maybe because this is not a "good" open box. I see that C++ Builder itself has support of dark mode and add a good support of dark mode with the open box, which is different. I would be fine to switch to this open box.
Would you mind to see how to have the other open box and change to it?
PS: this is the exact reason we were reluctant to implement it on our side with impact on our sponsors requests, "just some lines of code, quick" is the expected duration then reality shows up 😄.

Drag and drop should work in all situations now :)

That's one more limitation gone. First post updated. The open file dialog is now available in both dark mode and high DPI but the open folder dialog is not available in dark mode. This is now Windows' fault for not yet implementing dark mode for some dialogs. 😆

By the way, should I make another PR for changing the font of 'Text' view to a newer, nicer font? I have already implemented it. It is because I noticed the current font does not look good and scale well on high DPI settings.

By the way, should I make another PR for changing the font of 'Text' view to a newer, nicer font?

I am fine with that, if it stays monospace.
Generally speaking, we are interested in lot of improvement for the Windows GUI, we didn't touch it for years due to priority elsewhere (more on the core of MediaInfo = format detection/analysis), so don't hesitate to provide hints about what you would like to implement and we say if we could accept it or not.

(just to be transparent: we are thinking to switch to Qt on Windows in order to have a fully open source workflow + removing one UI toolkit because Qt is already used on Linux and is more cross platform, but well, we are thinking about that for several years now, so it won't be immediate, and we are not yet sure Qt will fit at 100% on Windows)

Noted.

Also on Windows 11? Crazy... It is not the most difficult to update, but I guess that it is not used enough for MS to care about that...

Even this file properties window is not implemented....

You can see the background is file explorer which shows dark mode is enabled.

open folder dialog is not available in dark mode.

while working with VirtualBox, I see that there is another "open folder" windows! And in dark mode!
("Dossier" means "folder")

It seems to be a system one (2 other windows in the screenshots seems to be styled by VirtualBox itself internally, the "open folder" window seems to be from the system with a different style), maybe there is something callable directly in C++ Builder or there is a system call to find...

Even this file properties window is not implemented....

And this one is used by a lot of people... Not serious like Apple is...

It is only tested on Windows 11 and it is not known how it looks on other Windows versions

Tested on Win10 and is totally fine.
On Win7 it is mixed:

But expected (software style is by C++ Builder and integrated in their libraries, as for Qt, and the system windows are not aware of dark mode).

@JeromeMartinez Just to let you know, I just discovered there are still some bugs with this dark mode. I will not be fixing them as I have no idea what is the cause.

Every time dark mode is toggled, the currently open file will be re-added to the list of open files.

If dark mode is toggled while in another view such as text view, an error may pop-up and the view will switch back to the default view.

I just discovered there are still some bugs with this dark mode.

Thanks for letting us know, we'll fix that before the release.

Hi thanks for the efforts to bring dark mode.  Just wanted to no what version I need of mediainfo for it to be available.

Just wanted to no what version I need of mediainfo for it to be available.

The next public version or the build listed in https://github.com/MediaArea/MediaInfo/pull/836#issuecomment-2124663164.

Just wanted to no what version I need of mediainfo for it to be available.

The next public version or the build listed in #836 (comment).

But the build in that comment doesn't include the drag & drop or dialog fix right?

But the build in that comment doesn't include the drag & drop or dialog fix right?

Right.
latest build.

For opening folders, maybe can consider using this other 'Select Folder' dialog instead. This screenshot is from Mp3tag.
Some advantages include dark mode support, ability to copy paste directory path and view directory history in the address bar, ability to select whether to include subdirectories, as well as possibility of having thumbnail view of folders and viewing folder contents.

I looked at MediaInfo code and see that it appears to use a complex custom implementation from ZenLib for handling the folder opening so it is not something that I will touch.

I looked at MediaInfo code and see that it appears to use a complex custom implementation from ZenLib for handling the folder opening so it is not something that I will touch.

ZenLib should not be an issue there, just dealing with provided path from the system dialog, please provide a PR with a PoC and I check if there are issues.

ZenLib should not be an issue there, just dealing with provided path from the system dialog, please provide a PR with a PoC and I check if there are issues.

Took some time to figure out but turns out to be very simple. 😮 😅
Making PR now...


[pure localization ERROR] when i run the demo_backpack_2d_localization.launch

In the actual operation of the robot, with IMU and ODOM, it is normally running at the beginning, but after running for a period of time, the following errors will occur, resulting in the failure of normal operation. I hope you can help me, thank you very much
*** Check failure stack trace: ***
    @     0x7f81b84a05cd  google::LogMessage::Fail()
    @     0x7f81b84a2433  google::LogMessage::SendToLog()
    @     0x7f81b84a015b  google::LogMessage::Flush()
    @     0x7f81b84a2e1e  google::LogMessageFatal::~LogMessageFatal()
    @           0x6ae916  cartographer::mapping::optimization::OptimizationProblem2D::AddOdometryData()
    @           0x6c6bd1  _ZNSt17_Function_handlerIFN12cartographer7mapping8WorkItem6ResultEvEZNS1_11PoseGraph2D15AddOdometryDataEiRKNS0_6sensor12OdometryDataEEUlvE_E9_M_invokeERKSt9_Any_data
    @           0x6ca884  cartographer::mapping::PoseGraph2D::DrainWorkQueue()
    @           0x6d367b  cartographer::mapping::PoseGraph2D::HandleWorkQueue()
    @           0x71f9f2  cartographer::mapping::constraints::ConstraintBuilder2D::RunWhenDoneCallback()
    @           0x678ab1  cartographer::common::Task::Execute()
    @           0x676959  cartographer::common::ThreadPool::DoWork()
    @     0x7f81b6293c80  (unknown)
    @     0x7f81b7b186ba  start_thread
    @     0x7f81b59f94dd  clone
    @              (nil)  (unknown)
[ WARN] [1604721104.902795190]: Costmap2DROS transform timeout. Current time:<PHONE_NUMBER>.9026, global_pose stamp:<PHONE_NUMBER>.3629, tolerance: 0.5000
[ WARN] [1604721104.903309763]: Could not get robot pose, cancelling reconfiguration
[cartographer_node-10] process has died [pid 8074, exit code -6, cmd /home/lyh/catkin_carto/install_isolated/lib/cartographer_ros/cartographer_node -configuration_directory /home/lyh/catkin_carto/install_isolated/share/cartographer_ros/configuration_files -configuration_basename backpack_2d_localization.lua -load_state_filename /home/lyh/lyric_ws/src/Core/lyric_maps_manager/scripts/map/lyric_05.pbstream echoes:=scan imu:=imu_data __name:=cartographer_node __log:=/home/lyh/.ros/log/6458be20-20ac-11eb-9ce9-70e04f682b80/cartographer_node-10.log].
log file: /home/lyh/.ros/log/6458be20-20ac-11eb-9ce9-70e04f682b80/cartographer_node-10*.log
[ WARN] [1604721105.903731810]: Costmap2DROS transform timeout. Current time:<PHONE_NUMBER>.9035, global_pose stamp:<PHONE_NUMBER>.3629, tolerance: 0.5000
[ WARN] [1604721105.904021126]: Could not get robot pose, cancelling reconfiguration
[ WARN] [1604721106.940985045]: Costmap2DROS transform timeout. Current time:<PHONE_NUMBER>.9408, global_pose stamp:<PHONE_NUMBER>.3629, tolerance: 0.5000
[ WARN] [1604721106.968826656]: Could not get robot pose, cancelling reconfiguration

  map_frame = "map",
  tracking_frame = "laser_frame",
  published_frame = "odom_combined",
  odom_frame = "odom_combined",
  provide_odom_frame = false,
  publish_frame_projected_to_2d = false,
  use_pose_extrapolator = true,
  use_odometry = true,
  use_nav_sat = false,
  use_landmarks = false,
  num_laser_scans = 1,
  num_multi_echo_laser_scans = 0,
  num_subdivisions_per_laser_scan = 1,
  num_point_clouds = 0,
  lookup_transform_timeout_sec = 0.2,
  submap_publish_period_sec = 0.3,
  pose_publish_period_sec = 5e-3,
  trajectory_publish_period_sec = 30e-3,
  rangefinder_sampling_ratio = 1.,
  odometry_sampling_ratio = 1.,
  fixed_frame_pose_sampling_ratio = 1.,
  imu_sampling_ratio = 1.,
  landmarks_sampling_ratio = 1.,
}

MAP_BUILDER.use_trajectory_builder_2d = true

TRAJECTORY_BUILDER_2D.submaps.num_range_data = 35
TRAJECTORY_BUILDER_2D.min_range = 0.3
TRAJECTORY_BUILDER_2D.max_range = 8.
TRAJECTORY_BUILDER_2D.missing_data_ray_length = 1.
TRAJECTORY_BUILDER_2D.use_imu_data = false
TRAJECTORY_BUILDER_2D.use_online_correlative_scan_matching = true
TRAJECTORY_BUILDER_2D.real_time_correlative_scan_matcher.linear_search_window = 0.1
TRAJECTORY_BUILDER_2D.real_time_correlative_scan_matcher.translation_delta_cost_weight = 10.
TRAJECTORY_BUILDER_2D.real_time_correlative_scan_matcher.rotation_delta_cost_weight = 1e-1

POSE_GRAPH.optimization_problem.huber_scale = 1e2
POSE_GRAPH.optimize_every_n_nodes = 35
POSE_GRAPH.constraint_builder.min_score = 0.65

A CHECK() statement in the code failed. Normally it prints the details of the check in the log before the stacktrace, somehow this seems to be missing in your log.
But since it happens in the MapByTime::Append() call, it's probably this check here that fails: https://github.com/cartographer-project/cartographer/blob/fdcf0eb0df72b2eb7ae30c08c873b5ab2b855239/cartographer/sensor/map_by_time.h#L43
This happens when the timestamps of the sensor data are not strictly increasing. Cartographer is very picky here, so you need to ensure that your odometry, IMU and scan topics have correct timestamps.

Thank you so much for your assistance. i had solve this problem for using my correct topic of odom_tf ,because it is publish by robot_pose_ekf


Improving documents

Improving documents

@jiminwen-msft for future reference, PRs should not be merged without the approval of the the Azure SDK team.


Add anchor on features so links are easier to share

Resolving as duplicate of #167 (this is older, but #167 contains more info).


Command based ready checks

At the moment we only support HTTP based ready checks, which limits the kinds of services people can host on the platform.
We should aim to also support a command style ready check which executes within the container being checked (rather than within it's own container):
readyCheck:
    frequency: 2
    maximum: 30
    command:
      command: ["stat" "/ready.txt"]
      expectedExitCode: 0

This could essentially use the ContainerExec API from Docker.


client: add --datadir flag as alternative to ENV VAR

https://github.com/ark-network/ark/blob/master/client/flags/flags.go#L11-L17


No Linux release for <IP_ADDRESS>

In release <IP_ADDRESS>, there was this Linux x86_64 tarball in the releases. For <IP_ADDRESS>, there is only a Mac tarball. I don't really want to have to build from source.

I ended up building from the latest source since I couldn't get the source to build from the v<IP_ADDRESS> tag. I don't recall the error.

Okay sorry about that- but you were able to get it built from head?

Yes, I was.

If you're curious, here's the code for my docker image.

Aha thanks. How big is the resulting image?

Here's the uncompressed size:
$ docker images
REPOSITORY         TAG         IMAGE ID       CREATED      SIZE
rzuckerm/granule   <IP_ADDRESS>-4   9eac70670373   2 days ago   235MB