Announcing the raster server

Shields has changed the way it presents raster badges – i.e. badges in PNG, GIF, and JPG format.

https://img.shields.io/v/npm/express.png
↪https://raster.shields.io/v/npm/express.png

What has changed?

img.shields.io renders only SVG and JSON badges.
Previously it also supported PNG, GIF, and JPG.
PNG badges are rasterized downstream at raster.shields.io

For backward compatibility, the PNG URLs will 301 redirect to raster.shields.io

Support for GIF and JPG has been removed.

Note: The legacy raster static badge route, https://img.shields.io/label/message.png has been discontinued. (#3659) URLs of that form should be updated to use one of the modern routes:

SVG: https://img.shields.io/badge/label-message-color
PNG: https://raster.shields.io/badge/label-message-color

Why was this changed?
Shields has always been about SVG badges first, because they look better and perform better.
PNG badges are a fringe but meaningful use case for badges in emails, and services like Slack which don’t support SVG.
Moving rasterization off-board simplifies the responsibilities of the badge server and hardens the server, as it prevents someone from overloading the server with raster requests. (Each raster request used a subprocess, making them much more expensive than SVG badges.)
GIF and JPG were removed because, on a monthly basis, these were barely touched.
Read more: #3112
How does it work?

The new service is implemented as a web function using the micro framework.
The server code is here: rasterize.js
Rasterization is handled by convert-svg-to-png which uses headless Chromium via Puppeteer.
The service is hosted on Zeit Now 2.0.

Notes

If you self-host a copy of Shields and need to support raster badges, you will need to host your own copy of the raster server. You can do that inexpensively on Zeit Now or on your own hardware.

If you've gotten set this up, we'd welcome better documentation! (#3648)

The raster server will always return PNG, and will work even if you don't replace .svg with .png in the URL (e.g. https://raster.shields.io/v/npm/express.svg)

Acknowledgements
Thanks to the Shields maintainer team, to @styfle whose post about Serverless Chrome on Now 2.0 helped me get this off the ground, and to @neocotic for convert-svg.
Feedback
Your feedback on the new feature is much appreciated! Feel free to comment in this issue with feedback or questions, or in the #support room in Discord.
Related issues

[x] #3631
[x] #3644
[ ] #3648
[ ] #3647

Thanks for taking on this piece of work Paul, it's nice to have separated this out!

Thanks for the acknowledgement! Great to see my little tool being used. I realised that I haven't touched it in a little while so might go back and see how it's faring. As a user and fan of shields.io, I was wondering if there's anything that you'd like to see specifically.

Thanks for asking @neocotic!
I guess there are two things that might be useful:

I'd love to support retina png's… though I actually have no idea what is involved in that!
Support for puppeteer-core would be nice in Shields' case, and might be useful for other users as well. I think this would simplify the install. For us, installing puppeteer on Now means passing an env var that prevents Chromium from being downloaded. This wouldn't be necessary with puppeter-core (which is "bring your own Chromium").

@neocotic To add a little more context about why puppeteer-core would be useful: I'd like to publish the web function on npm, and make the app a tiny page that includes just the web function. I've had trouble doing that, in part because I can't just declare the dependencies I want. The env var, for example, must be set when npm install runs. If I could just declare the dependencies, I'd make it easier for people to host their own svg-to-image-proxy without needing to create their own fork.

I noticed that the raster example, badge/label-message-color,  sets a header cache-control: max-age=86400 but this does not take advantage of the CDN Edge.
Consider adding s-maxage like the caching docs mention:
cache-control: max-age=86400, s-maxage=86400, public

You could even add immutable for static badges that you know will never change.

Thanks so much! That could help a lot here! We've tried to avoid immutable because of changes in the template and server bugs, but setting fairly long timeouts should have a pretty similar effect.

No problem!
Also note that deploying will purge the CDN Edge cache https://vercel.com/docs/v2/edge-network/caching#cache-invalidation

@paulmelnikow - do you have a handle on how the headers are set/forwarded with the raster proxy? I know @calebcartwright is following up on https://github.com/badges/shields/issues/3647#issuecomment-621582157 but if we can cut our costs in the meantime with a header, that seems like its worth following up.

There's a short list of headers that are copied from the upstream svg response: https://github.com/badges/svg-to-image-proxy/blob/aa3486603f6e67295d1857351a1de8b47732257e/index.js#L20-L25
So the place to make this change is probably in Shields:
https://github.com/badges/shields/blob/7af965113b7c202abbb57ce6023d754ed1cf2018/core/base-service/cache-headers.js#L72
I'm not 100% sure we want public in there. We should think about whether that's a problem for self-hosting.


Support HTTPStatus IntEnum for status code return values.

I wound up using the HTTPStatus IntEnum for a status code in my func.HttpResponse returns but noticed the actual web response I was getting was always 200, even when doing something like status_code=HTTPStatus.BAD_REQUEST.
Just to verify, I did confirm that returning int(HTTPStatus.BAD_REQUEST) on my side of things seemed to fix. So maybe this is addressable in the azure-functions-python-library code (not even sure if it is a worker thing in the first place).
Minor, but it would be convenient to be able to use HTTPStatus. Thought I'd suggest the enhancement since it caused me some head scratching to figure out why I was getting 200's that didn't make sense and it should be an easy tweak.

https://github.com/Azure/azure-functions-python-library/pull/161


ESLint Overhaul

Our code doesn't seem to follow ESLint principles...
Then again, I was trying to follow ALL of the rules...Maybe that's overkill? But it'd be real nice to know that all of the code is compliant.
This will be a mission for later!

On second thought, we'll go with the recommended set of rules, and progressively enable rules over time.
https://eslint.org/docs/rules/

ESLint configured with the eslint:recommended preset seems to pass just fine into all of the code files. We seem to be good for now.
eslint:all completely shits on EVERYTHING, but it's so strict that I think it would be a great waste of time to adhere to all of those rules (that, and it gives me a headache when some rules just conflict with Webstorm's integrated inspections).
Basically, eslint:all is not worth the trouble. The code is relatively clean as it is, and since we're following a very distinct approach to ES6 (as in, following OOP concepts that are foreign to Javascript), we should be fine.


KDE Plasma Ksplash stuck

Describe the bug
After updating plasma from 6.2.0 to 6.2.1, it takes over 30-40 seconds to load instead of like 3-5
Steps To Reproduce
Steps to reproduce the behavior:

Update to latest nixos-unstable
Observe painfully long kde splash screen

Expected behavior
Plasma should start in a couple of seconds
Screenshots
N/A
Additional context
N/A
Notify maintainers

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.11.4, NixOS, 24.11 (Vicuna), 24.11.20241018.4c2fcb0`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.25.0pre20240920_ca3fc169`
 - nixpkgs: `/nix/store/riqkpszjqk02bi1wppfg8ip5xvh102qd-source`

Add a :+1: reaction to issues you find important.

Seems like shell starts normally, but
Oct 21 10:46:59 dell-pc (-session)[10419]: plasma-ksplash-ready.service: Unable to locate executable '-session': No such file or directory

[Unit]
Description=KSplash "ready" Stage
Wants=plasma-core.target
After=plasma-core.target
PartOf=graphical-session.target

[Service]
Type=oneshot
ExecStart= --session --reply-timeout=1 --type=method_call --dest=org.kde.KSplash /KSplash org.kde.KSplash.setStage string:ready
Slice=session.slice

ksplash service is broken

Maybe fixed by https://github.com/NixOS/nixpkgs/commit/ed8c856ebfc87f211e60c1699e091c39540e93b1?

Hacky workaround: (assuming dbus-send is in /etc/profiles/per-user/user/bin/dbus-send)
 home.file."/home/user/.config/systemd/user/plasma-ksplash-ready.service.d/override.conf" = {
    # fix bug where splash screen doesn't disappear
    text = ''
      [Service]
      ExecStart=/etc/profiles/per-user/user/bin/dbus-send --session --reply-timeout=1 --type=method_call --dest=org.kde.KSplash /KSplash org.kde.KSplash.setStage string:ready
    '';
  };

Should be fixed in the next channel update.


ValueError: Failed to read CAR header. Invalid uvarint

Traceback (most recent call last):
  File "/Users/awd/miniforge3/envs/bsky/lib/python3.10/site-packages/atproto_firehose/client.py", line 221, in _process_message_frame
    await self._on_message_callback(frame)
  File "/Users/awd/Projects/PhenomML/BSky2GBQ/main.py", line 91, in async_on_message
    blocks = atproto.CAR.from_bytes(message.blocks).blocks
  File "/Users/awd/miniforge3/envs/bsky/lib/python3.10/site-packages/atproto_core/car/car.py", line 51, in from_bytes
    header, blocks = libipld.decode_car(data)
ValueError: Failed to read CAR header. Invalid uvarint

Happy to help and dig deeper.
Anon,
Andrew

# <==  is the marker to the line that generated this ValueError.
async def async_on_message(message, test_function, handler):
    message = hose.parse_subscribe_repos_message(message)
    if isinstance(message,
                  atproto.models.ComAtprotoSyncSubscribeRepos.Commit):
        blocks = atproto.CAR.from_bytes(message.blocks).blocks  # <== This is the line that creates the above trace.
        for op in message.ops:
            uri = atproto.AtUri.from_str("at://" + message.repo \
                                         + "/" + op.path)
            raw = blocks.get(op.cid)
            if raw:
                record = get_or_create(raw, strict=False)
                if record.py_type is not None:
                    rdict = record.model_dump()
                    item = {
                        "repo": message.repo,
                        "revision": message.rev,
                        "sequence": message.seq,
                        "timestamp": message.time,
                        "action": op.action,
                        "cid": str(op.cid),
                        "path": op.path,
                        "collection": uri.collection,
                        "record": rdict
                    }
                    if test_function(item):
                        await IOLoop.current().run_in_executor(None, handler, item)

Hi! Today is the day when everyone tries devil payloads over the network. These are special malformed messages that raise issues like that. It would help if you would catch the value of .blocks and write to the file. And pls share. Otherwise, just catch the ValueError exception and ignore this commit

OK, I'll try to get to this after dinner.

File "/Users/awd/Projects/PhenomML/BSky2GBQ/main.py", line 91, in async_on_message
blocks = atproto.CAR.from_bytes(message.blocks).blocks

I think it is worth to add check that blocks is not empty :)


Unhandled JS Exception: TypeError: Cannot set property 'merchant_email' of undefined

@Zakyyy Did you solve this issue? I am also facing same issue in my app. Please suggest solution.

Please try this version of the package:
https://github.com/paytabscom/react-native-paytabs-library/packages/81259

@AdlyPayTabs I need your help to test ApplyPay for Paytabs, happy to pay for your service<EMAIL_ADDRESS>


Replicating sklearn elastic net results in PiML

Hi,
I trained an elastic net model using sklearn in python (from sklearn.linear_model import ElasticNet). I used the following parameters: ElasticNet(alpha = 0.434, l1_ratio = 0.5) . What can I enter into the L1 and L2 input in the GLM model in PiML to replicate these results (assuming I am using the same data with the same random seed)?
Thanks

@xloffree Hi, L1 and L2 should both be 0.217.
The example is attached.

I tried this and when I run model_explain it says that all coefficients are equal to zero. What is going on here? The elastic net worked fine with sklearn and with R. Is there something wrong?
Thank you

@xloffree Can you show the example to me? In my test, our GLM result is always same to the sklearn elasticnet.

When using GLMRegressor, I get very different results than from when I use the method shown in the README, which gives a very poor R2 and says all coefficients are 0. I am certain that this is wrong because we have run elastic net on this data in R and python and have gotten similar results that show a much better R2. The results from the glmRegressor look more reasonable. I am trying to use piml to compare many models at once on the same train/test split (the elastic net models with specific parameters included). We would like to use the results from PiML in an upcoming paper, but this problem with PiML's elastic net is preventing us from being able to do so. Please let me know if there is a solution to this.
Thank you


[Tailwind CSS] Filter by class name instead of selector name, or both

Extension
https://www.raycast.com/vimtor/tailwindcss
Description
Currently the extension will filter by the selector name, where this might be an issue is when you want to find the tailwinds equivalent but it will yield no results. For example, if you search "width" it will return no results, but if you search "w-1" it will return that class.
An update to either accept only the CSS class to return the tailwindcss selector equivalent, or search both the selector or classes to return the tailwind equivalent would be absolutely amazing for reasons listed in the next section.
You can view the problematic line here: https://github.com/raycast/extensions/blob/076c06daeba71dd1c2efd621e82923046b309a33/extensions/tailwindcss/src/search-classes.tsx#L8
Who will benefit from this feature?
Anyone who might have issues with memorizing all the tailwindcss selectors, or are not able to have an extra browser tab open to search through the documentation, or someone who would like to view the tailwindcss selectors without having to navigate away from their code as it is very quick to open Raycast and start searching for what they need.
Anything else?
No response

Great idea I will try it. I don't promise anything since the list of class names is really huge and it may get a bit slow.


WPF Integration: D3D11Image Implementation

This is an integration with WPF through Direct2D / Direct3D11Image using WPFDXInterop library from Microsoft.
@kekekeks said he will review this and help me with the remaining issues.
How to build
Build the project as usual, run WindowsInteropTest, click "Embed to WPF" button.
Remaining issues

There's notable flickering when switching the tabs in the sample. We still don't know the reason for flickering, but we want to fix it before the merge.
The WPFDXInterop library is written in C++/CLI, and @kekekeks says he'd like to avoid the maintenance burden of having to produce both x86 and x64 artifacts. We're still not sure how to solve that; probably I'll port the relevant parts of C++/CLI code to C# later (and it won't be an easy task).

After a couple of experiments it seems to be working fine now.

Superceded by managed implementation: #1069


Marker with popup "Close popup" does not work

Hi. I've tried clicking on the "Close popup" link on the Marker with popup example page and received this error:

Any ideas why this happens?

Hi
Thanks for reporting this issue. Dunno how it ever works, but I released the version 7.6.2 that fix thiss issue.


Add an exception to the uppercase rule for gRPC

@michalszynkiewicz this one is just for you :).

thank you @gsmet !


Whenever give ``require': cannot load such file -- bundler/setup` error in production server

Im trying to setup whenever gem in my Ubuntu 14.04 server. In my local machine it works fine. I use capistrano to deploy the site. I use rbenv in the server. But when installing passenger it installs Ruby 1.9 which I do not use. Only use rbenv. Here's my shcedule.rb :
set :output, "#{path}/log/cron.log"

every 30.minutes do
    runner 'UploadmailWorker.perform_async'
end

every 12.hours do
    runner 'SubscriptionWorker.perform_async'
end

every :month do
    runner 'PaysellerWorker.perform_async'
end

My deploy.rb :
set :default_environment, {
  'PATH' => "$HOME/.rbenv/shims:$HOME/.rbenv/bin:$PATH"
}

set :whenever_roles,        ->{ :app }
set :whenever_command,      ->{ [:bundle, :exec, :whenever] }
set :whenever_command_environment_variables, ->{ {} }
set :whenever_identifier,   ->{ fetch :application }
set :whenever_environment,  ->{ fetch :rails_env, fetch(:stage, "production") }
set :whenever_variables,    ->{ "environment=#{fetch :whenever_environment}" }
set :whenever_update_flags, ->{ "--update-crontab #{fetch :whenever_identifier} --set #{fetch :whenever_variables}" }
set :whenever_clear_flags,  ->{ "--clear-crontab #{fetch :whenever_identifier}" }

namespace :deploy do

    desc 'Restart application'
    task :restart do
        on roles(:app), in: :sequence, wait: 5 do
            execute :touch, release_path.join('tmp/restart.txt')
        end
    end

    after :publishing, 'deploy:restart'
    after :finishing, 'deploy:cleanup'
end

After deploying, the whenever does not work. In the cron.log I find this error :
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- bundler/setup (LoadError)
        from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /home/deploy/istockseller/releases/20150519112817/config/boot.rb:3:in `<top (required)>'
        from bin/rails:7:in `require_relative'
        from bin/rails:7:in `<main>'

How can I fix that?

Same problem here

@THPubs I was able to fix it by inserting:
env :PATH, ENV['PATH']
At the beginning of my schedule.rb file

I've had the same problem of you guys and the solution is to fix the setup of capistrano so you have to remove bin from the linked dir and put binstubs to nil in order to make it working. I've been struggling for hours on that.
But if you still can't make it working properly I would suggest to use my free service to outsource the cronjobs, you create an endpoint on you app and make http GET or POST request as often as you need with http://guardiano.getpeople.in
For some of my prototype or when I don't want to struggle with cronjobs I'm using this (that is why I created it and made public) and is working fine. Hope this helps!

I am facing same issue on production. @jpmermoz the solution you suggested doesn't work either.

I figured it out. See details http://stackoverflow.com/questions/32293583/error-running-cron-job-require-cannot-load-such-file-bundler-setup-loader

Same problem here.See details http://stackoverflow.com/questions/32293583/error-running-cron-job-require-cannot-load-such-file-bundler-setup-loader

Some problem, solved with @zongyannan and @AmitPatel-BoTreeConsulting tips. But if you are using rbenv, should do this:
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(rbenv init -)"' >> ~/.bash_profile

Thank's


Update default strategy sort

On both "my strategies" and "explorer", we use the same default sort option.
current:
ascending by strategy ID (old to new, where the oldest strategy is first)
updated:
descending by ROI (where highest ROI is shown first)

fixed by #800


Usage of the wrong variable 'dist' for loading parameters in train.py

While resuming a training I noticed the output on the command line saying that the weights I am loading in are loaded to the CPU. After further inspection I found that in train.py line 129 the _to_cpu parameter is assigned dist.
https://github.com/open-mmlab/OpenPCDet/blob/8e055f02df245b1f0675e874792dd708fb84c04b/tools/train.py#L126
This seems wrong and I would assume dist_train should be used here, since dist is just an alias for the imported package import torch.distributed. Using dist leads to always loading the weights to the CPU.
I created a pull request (https://github.com/open-mmlab/OpenPCDet/pull/621) changing this.

Merged


Can anyone help me

Challenge Understand String Immutability has an issue.
User Agent is: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.
My code:

// Setup
var myStr = "Jello World";

// Only change code below this line

myStr[0] = "H"; // Fix Me
var myStr = "Hello World";

In the challenge, it specifies why you cannot change the value of myStr through myStr[0].
It also tells you the correct way to change a string.
var myStr = "Bob";
myStr = "Job";


Correction for raw affiliation IRSTEA Montpellier, UMR TETIS, Montpellier, France,LIRMM CNRS Montpellier, Montpellier, France#TAB#

Correction needed for raw affiliation IRSTEA Montpellier, UMR TETIS, Montpellier, France,LIRMM CNRS Montpellier, Montpellier, France#TAB#
raw_affiliation_name: IRSTEA Montpellier, UMR TETIS, Montpellier, France,LIRMM CNRS Montpellier, Montpellier, France#TAB#
new_rors: 013yean28;02feahw73;0458hw939
previous_rors: 013yean28;02feahw73
works_examples: W1553780760;W2952629719
contact: e9908fcef02d87eec3c69a7a453a3da8:973a19b01df0a88c959d41b3ae0586e38572 @ inrae.fr

This issue was accepted and ingested by the OpenAlex team on 2024-12-19. The new affiliations should be visible within the next 7 days.


Pathfinding: take channel capacity into account

In c-lightning (https://github.com/ElementsProject/lightning/pull/4771/files) and rust-lightning (https://github.com/rust-bitcoin/rust-lightning/pull/1166), moves have been / are being made to factor in the channel capacity relative to the payment amount.
I think that makes a lot of sense and is easily added to lnd's probability estimator. The logic could be applied in the probability estimation for channels that haven't been tried before.

Note that especially the channels which have been tried before can switch to conditional probabities as described in https://arxiv.org/abs/2103.08576 and more specifically with the concrete formulas in https://arxiv.org/abs/2107.05322
C-lightning currently ignores the uncertainty network as they would have had to change their logic and mission control does it not according to the capacity model which are suggest to create here. Of course your temporal decay of forgetting the information could persist.
I think @stefanwouldgo has code in scala that does it for lnd (even with tracking in flight htlcs) also I think @Roasbeef has seen part of the code in the repo mentioned in the second paper

Is there a centralization effect on demanding excess capital for a route?

Note that especially the channels which have been tried before can switch to conditional probabities as described in

We already do this and have for a few years now, the system will continually update these conditional probabilities and also decay them over time to reflect the uncertainty of the network overtime since our last observation. Adding capacities to the mix should be a pretty straight forward change code wise.

Is there a centralization effect on demanding excess capital for a route?

That's something we'd need to examine and way the tradeoffs for. Hopefully we'll have an initial version of the distributed A/B testing framework we're working on so we can properly test changes like this. Heuristically, one would say that if an amount is <<< than the channel size, then it should have a high success probability. However as you're getting at, we'd need to make sure individuals with very very large channels don't become a sort of attempt/retry sync.

I think @stefanwouldgo has code in scala that does it for lnd (even with tracking in flight htlcs) also I think @Roasbeef has seen part of the code in the repo mentioned in the second paper

This is another (imo) important thing that we don't currently do: in-flight HTLCs in the same payment session don't propagate back any information to the main mission control instance. This ends up triggering weird scenarios where we'll continue to attempt payment amounts that we know can't work (example if a 1 BTC channel with an 0.9 BTC HTLC present, and we try > 0.1 BTC).

if a channel capacity is 1, any attempts above 0.5 will have a high % in failing since bandwidth of a channel in practice will most of the time only be able have a portion at any time as available liquidity. Retry rule if else fails.
pay 0.1 try use channel sizes above 0.4 if not in MC

Is there a centralization effect on demanding excess capital for a route?

Agreeing with @xraid. Probably only lowering the probability for amounts that are a considerable part of the channel capacity is sufficient. I don't think it needs to be taken to extremes where a 1BTC channel is selected for a 10 sat payment over a 10M channel.

Strongly disagree with @joostjager : there is a whole theory and strong emperical evidence why -log((cap+1-amt)/(cap+1)) is a good feature In pathfinding. And also on how to adopt this formular after successful, failed attempts and with htlcs logged in. Mainly 2 question remain

how fast to decay gained information back to the full uncertain case)
How strong to enorporate it with other features?

Plot this formular for a 10M channel (in the discussion of the c Lightning pr I have done it with various other combined features for a 20k sat channel) and you will see that the cost of this feature hardly triggers for small amounts anyway due to its convex nature. So why would one choose arbitrary case distinctions of not choosing this feature for low amounts if the feature itself Already encodes such behavior?

Is there a centralization effect on demanding excess capital for a route?

Yes but that is not a consequence of the features chosen in selfish routing (c.f. Selfish routing and the price of anarchy work by Tim roughgarden https://www.amazon.de/Selfish-Routing-Price-Anarchy-Press/dp/0262182432 ) but a fundamental property of the lightning network itself that liquidity is necessary to guarantee certain service level agreements / objectives.
People who wish to send money will be selfish in making routing decisions and routing nodes / liquidity providers have to change the channel graph in adoption to the selfish behavior of users. It's the same as street planning: It needs resources so that folks who chose the fastest / quickest route can all do so without congestion. Just look at braess paradox https://en.wikipedia.org/wiki/Braess's_paradox which is actually an answer (to your concern)  that indicates that people don't want to open too large channels if we choose this feature
Anyway I am getting nerd snipes here

I think you might be interested in concrete formulas how to update your uncertainty / probabilities in case of failed / success full payment attempts. I have summarized the two papers in an issue of another PR https://github.com/rust-bitcoin/rust-lightning/issues/1170#issuecomment-972396747

Ldk (rust lightning) has not only implementated probabilistic pathfinding but also the proper maintanance of the uncertainty network with bayesian updating in this pr https://github.com/lightningdevkit/rust-lightning/pull/1227 might be easier than reading the paper to just look at how they did it. Note they have two arbitrary choices in their code:

The half life time of 3600 seconds for bayesian updates
The way they combine negative log probabilities with other features

Thanks for all the links/notes Rene! Based on current timing/velocity, I'm not sure we'll be able to properly implement, review, and test (prob the most important thing!) this change for the 0.15 release. As a result, I'm moving this to the 0.16 tracker. On the up side, pushing this back a few more months gives us more time to learn from what others are doing, and also gives us more time to draft up something that can in theory kill several birds with one stone.

That sounds reasonable to me. @Roasbeef could you take care of #5746 a bit earlier, though? This way one could use lnd with external pathfinding solutions to gather first insights. Currently, this isn't possible as described in the issue. I'd be very happy to provide a usable implementation as part of lnd-manageJ rather soon, which depends on the issue being fixed before it can be used for real MPPs.

from @Roasbeef

and test (prob the most important thing!)

I fully agree though I am very confident that you will end up doing an adoption / extension of what I propose here:  https://github.com/renepickhardt/mpp-splitter/pull/11 or the notebook directly on the branch at: https://github.com/renepickhardt/mpp-splitter/blob/pickhardt-payments-simulation-dev/Pickhardt-Payments-Simulation.ipynb
This is besides engineering challenges the gist of how I belief payments should be delivered over lightning.
In any case the notebook as well as the stuff that @C-Otto works on may yield great chances or basis for testing. In my case You could just overload the OracleLightningNetwork class (and make the pay loop async which happens natively in go I guess) and you can test against mainnet. While I start with complete uncertainty the code of @C-Otto pulls info from mission control AND (!) also collects necessary data that mission control currently does not even store (as far as I understand). success on a channel from a probabilistic point of few needs to be handled differently depending on the entire onion settleing of failing downstream, as the information about liquidity is to be interpreted differently.

For Neutrino nodes taking capacity into account may also be an issue because Neutrino isn't pulling the capacity of the channels, it's looking at the max HTLC size. You wouldn't want to use max HTLC size as a substitute for capacity there because that size is mutable and is used to help set expectations itself

try to keep your channels reasonably balanced, with respect to where coins are moving to avoid a superfluous requirement to keep your coins on a side where there is no traffic.

Even if all payment pairs were equally distributed the fact that fees are not (and taken as a feature in (selfish) routing) puts usually a direction of experienced flow on channels. (I belief this is often called drain.) While I guess this can also be seen on chain from channel closes and openings I just created a simple modle showing what failure rates we can expect on a channel because it is being depleted](https://twitter.com/renepickhardt/status/1526575556032180226) (depending on the drain / flow of coins that may come from selfish routing) I would not be surprised if @bitromortac has thoughts about this

What I try to say: of course there is a failure rate to be expected that is independent of the uncertainty of liquidity in the channel and more dependent on the drain and selfish routing of sending nodes

Motivated by @renepickhardt's work I took a look at mission control data to check the assumption of a uniform balance distribution in the LN. I computed node pair-based success probabilities grouping node pairs into capacity ranges and averaged over successes and failures to route ~500000 sat. This results in the following plot:

Comparing the data (dots) with estimates from uniform and bimodal models (curves) we see that at least for this amount, the measured points are not in line with a uniform distribution where in the limit of large channels the success probability should approach unity. (There may be unknown biases here.)
As a second step, I assumed a bimodal distribution following a liquidity distribution X of P(x) ~ exp(-x/s) + exp((x-c)/s) ("bimodal constant model"), where s is a liquidity broadening scale and c is the channel capacity. This distribution describes situations in which channel liquidity is typically found on the local or the remote side, but rather not in between (unbalanced channels). It seems like this model is able to better describe the observed data (there is a model variation "bimodal variable", which works even better, but I won't describe it here as it's similar).

Following Rene's method, I derived prior and posterior success probabilities for the bimodal distribution.
Comparison Bimodal and Uniform model:
The characteristic behavior from the bimodal model (s=400000 sat) can be seen here:

The graph displays estimated success probability under a bimodal prior distribution for different channel capacities. Curves for different amounts a are shown, where red color code means sending an amount of the full capacity limit, i.e., no channel is able to forward the amount (P=0), because capacities do not permit that. The other amounts are chosen with a log spacing, where the amount value can be read off the intersection of the curve with the x axis. We see that the bimodal model is independent of the capacity given the amount is larger than the channel capacity. For large amounts, we have a success probability of 1/2 as expected (the channel either forwards or not). For smaller amounts the success probability increases generally as more and more channels will have residual liquidity to forward due to the parameter "s" in the model.
The strategy in such an environment would be to decrease amounts to get to high success probabilities.
The equivalent plot for the uniform model is shown here:

The uniform model has the property that if one wants to send any amount, it is the best strategy to select the largest channel one can find to maximize success probability (statistically speaking).
For brevity I won't show how the posterior success probabilities behave, but I would like to contrast the two models:
Uniform:
Disadvantages:

tends to select large channels, thus may lead to congestion and centralization

Advantages:

can be used to calculate min-cost flows as negative log probabilities
is more "caring", as it leaves over some balance for the next payment, could have a balancing effect

Bimodal:
Disadvantages:

is more greedy: it  exhausts channels (but leaves back amounts on the scale s)  once it learns that liquidity is present (a user might act selfishly in the uniform model as well, so this is not a strong argument)
as a pure bimodal model, it is not compatible with min-cost flow computation as the negative log function is not convex

Advantages:

reflects current network conditions
is very generic and could be more flexible to describe changing network conditions: we can use the bimodal model to interpolate to the uniform model (by increasing s), reintroducing convexity
is in line with current LND pathfinding philosophy (constant a-priori probability, high probability after success)

Both models enable us to make smarter decisions based on prior events like successful/failed/in-flight payments.
I suggest to implement the bimodal-model based probabilities because:

it is more in line with the pathfinding system as currently implemented in LND, thus being a less disruptive change
it distributes HTLCs across different channel size ranges and doesn't focus on the "highways"
the bimodal model is tunable between a sharp bimodal and a uniform distribution therefore it is able to incorporate the uniform model's advantages and can be adapted to network changes in the future

Hope the comparison was fair and am happy to gather some feedback from you.

Interesting observations and analysis. I guess you could always question the conditions under which mission control was populated. Is it based on a representative set of payments, is this what the average user is also going to encounter? But that seems to be a question that's almost impossible to answer given the dynamic nature of the network and variation in positioning of senders and receivers.
Maybe an additional way to validate could be to ask node operators to share a snapshot of channel balances, possibly pre-aggregated into a histogram to minimize the disclosure of information. Or make it even simpler by offering a small tool that does this. For the goal of making payments more reliable, I'd think that there must be operators willing to do this?
Assuming the described distribution is indeed a good match, what parts of lnd's probability estimation would need to change? I suppose the apriori probability is the main one?
Besides that, there is probably going some interaction with the way we factor in failures and scores of other channels of the same node too?

Interesting observations and analysis. I guess you could always question the conditions under which mission control was populated. Is it based on a representative set of payments, is this what the average user is also going to encounter? But that seems to be a question that's almost impossible to answer given the dynamic nature of the network and variation in positioning of senders and receivers.

Thank you, right, I think it will give us a chance to set up something smarter, but it makes sense to do it in a conservative way.

Assuming the described distribution is indeed a good match, what parts of lnd's probability estimation would need to change? I suppose the apriori probability is the main one?

Yes, I see most changes around getPairProbability, possibly it interacts also a bit with the node probability.

In this thread, we've been talking quite a bit about channel capacity as an input for probability estimation. It's also the title of this issue. But what will happen if channel capacities are no longer available, for example because of a new on-chain-privacy-preserving gossip system?
In a world where a single max_htlc is the only advertised property of a connection between two nodes, is it even possible estimate an a priori probability that is more than a fixed value?

It may also be worth to think about what users would set max_htlc to if that's the only thing that can be communicated. Wouldn't the rational decision be to set it to the channel capacity, to not risk missing a forward that could be forwarded under ideal conditions?

People do set the max htlc to a smaller amount than capacity in order to communicate a sustainable amount that they can forward, many node operators don't want people to try routing through their channel and fail. Economically speaking this can be represented in not wanting a mission control penalty for a future route

Penalize temporary channel failures for channels with max_htlc harder (longer half life time). This is an incentive to set max_htlc to something that can be delivered.

Updating max_htlc all the time would create a lot of gossip.

If it doesn't scale, then the operator would set it to something that is smaller than what he can route, to accommodate multiple routing events. In this case capital is not used efficiently.

Maybe this is the cost of fast payment network? I believe there will be a point in the future where even a single failure is no longer tolerated and results in a hard ban. It is already difficult enough to compete on speed with a conventional card payment for just a single payment attempt.

I agree with the fees direction @bitromortac mentioned and agree that setting this number in a very fine tuned way is not scalable and also has negative privacy implications
For max_htlc I could also see it relating to the capacity question by setting the default at what is the high probability number for what can be routed in a very big HTLC that would want to use the channel. Like 99.9% of capacity = low chance of success, 80% of capacity = reasonable chance of success, or whatever the model shows. This would be essentially free from a gossip point of view

Is there a centralization effect on demanding excess capital for a route?

Yes but that is not a consequence of the features chosen in selfish routing (c.f. Selfish routing and the price of anarchy work by Tim roughgarden https://www.amazon.de/Selfish-Routing-Price-Anarchy-Press/dp/0262182432 ) but a fundamental property of the lightning network itself that liquidity is necessary to guarantee certain service level agreements / objectives.
People who wish to send money will be selfish in making routing decisions and routing nodes / liquidity providers have to change the channel graph in adoption to the selfish behavior of users. It's the same as street planning: It needs resources so that folks who chose the fastest / quickest route can all do so without congestion. Just look at braess paradox https://en.wikipedia.org/wiki/Braess's_paradox which is actually an answer (to your concern) that indicates that people don't want to open too large channels if we choose this feature
Anyway I am getting nerd snipes here

Dont you think there is not enough economic incentives for large masses to conduct routing and hence we might be facing this issue which could be temporary Anyways I am just a newbie learning kindly excuse if the question is not valid.

First of all kudos to @bitromortac for digging into challenging the uniform prior assumption. I unfortunately don't have enough time right now to address all of the points that you are raising (as promised out of band I will come back to this) but I think I can explain why we do not see a uniform distribution of liquidity as indicated on the mailinglist and in this blogarticle.
TL;DR: the distribution that we currently see mainly emerges from the flow of payments (who wants to pay whom) and the optimization critera that nodes chose in pathfding (e.g. optimizing for fees). I would be very careful to draw conclusions from any of those results. Because:

it does not take into account how the distribution changes if we adopt the optimation goals
it does not take into account that the actual selfish behavior may be against other nodes.

With respect to the 2nd point: As of right now - where literally everyone optimizes for fees - I can write down a very easy and probably excellent working path finding strategy which reads:

"Compute expected drains on channels only take edges against the direction of drain and optimize for fees".

I am very sure as of now this path finding strategy will work amazingly until a software implements it and people start to using it widely which would change the prior belief about the drain... (Long story short: I think we need to figure out the full game theory, dominant strategies and nash equilibrium instead of having ad-hoc solutions (like the ones that we and everyone else has been proposing so far)). Also the bigger concern will be how we can design the routing game so that the price of anarchy of the domant strategies will be low. I hinted to this at the liquidity ads proposal
@bitromortac wrote:

As a second step, I assumed a bimodal distribution following a liquidity distribution X of P(x) ~ exp(-x/s) + exp((x-c)/s) ("bimodal constant model"), where s is a liquidity broadening scale and c is the channel capacity. This distribution describes situations in which channel liquidity is typically found on the local or the remote side, but rather not in between (unbalanced channels). It seems like this model is able to better describe the observed data (there is a model variation "bimodal variable", which works even better, but I won't describe it here as it's similar).

This is actually the reason why I am giving such a short and quick reply: While I was trying to model the random process a bit more specifically in this notebook I realized that the bimodel assumption seems pretty weired. As soon as we have drain the liquidity will be on one end. Is there a way of modelling this with removing one mode and starting from a random drain which would put the liquidity for a specific channel either on one side or the other? So making P(x) ~ exp(-x/s) or P(x)~exp((x-c)s) where your choise of s probably depends on the strenght of the drain and the choice of the first or second deistribution on the direction. I would be very surprised if such a simpler (non bimodal model) could not predict the data from mission control.
Actually I think we could even go one step further: for each direction of the channel one could chose P(x) ~ exp(-x/s) which would make the information content - that I called uncertainty cost or negative log probabilies - (-log(P(x) = x/s) convex again.
Specifically it is very similar to the linearized uncertainty unit cost that I used which was defined as 1/c where c was the capacity of the channel. In your case without the necessity to linearize things I guessed above that 1/s would be related to the prior assumed drain of the channel. While the drain of the channel might not correlate with the size I can imagine the similarities of the outcome would explain why @C-Otto was succesfully sending 0.6 Bitcoin to my node using the probabilistic model based on channel capacities.


Pull joints should break based on distance

Being moved a large distance while pulling something should "yank it out of your hands" so to speak. As of right now, being teleported causes you to do your best Mr. Danny Phantastic impression, using your elastic ghost powers to pull you and the object you were dragging to a midpoint between whre you started and your intended location. Both portals and the QSI have* logic to handle this, but both come with client-side prediction issues, and ideally it should be handled by the joint to break itself if you're far enough away.
*At time of writing there is an open PR to add logic to the QSI for this. That's where this comment comes from.
          i feel like this is better served by making pulling logic more robust so it actually breaks joints when outside of a reasonable distance rather than just keeping it specifically here.

Originally posted by @EmoGarbage404 in https://github.com/space-wizards/space-station-14/pull/31177#discussion_r1721117023

The existing band-aid code to handle this situation that I've identified is Content.Shared/Teleportation/Systems/SharedPortalSystem.cs at line 95, and SwapTeleporterSystem.cs at line 189*
*once again, at time of writing, contingent on pr #31177. Prior to this getting pulled, using a QSI while dragging something will just give you a teleporter accident.


It returns the output 1 through 5 but doesnot pass the test

Challenge catch-off-by-one-errors-when-using-indexing has an issue.
User Agent is: Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.

function countToFive() {
  let firstFive = "12345";
  let len = firstFive.length;
  // Fix the line below
  for (let i = 0; i <= len-1; i++) {
  // Do not alter code below this line
    console.log(firstFive[i]);
  }
}

countToFive();

As the condition check says

Your code should set the terminal condition of the loop so it stops at the last index.

The last index should be the length itself.
for (let i = 0; i < len; i++)
should solve the issue

@rajibmp is correct, please check your solution once again.


[real-world uses] Add implode.io

That appears to be using ACE, so it's not much of a CodeMirror use example.


Cart "completedAt" check

I've noticed a "bug" in my use of gatsby-theme-shopify-manager related to the Cart "completedAt" check.
If a customer completes a checkout but still has the website open in a tab and goes back to the website, they will have a "completed" cart that they cannot get out of until they reload the website. This prevents the customer from adding new items to their cart because you cannot add items to a completed cart.
This happens because the only time the cart completedAt field is checked is when the app context is initiated (line 45 in ContextProvider.tsx).
We may want to add some functionality that checks if the cart is completed/valid before doing certain things like adding items to the cart.

Maybe you could redirect users to a custom thank-you page after they complete checkout, and have this page clear your cart.
1. Edit Shopify Checkout
In Shopify, go to Settings > Checkout, and in the Addition Scripts section, add your redirect URL. (credits to @mister-cairns for this.)

2. Create a thank-you page on Gatsby
Make a static gastby page (like src/pages/thanks.) This thank you page will clear the cart on mount.
// src/pages/thanks.tsx
import { useClearCartOnMount } from '../hooks/use-clear-cart-on-mount'

export default function ThankYouPage() {
  useClearCartOnMount()

  return
Thanks for shopping with us!

}

And then the React hook that makes the magic happen:
// src/hooks/use-clear-cart-on-mount.ts
import { useRemoveItemsFromCart, useCartItems } from 'gatsby-theme-shopify-manager'
import { useRef, useEffect } from 'react'

export const useClearCartOnMount = () => {
  const items = useCartItems()

  const removeItemsFromCart = useRemoveItemsFromCart()
  // this is a work-around, since the function isn't memoized
  const removeRef = useRef(useRemoveItemsFromCart)

  useEffect(() => {
    const cartItemVariantIds = items.map((variant) => variant.id)
    if (items.length > 0) {
      removeRef.current(cartItemVariantIds)
    }
  }, [items])
}

Since the functions aren't memoized, we'll have to use a useRef workaround. This work-around avoids infinite loops, ensuring we only clear the cart when the page mounts.
However, as seen by the [items] dependency in the useEffect function, we do run the effect again when the items array changes. We do this since the line items could be fetched asynchronously, and they might not be available on the first useEffect call.
Finally, if the cart has 0 items (i.e. items.length > 0 doesn't apply), nothing happens.
Edgecase

If a customer completes a checkout but still has the website open in a tab and goes back to the website, they will have a "completed" cart that they cannot get out of until they reload the website.

Interesting. My first thought is that you shouldn't open the checkout in a new window from your custom Gatsby cart. However, this edge case could still happen. You could maybe check if the checkout still exists when the window refocuses, and if not, clear the cart, using the method from above?


Add CLI capability to create a podsetting

User should be able to pass in a JSON with pertinent parameters(cpu / gpu count, image, etc) and we should create a podsetting and apply it for the user...returning the name of that podsetting

Weise Yin has provided feedback that something similar to 'orbit deploy profile' in CLI is needed and has suggested a json such as :
{
"display_name": "Small (GPU Enabled)",
"description": "2 CPU + 4G MEM + 1 GPU",
"image": "342862375695.dkr.ecr.us-west-2.amazonaws.com/orbit-dev-env-users-gpu-jupyter",
"resource_limit": {
"cpu": 2,
"mem": 2,
"gpu": 1
}
}


Outdated version of NodeJS?

When I ran the script it downloaded NodeJS V0.12.2. Is this version production ready. I followed the link http://nodejs.org/dist/ and it shows v5.x . I'm not really sure if the script is downloading an outdated version.

0.12 is still a preferred version by many.  4.x was a merge with another fork of the Node.js project.
You should be fine with 0.12.  It is the version I use.


Can it do graphs like the one I show ?

I would like to do graphs like those below which are from a '70s publication in which the authors specifically mention they used the Calcomp plotter. I've used such plotter in the '80s but for basic graphs, not fancy stuff like shown here. Would your package be able to do this ? Or, maybe any ideas on how to do this with modern graphics ?
Thanks, Peter

This appears to be a highly customized plotting style done at a relatively low level.  So I do not personally know of any plotting package that provides the style shown as a high-level routine. Therefore it looks like you need to use low-level graphics routines directly yourself to create that exact plot style.
Do you have the original Calcomp code?  If the Calcomp library is close to the one emulated by M_calcomp (unfortunately there were a LOT of variants) it might be easier to use M_calcomp than other graphics libraries.  If you are trying to emulate the style but from scratch (without the original code) I would suggest something like M_draw from the same github site or calling something like gnuplot(1).  It depends a lot on what Operating Systems you need to run on, how long you expect the code to be in service (which strongly weighs how self-contained you want the code to be) and enough other factors I cannot answer the question on what other package to use without a lot of further details.  So you can use M_calcomp, but if you do not have the original code there is not much reason to, and practically any graphics package is a candidate without knowing more.

On 12/09/2023 12:28 PM EST fortranboomer @.***> wrote:
I would like to do graphs like those below which are from a '70s publication in which the authors specifically mention they used the Calcomp plotter. I've used such plotter in the '80s but for basic graphs, not fancy stuff like shown here. Would your package be able to do this ? Or, maybe any ideas on how to do this with modern graphics ?
Thanks, Peter
Screen.Shot.2023-12-09.at.12.23.48.PM.png (view on web) https://github.com/urbanjost/M_calcomp/assets/65827952/c86e4ccc-8c3d-44bd-b710-51cb205f9732
—
Reply to this email directly, view it on GitHub https://github.com/urbanjost/M_calcomp/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHDWN3JKQE2DPPEBCLL25IDYISNU3AVCNFSM6AAAAABAN5YMQ2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGAZTGOJVGQ3TMMY.
You are receiving this because you are subscribed to this thread.Message ID: @.***>


Use Crystal's remove in macro

Saw a FIXME note on gio/resource.cr about rm. This PR calls Crystal's File#delete using crystal eval as a cross-platform alternative.
Not sure if using crystal eval like that has any downsides, but the spec passes (& the gresource binary gets deleted successfully)!

That looks very inefficient.
Why not just a macro, using del on windows and rm on unix?

You are right, I had future platforms in mind but I don't think gtk4.cr should target them anyway. Closing!


Networksettings

all tests passing for network member, 100% line and func coverage, 69% branch coverage. (needs token pool tests)

replacing this PR with a new one with updated code


Added ability to listen on all interfaces with afpacket.

As discussed in #104, exposing the port on which the packet came in is useful in certain situations. I have added the ability to extract this information from the packet header, with the index of the port now stored in the CaptureInfo metadata field of the packet. This index can later be translated to a name by invoking e.g. net.InterfaceByIndex().
Note: To perform the proper tpacket alignment calculations, I had to add a different alignment function "tpAlign" which seems to work, where the included "tpacketAlign" did not. I see that "tpacketAlign" is used for processing of the packets in the blocks in v3-headers, but I'm not sure how that is working if the alignment is incorrect. I tested using my "tpAlign" function in place of "tpacketAlign", which seemed to work fine, but I didn't want to make the change in this PR. Since "tpacketAlign" is only used one place, it would be an easy fix in the future.

I'm somewhat concerned about the addition of InterfaceIndex to CaptureInfo, specifically the fact that it's only currently set in afpacket.  pcap, for example, doesn't set it right now.  Are we sure an interface index is the correct way to export things?  What will be set, for example, when reading from a pcap file?  Is 0 a valid interface index, and if so, how do we differentiate between "not set" and "set, valid, and zero"

This is a good point. To think it though:

I am pretty sure that 0 (zero) is never a valid index value in Linux, given the behavior of many of the network functions such as if_nametoindex() which returns 0 with an error.
CaptureInfo.InterfaceIndex will (as of this writing) be 0 in every case where pcap or pfring is used. This could be left as-is for now, or we can try to add similar functionality for pcap and pfring.
To add this for pfring, we can add a reference to the pfring_extended_pkthdr which has a similar if_index field. Pfring is set to capture from all interfaces by setting the device string to "any".
Adding this for pcap may be difficult because there is no interface information in the pcap_pkthdr, in fact when you listen to all devices with pcap, you don't get real layer 2 headers at all. However, you are still able to listen in all devices by passing in "any" or NULL. Here an index of 0 may be appropriate for the "any" case.
We could try to add the functionality in cases where the device is set, so that it is not 0 when it clearly shouldn't be.

For pcap, do a reverse-lookup of the handle device name to the index and add to the handle struct
For pfring, do a reverse-lookup from device to the index and add the index to the Ring struct
In above cases just plug in this index to CaptureInfo.InterfaceIndex

For pcap files, there is by definition no device, so an index of 0 is appropriate. Of course it is possible/probable that the capture is from a real device, but given the information above this will have to be known a priori anyway.

I'm not sure which approach is the best here, but it seems like we at least should add the proper index for pfring captures, since the index is available. Adding the index number to the Ring/Handle is an option too, but may seem unnecessary since in the cases outlined above, the interface name is already known to the user.

I have added more code to enable pcap and pfring captures to add interface index to their packets' metadata.
The way to works now is:
- for pcap, the interface index is added when the handle is created. If the device does not exist, the index is set to 0, in case the device string is still accepted, which is the case for "any".
- for pfring, the interface index is added when the ring is created. If the flag pfring.FlagLongHeader is set, the ring will remember this and use whatever is in the extended packet header. If it is not set, we use the interface name passed in by the user. In the case of pfring, you are supposed to be able to pass in "any", and the if_index field of the extended header will show the correct interface where the packet came in, but I was unable to successfully capture packets by passing in the "any" device name. If this works on other machines or is fixed in the future, the added solution will just work as expected.
- For pcap files, the interfaceIndex will always be 0
- For afpacket, the interfaceIndex will always be correct.

Any feedback on the latest commit @gconnell ?

Sorry for the slow reply on this, slipped my mind.  Merged.


Vacuum is very slow

Hi,
I just started using delta lake and noticed that vacuum creates a single job per file in spark, does this make any sense?

Hey @lyogev Have you tried setting a lower value for park.sql.sources.parallelPartitionDiscovery.parallelism, it is 10000 by default in Apache Spark.

Yes, but it doesn't change the fact that I'm getting 200 job each vacuum (my output is 200 files each time)
Note that this is a non-partitioned table

@lyogev If you're dealing with smaller tables, you can definitely reduce that configuration. It doesn't launch 1 Spark job per file, it actually launches 1 Spark job with 200 partitions that's supposed to list your table directory. It pulls them to the Driver 1 by 1 to avoid OOMing the Driver, causing Vacuum to launch 200 Spark jobs.
There is room for optimization here. We cache the file listing. We can count the number of files, and then repartition the result according to a heuristic to avoid launching 200 small jobs. Is this something you would consider contributing? :)

Hi @brkyvz I would love to contibute.
However, right now we are heavily invested in hudi.
I have performed some tests on delta as the open source seemed mature enough to test it.
So i would love to share some insights, in the end if we decide to switch from hudi to delta I would definitely contribute.
So this is what I have right now :)
Pros for delta

Ingestion is much faster, 2X to 4X when using merge in delta vs upsert in hudi (copy on write).
System is very simple to use, much less configurations and API is clean.

Cons for delta

No hive metastore support, without this we can't use it
Slow vacuuming (the above issue)
Queries are slower with delta, simple select queries return in 2-3X time but in a count(1) query, it's 10X slower (using databricks to test this)
Missing the auto optimize and auto compaction feature in the open source version, these are included in hudi.

So basically the slow query performance + missing hive is what makes this unattractive for now, although I would love that ingestion speed!

There is room for optimization here. We cache the file listing. We can count the number of files, and then repartition the result according to a heuristic to avoid launching 200 small jobs. Is this something you would consider contributing? :)

@brkyvz If I understand the code correctly, it would be a matter of changing VacuumCommand.scala from this
        val diff = allFilesAndDirs
          .where('modificationTime < deleteBeforeTimestamp || 'isDir)
          .mapPartitions { fileStatusIterator => ...

to something like this
        val diffFiles = allFilesAndDirs
          .where('modificationTime < deleteBeforeTimestamp || 'isDir)
        val nrFiles = diffFiles.count
        val diff = diffFiles
           .repartition(repartitionHeuristic(nrFiles))
           .mapPartitions { fileStatusIterator => ...

        def repartitionHeuristic(nrFiles: Long): Long = {
           math.max(nrFiles / 1024L, 1L)  // 1024 files per partition
        }

Or would you rather envision some more sophisticated?

Hi @brkyvz I would love to contibute.
However, right now we are heavily invested in hudi.
I have performed some tests on delta as the open source seemed mature enough to test it.
So i would love to share some insights, in the end if we decide to switch from hudi to delta I would definitely contribute.
So this is what I have right now :)
Pros for delta

Ingestion is much faster, 2X to 4X when using merge in delta vs upsert in hudi (copy on write).
System is very simple to use, much less configurations and API is clean.

Cons for delta

No hive metastore support, without this we can't use it
Slow vacuuming (the above issue)
Queries are slower with delta, simple select queries return in 2-3X time but in a count(1) query, it's 10X slower (using databricks to test this)
Missing the auto optimize and auto compaction feature in the open source version, these are included in hudi.

So basically the slow query performance + missing hive is what makes this unattractive for now, although I would love that ingestion speed!

So when you said no hive metastore support - what do you mean exactly?

@prasadvaze I mean you can't register tables yet per #85

@lyogev
per comment by @ abiratsis in  https://github.com/delta-io/delta/issues/395 ,  looks like he was able to register delta table in hiveMetastore
val someDf = spark.emptyDataset[SomeSchema].toDF
// first create Delta table
someDf.write.format("delta").partitionBy("day").save(outputPath)
// then register it to hiveMetastore
spark.sql(s"CREATE TABLE ${tableName} USING DELTA LOCATION '${outputPath}'")
The only strange behaviour is the unexpected path of the Delta table as already mentioned here

@lyogev @prasadvaze MetaStore support should be coming in the next two weeks. We're working on getting Delta to compile with Spark 3.0 this week. Then we can release the support for Hive MetaStore tables.
@redsk I would do something very similar to that a bit later in the code (right before the delete).
I wouldn't cap max parallelism, that would lead to scaling issues. However, the simple heuristic of math.max(nrFiles / 1024L, 1L) seems good to me. We may want to make the 1024 configurable instead of hardcoded.

@brkyvz I think the bottleneck is deleting files one by one, I was able to parallelize it and deleting around 300K files in 10 minutes using >100GB,32CPUs
Kindly check my PR https://github.com/delta-io/delta/pull/529


Does openvino MKLDNN support 2 CPUs (ex: xeon 6148 x 2)

https://www.asus.com/us/Commercial-Servers-Workstations/WS-C621E-SAGE/He...
https://software.intel.com/en-us/articles/intel-and-facebook-collaborate...
If I use 2 xeon 6148 cpus in 2 socket  of one mainboard (asus WS C621E)
Question: 2 cpus are faster than 1 cpu?

Hi @fatalfeel,
I think this question is better to ask on OpenVINO forum.
But AFAIK OpenVINO has some optimizations for multi-socket systems.

I want to add that MKL-DNN itself does work fine on 2-socket systems and can schedule computations across two sockets. However, there may be cases when it is better to run two independent 'instances' one per socket, and it is OpenVINO that has to set such configuration up.

I hope next release can do that well~some mainboard can put 4 cpus~

Intel MKL-DNN and OpenVINO do support multi-socket systems. For the best inference results on multi-socket systems we typically recommend running several instances pinned to cores within socket.


Scanner takes too long to scan or sometimes fail on some devices in Flutter 3.19

Describe the bug
When i open the scanning, the camera just scans forever and does not pick QR code
To Reproduce
Steps to reproduce the behavior:

Invoke the scanner by calling await FlutterBarcodeScanner.scanBarcode('#ff6666', 'Cancel', true, ScanMode.BARCODE)
Try to scan a QR code

Expected behavior
Returns QR code content result
Screenshots
In some devices it does not return, just scans forever
Smartphone (please complete the following information):

Device: Samasung A series,
OS: Android 13
Browser [e.g. stock browser, safari]
Version 13

Additional context
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.19.4, on Kali GNU/Linux Rolling 6.5.0-kali3-amd64, locale C.UTF-8)
[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
[✓] Chrome - develop for the web
[✓] Linux toolchain - develop for Linux desktop
[✓] Android Studio (version 2023.1)
[✓] VS Code (version 1.85.1)
[✓] Connected device (3 available)
[✓] Network resources
• No issues found!

Any recommendations


SheenCloth sample model crashes gltf_viewer with assertion

Describe the bug
Debug builds of gltf_viewer crash when loading the SheenCloth.gltf sample model.
in void utils::panic(const char *, const char *, int, const char *):32
in file ../../libs/utils/src/debug.cpp
reason: ../../libs/gltfio/src/FilamentAsset.cpp:130: failed assertion `info.bindings.size() == 0 || info.flags == flags'

To Reproduce
Steps to reproduce the behavior:

Build gltf_viewer for debug
Download the SheenCloth glTF 2.0 sample model.
Load the SheenCloth.gltf model into gltf_viewer

Investigation
A comment above the assertion reads:
    // All bindings for a particular glTF texture must have the same transform function.

However, as seen with SheenCloth.gltf, this isn't always the case.
In the SheenCloth model, sheenColorMap (which wants a sRGB texture), and sheenRoughnessMap (which wants a linear texture) could both map to the same glTF texture.
In that case, one solution is to create two copies of the texture: one sRGB, and one linear. This, however, doubles the required texture memory.
Desktop (please complete the following information):

OS: macOS
Backend: All backends

Or maybe we could do the conversion in the shader for one of the input when they are mixed. With spec constants, this could even be "free" when both match.

Yeah, I was thinking of something along those lines could work too. I think this is a somewhat rare edge-case.

maybe in the meantime, we could do something to avoid the crash (and I guess get bad roughnessMap instead), and downgrade this bug to "correctness" from "crash"?

Yeah I'm fine with that


compare_config returns show instead of the config diff

Description of Issue/Question
Did you follow the steps from https://github.com/napalm-automation/napalm#faq

[X] Yes
[ ] No

Setup
napalm-panos version
(Paste verbatim output from pip freeze | grep napalm-panos between quotes below)
pip freeze | grep napalm-panos
napalm-panos==0.5.2

PanOS version
(Paste verbatim output from show system info between quotes below)
mikhaelc@fw-dr-02> show system info

hostname: fw-dr-02
ip-address: <IP_ADDRESS>
public-ip-address: unknown
netmask: <IP_ADDRESS>
default-gateway: <IP_ADDRESS>
ip-assignment: static
ipv6-address: unknown
ipv6-link-local-address: fe80::ea98:6dff:fe09:500/64
ipv6-default-gateway:
mac-address: e8:98:6d:09:05:00
time: Fri Nov  1 00:45:35 2019
uptime: 71 days, 0:36:01
family: 800
model: PA-850
serial:<PHONE_NUMBER>72
cloud-mode: non-cloud
sw-version: 8.1.9-h4
global-protect-client-package-version: 0.0.0
app-version: 8205-5740
app-release-date: 2019/10/30 21:30:31 PDT
av-version: 3148-3659
av-release-date: 2019/10/31 04:00:56 PDT
threat-version: 8205-5740
threat-release-date: 2019/10/30 21:30:31 PDT
wf-private-version: 0
wf-private-release-date: unknown
url-db: paloaltonetworks
wildfire-version:<PHONE_NUMBER>38
wildfire-release-date: 2019/11/01 00:40:47 PDT
url-filtering-version: 0<IP_ADDRESS>
global-protect-datafile-version: unknown
global-protect-datafile-release-date: unknown
global-protect-clientless-vpn-version: 0
global-protect-clientless-vpn-release-date:
logdb-version: 8.1.8
platform-family: 800
vpn-disable-mode: off
multi-vsys: off
operational-mode: normal

Steps to Reproduce the Issue
Error Traceback
(Paste the complete traceback of the exception between quotes below)
There is no traceback,
It looks like
# if BACKSPACE_CHAR in first_line: from _first_line_handler does not catch specific reprint
If I changed it to if True: it works, but Im not sure why it was set to BACKSPACE_CHAR

This would seem to be something in netmiko, giving the time elapsed and likelihood it is not an issue in this library, going to close. If you feel that there is still an issue in this library, feel free to open back up.


stm32h7x3: spi DXPIE/TXPIE enables are in fact read-write

These two bits may be written by software, although they may also be
cleared by hardware. The RM says "TXPIE is set by software and cleared
by TXTF flag set event."
The bitfield in the RM describes these bits as rs, rather than
rw. Perhaps why they are mis-classified as read-only in the SVD.

Agreed


#keepTileSprite on tile doesn't work?

I have an example project demonstrating the bug, using a LDtk template edited to add a ladder tile entity.
The problem, to be specific, is that the importer isn't finding the Sprite and setting its texture.
testproject.zip
(I was debugging the error myself, because apparently that's possible with Godot addons, which is why there's an extra print statement in your code.)
UPDATE: It seems to work only if I make the Sprite the scene root.

testproject.zip

I will try to fix it during the day. Thank you for all the reports

The bug is that Godot does not save the change of the child sprite to the scene file.
At first I thought it was easy to fix this by setting the Owner field of the sprite node, but that ends up making a copy rather than changing the original sprite.
Godot has "Editable children" checkbox in the context menu, but there's no way of accessing it in code. I found mentions of the set_editable_instance method on the forum, but it does not seem to be in the current version of the engine. It sounds like a perfect solution.
Honestly i don't know how to fix it right now. Bad I didn't test it first :/
There are #tileId and #tileSrc fields for tile entities, it's possible to write a code that assigns correct sprite region. #keepTileSprite was intended as a shortcut

Yeah, I'm good with using #tileSrc as a quick fix. I had a feeling this didn't have an easy solution.


MNT: rerender

Hi! This is the friendly automated conda-forge-webservice.
I've rerendered the recipe as instructed in #49.
Here's a checklist to do before merging.

[ ] Bump the build number if needed.

Fixes #49

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.


opentofu/terraform resources for secure bootable images on Azure

What would you like to be added:
We need changes to the upstream terraform provider for Azure to enable secure bootable images that let us bring our own secure boot config and keys.
Why is this needed:
I identified potential blockers to go forward with the openTofu implementation of our secure boot use cases.

security_profile.uefi_settings needs to be passed to the shared image version
https://learn.microsoft.com/en-us/python/api/azure-mgmt-compute/azure.mgmt.compute.v2023_07_03.models.galleryimageversionuefisettings?view=azure-python
https://learn.microsoft.com/en-us/azure/virtual-machines/trusted-launch-secure-boot-custom-uefi
Terraform azure shared_image_version resource

The terrafom shared_image_version resource cannot define uefi_settings at the moment.

This can be implemented upstream.

Upstream Issue opened here: https://github.com/hashicorp/terraform-provider-azurerm/issues/28064

Upstream PR has been created: https://github.com/hashicorp/terraform-provider-azurerm/pull/28076


Syntactic Sugar: Would it be possible to allow app.run() instead of aiohttp.web.run_app(app)?

Long story short

Would it be possible to create some syntactic sugar to allow the use of the simple app.run() syntax instead of the boilerplate of aiohttp.web.run_app(app)? Clearly this is not a vital feature but I see a lot of elegance in the aiohttp project and this might add to that.
Expected behaviour

Actual behaviour

Steps to reproduce

Your environment

No. Application should not create a socket, it's another level of abstraction.
Utility function run_app is a right place for non-core helpers.


Adds support for use of enum types as literals and nativeEnums

Why
Fixes #39
Warnings ⚠

~Relies on https://github.com/fabien0102/ts-to-zod/pull/38~ Rolled these together b/c #38 was a subset of this PR
Does not handle imports, assumes enum is exported & imported already

Codecov Report

Merging #40 (814a5ce) into main (e5e4961) will decrease coverage by 0.39%.
The diff coverage is 98.00%.

@@            Coverage Diff             @@
##             main      #40      +/-   ##
==========================================
- Coverage   98.49%   98.10%   -0.40%
==========================================
  Files           8        9       +1
  Lines         332      369      +37
  Branches      102      113      +11
==========================================
+ Hits          327      362      +35
- Misses          5        6       +1
- Partials        0        1       +1

Impacted Files
Coverage Δ

src/utils/resolveDefaultProperties.ts
95.23% <95.23%> (ø)

src/core/generate.ts
100.00% <100.00%> (ø)

src/core/generateZodSchema.ts
97.41% <100.00%> (+0.21%)
:arrow_up:

src/core/jsDocTags.ts
98.46% <100.00%> (-1.54%)
:arrow_down:

src/core/validateGeneratedTypes.ts
97.43% <100.00%> (+0.06%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d6ae0a1...814a5ce. Read the comment docs.

I think I addressed all the feedback, thanks again for taking a look @fabien0102 🙇

I wonder whether the following is a bug:
export enum LoggerLevel { debug, info, warning, error, critical }
interface ILogger {
  /**
   * @default LoggerLevel.info
   */
  level: LoggerLevel
}

compiles down to
export const loggerLevelSchema = z.nativeEnum(LoggerLevel);
const iLoggerSchema = z.object({
  level: loggerLevelSchema.default("LoggerLevel.info"), //// <----- notice enum default is quoted
});

TIA


Want navbar design like this

@Anu27n Please assign me this issue , this will be done within 1 hr , i would love to work on this .

@Anu27n Sir i have made the updates in Navbar and raised a PR and attached the ScreenShots , Please review it .

@Anu27n Sir please do not merge the chnages from branchAnimation for the navbar issue
and do so from this "Branch navbar updates " because this is a separate PR
Thank You


Compilation is failing because of std::collections::range::RangeArgument

error[E0432]: unresolved import `std::collections::range`
  --> /Users/yannsimon/.cargo/registry/src/github.com-1ecc6299db9ec823/tantivy-0.5.2/src/query/range_query.rs:11:23
   |
11 | use std::collections::range::RangeArgument;
   |                       ^^^^^ Could not find `range` in `collections`

Which version of rust are you using?
tantivy 0.5.2 requires a fairly recent version of nightly rust.
If everything goes ok, 0.6 will be released this week and it will work with the imminent stable Rust 1.27

I used the last nightly version.

Oh right! The range package apparently got deprecated recently.
I am afraid you will have to wait for 0.6.0 later this week.

OK thanks for quick feedback!!

@yanns. Tantivy 0.6.0 is released.

Thx it's working. ❤


Error when loading Lottie Asset, and precacheImage on initState

So, i'm building my own wedding app, the issue i'm having is this:
The login page has an image as background,  a lottie animation (GIF), and a widget that calls a Rive animation.
The first time the login page loads , it calls an error in the framework:

but i did not called those two methods mentioned in the error
My loginPage is this:
class LoginPage extends StatefulWidget {
  const LoginPage({Key? key}) : super(key: key);

  @override
  State<LoginPage> createState() => _LoginPageState();
}

class _LoginPageState extends State<LoginPage> {
  bool isTextOn = false;
  late final LottieComposition composition;
  final GlobalStore globalStore = Modular.get<GlobalStore>();
  final AuthStore authStore = Modular.get<AuthStore>();

  Future<void> _loadResources() async {
    final imageSize = MediaQuery.of(context).size;

    // await precacheImage(
    //   const AssetImage('assets/images/bgNew3.jpg'),
    //   context,
    // );

    AssetLottie('assets/lottie_animations/noivoscinza.json')
        .load()
        .then((value) => composition = value);
    // You can preload other resources like animations here
  }

  @override
  void initState() {
    super.initState();
    _loadResources();
  }

  @override
  Widget build(BuildContext context) {
    globalStore.setBuildContext(context);
    return Observer(builder: (context) {
      return Scaffold(
        body: SingleChildScrollView(
          child: Stack(
            alignment: Alignment.topCenter,
            children: [
              FutureBuilder(
                  future: _loadResources(),
                  builder: (context, snapshot) {
                    if (snapshot.connectionState == ConnectionState.waiting) {
                      return const Center(
                        child: CircularProgressIndicator(
                          color: Colors.black,
                        ),
                      );
                    } else {
                      return Container()...

I know that if i deactivate the Uncaught Exceptions the application won't stop in debug. THe problem is that when the page is load, there's a noticeable delay in both lottie and rive animations.
How do i solve this?

Hi @gabriel-R-Lemes
The goal of this Github repo is not to give assistance on personal code or the whole package/plugin project, but to improve flutter reporting bugs or feature requests.
If you encounter behavior that you believe could be a Flutter framework bug, please try to provide all information indicated in the template so that we can address this issue properly (See bug issue template here).
Instead, if the issue only appears with Lottie, please open a new issue on its repository first. Once its developers verify that the issue comes from Flutter framework, please comeback here with more information that will help us verify this issue.
Thank you!


[DOCS] Sphinx docs warning fixes

This issue happens when the docker update status for ci-gpu to the lastest(the main change is is mkl-ml), and it also triggers a latest sphinx update.
Currently log for the sphinx pre-check warnings  due to the version upgrade http://ci.tvm.ai:8080/job/temp-ci-docker-staging/job/ci-stage/29/execution/node/205/log/ need to fix them before the we upgrade the image.
Actions to be taken

[x] Migrate the markdown to rst

cc @tmoreau89 for docs/vta/install.md
cc @kazum for docs/deploy/aws_fpga.md, aocl.md

[x] fix the remaining warnings

All the docs change are  landed by now


💡 Feature: automatic, tree-shaken resolution

An idea I had for automatically scanning entry files in a @pika/web project, resolving modules, and adding them to the @pika/web dependency chain. This re-uses a lot of ideas & code from an ESM bundler I was building in 2018 and never finished.
Only uses @babel/parser and raw Node.js to do its job.
Usage
pika-web --entry src/index.js

This will start at src/index.js, auto-detect all its imports, scan those, and if they’re local files it’ll scan those imports, and so on, until it reaches the end (similar to webpack). When it’s done, it’ll try and match any imports to dependencies in package.json, and pass those to @pika/web.
In this way, you don’t need to add anything to @pika/web.webDependencies—as your project changes, --entry will only bundle what you use (and if you stop importing it, it’ll no longer bundle it). But if desired, you could still manually add modules to webDependencies to bundle no matter whether it’s used or not.

Example: even though webDependencies is empty, @pika/web was still able to resolve all these imports within the entry file
Features

Tree-shaking: thanks to Babel it only grabs used imports
Dynamic imports: it’ll grab those, too
Caches (kinda): it won’t scan the same files over and over again. If it’s already hit a file before, it’ll skip it (thanks to the dependency tree it builds).

TODO

Tests. Screenshots are from a working project, but this code will be hard to amend without tests.
Optimization. Though it doesn’t add noticeable time to @pika/web, it’s still a first cut that could probably be optimized. Questions should be asked like: Could this be a Rollup plugin for easier use? Should the parser be something other than Babel?

Yes, so excited for this! Giving it a review now...

Can you tell me more about why someone would want a dependency-tree.json file output? i'm not sure I'm following what that is used for.

Yes! If you’re familiar with the webpack bundle analyzer (I know you are IDK why I asked that), that all comes from a stats JSON file.
This tree graph is built anyway to keep track of what it’s scanned, and a dependency list for each file (telling foo/index.js apart from bar/index.js, etc.). One way or the other this tree will get built (i.e. I didn’t add extra code for this), but I thought it’d be a neat opportunity to provide app analysis in the future by exposing the thing it’s already doing.
So while, yes, you probably couldn’t build a pika/web bundle analyzer from this just yet, something like this would be the base for it. And again, it was basically “free” based on the nature of what’s happening.

@dangodev thanks for all your help PRs and in the pika.dev community over the last year. I just wanted to publicly recognize that you are now a contributor on the repo 👍👍👍

@dangodev this looks close! Going to carve out some time to do some final testing on my machine tomorrow. Also, if you haven't started yet, I'll get some basic integration tests going in the repo as well.

Gonna review this this weekend and hopefully finish this!
I’m glad we punted on the dep logic. I found this today: https://github.com/pahen/madge. Would definitely be useful In the future for the dependency traversing and bundle analysis (I’m still stuck on the idea of a bundle analyzer for Pika Web to show you where your weight is)

Looks good, and tested working on www.pika.dev codebase! It even found a few dependencies that I was no longer using 👍
For reference, I still need to hardcode two types of imports in my webDependencies:

some that are inside of a

For reference, I still need to hardcode two types of imports in my webDependencies:
some that are inside of a

Yea, handling script tags inside of HTML shouldn't be too hard, and could probably plug into our existing glob->read->scan flow pretty easily (since we don't need to represent the script tags as files, we just need their code).
I had two back to back flights yesterday and was able to build a first draft of auto-treeshaking. The plugin itself was straightforward (thanks to the rollup team in https://github.com/rollup/rollup/issues/3047) but I had to change our representation of a "dependency" from just a string to include some metadata as well.
I'll push the branch up later today, would love your feedback on it.

Also a few related housekeeping things:

The libraries been around for a few months now, and these two features together feel big enough to warrant a v1.0 release cut.
I think this library would see more adoption under its own name/branding. I'd like to do that as a part of the 1.0 release.
A v1.0 release means we need a full documentation site. It can be simple, but I think we've grown out of the README being enough anymore, especially for the getting started case.


The border of the Detail Pane is partially lost when content in the pane is long

Describe the bug
The border of the Detail Pane is partially lost when content in the pane is long.
How to reproduce
https://stackblitz.com/edit/clarity-v3-light-theme-dzkhzm?file=src%2Fapp%2Fapp.component.html
Steps to reproduce the behavior:

Add datagrid
Add detail pane with content longer than the table rows height.

Expected behavior
The border should covert the whole height of the detail pane.
Versions
App
Cloud Director

Angular: 9.1
Node: v12.16.1
Clarity: 3.1.3

Device:

Type: MacBook
OS: Catalina
Browser Chrome, FireFox

Additional notes

Seems that .datagrid-table-wrapper does not go to 100% height. Have to check if it's intentional and I think it can be easily fixed.