Php running out of allocated memory

After 12 executions of Curl->get(...), I run out of allocated memory. The line that throws the error is Curl.php line 731 ($json_obj = json_decode($response, false);)
    /**
     * Set Default JSON Decoder
     *
     * @access public
     */
    public function setDefaultJsonDecoder()
    {
        $this->jsonDecoder = function($response) {
            $json_obj = json_decode($response, false);
            if (!($json_obj === null)) {
                $response = $json_obj;
            }
            return $response;
        };
    }

I can increase the memory_limit in php.ini to greater than 128M, but regardless of how large I make it this error occurs prior to reaching 100,000 records.
I have to curl for a few 100,000 records (~400,000), but have to do so in chunks of 1,000. Basically have the following:
$fp = fopen($path_to_file, "w");
while(True) {
   $curl = new Curl();
   $curl->setBasicAuthentication($username, $password);
   $curl->get($url);

   //check httpStatusCode for rate limiting purposes
   //possibly sleep for some amount of time then continue with the next iteration of this loop

   $response = $curl->response;
   foreach($response->a as $a) {
       // keep only the attributes I want
       fputcsv($fp, array('id' => $a->id, 'name' => $a->name));
   }

   //get url for next page of results
   if (property_exists($response, 'next-page') {
       $url = $response->{'next-page'};
   } else {
       break;
   }
}
fclose($fp);

I have tried to unset($curl); after I finish with it after each iteration of the while(True)..., but this has no effect.
I'm suspicious that there are some variables within this Curl class that are not getting cleaned up after each reoccurring call.
Worth noting, Php is not my primary language. I'm used to python handling garbage collection for me. Basically, I'm kind of lost when it comes to debugging this project.

Are you using PHP7?
It may be worth calling $curl->close(), as even though the object is overwritten, it is not destructed (see #310).
If you are using PHP 7, then you may be running into a known memory leak.  Try close() first.

I'm using Php Version 5.5.31.
I'll try using $curl->close()

$curl->close(); solved the issue! Thanks @thebeline !
Correct code below:
$fp = fopen($path_to_file, "w");
while(True) {
   $curl = new Curl();
   $curl->setBasicAuthentication($username, $password);
   $curl->get($url);

   //check httpStatusCode for rate limiting purposes
   //possibly sleep for some amount of time then continue with the next iteration of this loop

   $response = $curl->response;
   foreach($response->a as $a) {
       // keep only the attributes I want
       fputcsv($fp, array('id' => $a->id, 'name' => $a->name));
   }

   //get url for next page of results
   if (property_exists($response, 'next-page') {
       $url = $response->{'next-page'};
   } else {
       break;
   }
   $curl->close();
}
fclose($fp);


Role bans don't appear in adminremarks

Haven't personally verified it, but I think this is the 2nd time I've heard of it. Iirc, both role bans and game bans can have a hidden flag set, unless that flag is set they should be visible in adminremarks imo (assuming the cvar is enabled to let players see their notes)

#20388 should have fixed this- can you confirm this is still the case?

Wait nevermind this is something else- my bad


Track uncaught errors thrown in the bridge transaction flow

errors that are thrown out of the bridge.prepareTransaction or bridge.getTransactionStatus (not those in status.errors, but actually the THROWN errors) are kinda critical, and at the moment only displayed to the user but not sent yet to sentry errors = it means we don't see them at all. This add this feature.
❓ Context

Impacted projects: ledger-live-common and ledger-live-mobile
Linked resource(s): in order to see errors like TECSUPPORT-1245 on sentry

✅ Checklist

[x] Test coverage: for the first time, we have test coverage on useBridgeTransaction
[x] Atomic delivery
[x] No breaking changes: no impact on LLD because not used yet. on LLM we hook errors to logger.critical

🚀 Expectations to reach
Please make sure you follow these Important Steps.
Pull Requests must pass the CI and be internally validated in order to be merged.

Yeah the context was also my idea but like you said, it feels overkill.


Add sound to the game

Motivation
closes #280
What was changed/added
Sound was added as requested in the High Court Meeting, summed up in #280. All the sounds, and the implementation in bridges has been done by @koennixer, credits to him.
Testing
You can check everything for yourself, and we recommend you do so to check everything for yourself, especially because volume might be drastically different.
https://github.com/mango-gremlin/arch-enemies/assets/104799849/86434726-c035-4072-8426-e54ff2d5b9b4
https://github.com/mango-gremlin/arch-enemies/assets/104799849/3a1438ad-4e7a-4506-aefd-8c749b8fa815
Additional Information
!Important! Right now this isn't finished, overworld is missing dialogue sounds for the specific animals, which I will sort out tomorrow probably. Do not merge this before everything is included, and before everything is discussed.
Bugs found

Walking sounds depend on the global variable walking, which we should refactor anyways, but works for now. However, this causes the walking sound to repeat endlessly in some cases, even though no movement occurs, such as when initating dialogue

Discussion
Please review all the sounds, as well as the volume, as everything is highly subjective.
@koennixer mentioned that sound options might be something we could implement in the future, as it shouldn't be too hard (but also associated with new assets, testing etc.)

Alright. Let's go from top to bottom:
"Wouldn't say its my taste but thats not the topic"
This is not workable advice. If you don't like something tell me what you dislike and what it should rather be. If you don't like 8-bit as sound effect what would you rather have? 16-bit? Realistic? Cartoonish? If you don't like Chiptune what would you rather have? Orchestrated? 8-Bit?
"why is this commented ? (change)"
I don't know why that was commented. It is now uncommented.
"well we don't need to start this topic again, but maybe, just maybe try to avoid such code in future..."
This is not useful. You can be direct about your dislike for things as much as you want, but this is not constructive. What do you want instead? How do you want it done? Fundamentally if you don't want a node for each sound you have to load a new stream into the one you are using. That stream has to be preloaded, ideally this should happen in a Sound Node itself, because that is were it belongs not in the Grid or Drag. Given that obviously other branches of the code will have to tell the sound node to switch the stream and then play it. This can be done two ways, either you will have to, in every single node that is supposed to play sound, load that Node on ready, which I refuse to believe is an efficient way to go about it, or you signal the Node. If you signal the Node you will obviously need to inform it what sound it is supposed to be playing. Ideally this would be done with a Global Enum or something else, but given that the data we are transferring in drag is Strings this seems to be a workable option. Given all that you have given me as critique:

call the function directly, which sure is less lines of code, but I'm less sure about it being more efficient.

Avoid using the match, which is completely unworkable as a critique, because you don't state what you'd rather we use. If it's about the String then state that. If you'd rather we use something that isn't a match then elaborate. If you want us to completely refactor the sound into something else than elaborate.

This does not read as a constructive critique but rather as smugness.
"I guess this was for testing ?"
Yes. And as long as the code is not done I'm gonna leave it as that.
"I would note that one should use type annotation."
Marcel changed it.
"Timer"
Not in regards to a specific quote but the timer has also been changed. Marcel has implemented Evelynns suggestion and I made the timer be loaded on ready, avoid loading it with every delta. These changes are all specific to the grid scene. Marcel can deciede to apply them to the overworld.
"for the other bridge levels, we can just copy the nodes over, and they should work?"
Signal might have to be reapplied, but other than that it should be identical.

"for the other bridge levels, we can just copy the nodes over, and they should work?"
Signal might have to be reapplied, but other than that it should be identical.

-> how exactly, and where do we need to reapply the signal?
Or rather, since you two are familiar with it (and have my hands full constructing the final overworld, dialogue etc) - maybe when this is approved and merged, just merge main into aylin's and my branch (feature/final_overworld_and_bridges) and then you can add the sounds to the bridge levels? overworld should work seamlessly, as I just edited the main overworld for the final overworld.

Not in regards to a specific quote but the timer has also been changed. Marcel has implemented @ScatteredDrifter suggestion and I made the timer be loaded on ready, avoid loading it with every delta. These changes are all specific to the grid scene. Marcel can deciede to apply them to the overworld.

Just fyi, both overworld and bridge walking functions the same, as the player is same for both scenes, so this is refactored for both now.

"well we don't need to start this topic again, but maybe, just maybe try to avoid such code in future..."
This is not useful. You can be direct about your dislike for things as much as you want, but this is not constructive. What do you want instead? How do you want it done? Fundamentally if you don't want a node for each sound you have to load a new stream into the one you are using. That stream has to be preloaded, ideally this should happen in a Sound Node itself, because that is were it belongs not in the Grid or Drag. Given that obviously other branches of the code will have to tell the sound node to switch the stream and then play it. This can be done two ways, either you will have to, in every single node that is supposed to play sound, load that Node on ready, which I refuse to believe is an efficient way to go about it, or you signal the Node. If you signal the Node you will obviously need to inform it what sound it is supposed to be playing. Ideally this would be done with a Global Enum or something else, but given that the data we are transferring in drag is Strings this seems to be a workable option. Given all that you have given me as critique:
1. call the function directly, which sure is less lines of code, but I'm less sure about it being more efficient.

2. Avoid using the match, which is completely unworkable as a critique, because you don't state what you'd rather we use. If it's about the String then state that. If you'd rather we use something that isn't a match then elaborate. If you want us to completely refactor the sound into something else than elaborate.

This does not read as a constructive critique but rather as smugness.

Ok so right now it seems that we know via the signal connection which sound gets played by the function getting called and by its parameter. If we remove the multiple on[...]_play sound methods we need some sort of checking which sound to play, I would suggest a Dictonary but a match is also a solution in this case.
But why do we even bother if we already know in the on[...]_play sound which sound to play ?
In e.g. this code in bridge_sound.gd should output the same result, at least for what I tested:
extends AudioStreamPlayer2D
var squirrel = preload("res://assets/sound/Squirrel.wav")
var death = preload("res://assets/sound/Death.wav")
var drop = preload("res://assets/sound/Drop.wav")
var deer = preload("res://assets/sound/Deer.wav")
var jump = preload("res://assets/sound/Jump.wav")
var spider = preload("res://assets/sound/Spider.wav")
var step = preload("res://assets/sound/Step.wav")
var snake = preload("res://assets/sound/Snake.wav")

func _on_drag_grid_play_sound(sound):
    stream = drop
    playing = true

func _on_deer_item_play_sound(sound):
    stream = deer
    playing = true

func _on_snake_item_play_sound(sound):
    stream = snake
    playing = true

func _on_spider_item_play_sound(sound):
    stream = spider
    playing = true

func _on_squirrel_item_play_sound(sound):
    stream = squirrel
    playing = true

func _on_player_play_sound(sound):
    stream = jump
    playing = true

func _on_grid_play_sound(sound):
    stream = death
    playing = true

And sorry for not providing a alternative instantly in this case, this was not thought for any change, just for the feedback on the general code layout.

as @Raining-Cloud 's suggestion is refactoring -> do @marcel-iv or @koennixer want to do this right now, or can we merge this, so I can see whether there are any problems with the sound in the new bridge levels?


⚠ Young Discoverers has degraded performance

In c3dc101, Young Discoverers (https://youngdiscoverers.org.au) experienced degraded performance:

HTTP code: 200
Response time: 5795 ms

Resolved: Young Discoverers performance has improved in 82284f7.


Why "Console.ReadLine()" works when typing "clear" and then typing "dotnet run" but doesn't work when typing "clear | dotnet run"?

Issue Description
"Console.ReadLine()" works when typing "clear" and then typing "dotnet run" but doesn't work when typing "clear | dotnet run"
Steps to Reproduce
The steps are in the attached video
Expected Behavior
Actual Behavior
Logs
OmniSharp log
Post the output from Output-->OmniSharp log here
C# log
Post the output from Output-->C# here
Environment information
VSCode version: 1.69.2
C# Extension: 1.25.0
Dotnet Information
SDK do .NET (refletindo qualquer global.json):
 Version:   6.0.301
 Commit:    43f9b18481
Ambiente de runtime:
OS Name:     Windows
OS Version:  10.0.22622
OS Platform: Windows
RID:         win10-x64
Base Path:   C:\Program Files\dotnet\sdk\6.0.301\
Host (useful for support):
Version: 6.0.6
Commit:  7cca709db2
.NET SDKs installed:
6.0.301 [C:\Program Files\dotnet\sdk]
.NET runtimes installed:
Microsoft.AspNetCore.App 3.1.26 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 3.1.26 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.26 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
To install additional .NET runtimes or SDKs:
https://aka.ms/dotnet-download

Visual Studio Code Extensions

Extension
Author
Version

activitybar-separator
ctf0
0.0.1

csharp
ms-dotnettools
1.25.0

customize-toolbar
liangruliu
0.1.2

errorlens
usernamehw
3.5.2

jupyter
ms-toolsai
2022.6.1201981810

jupyter-keymap
ms-toolsai
1.0.0

jupyter-renderers
ms-toolsai
1.0.8

MagicPython
magicstack
1.1.0

msbuild-project-tools
tintoy
0.4.4

new-moon-vscode
taniarascia
1.8.8

python
ms-python
2022.10.1

rest-client
humao
0.25.0

terminal
formulahendry
0.0.10

vs-code-semicolon-insertion
chrisvltn
0.0.6

vscode-autohide
sirmspencer
1.0.7

vscode-pylance
ms-python
2022.7.30

vscodeintellicode
VisualStudioExptTeam
1.2.22

https://user-images.githubusercontent.com/77227512/179741356-9ecb826b-426e-40bb-9014-001bb11748d6.mp4

Since this is not an issue with the C# extension itself, you question would be better answered by the dotnet/runtime repo. Follow this link to open an issue https://github.com/dotnet/runtime/issues/new/choose
Looking at your video though I can take a stab at helping.

The reported error is that you cannot invoke Console.ReadKey when input has been redirected (See docs). Using pipes (such as clear | dotnet run) to launch programs in sequence does redirect the output from one to the input of the next. This causes the error that you are seeing. You can check whether the applications input has been redirected by testing the Console.IsInputRedirected property.


[Feature Request] Made the admin intro menu translatable and added to crowdin

When I wanted to translate the menu, I found that many words were not found in crowdin, forming a mixture of two languages.

Hope to make the menu translatable.

Just pushed the new strings to Crowdin, so these should be fully translatable.


Remove build method from Spec

We should remove the Spec::build method; constructing the SQL should be the responsibility of the DbType class, since it knows which database syntax to use.
https://github.com/AugustNagro/magnum/pull/61#issuecomment-2472439673

Closed by https://github.com/AugustNagro/magnum/pull/73


Default $orderby _id:1 on where() causing HUGE performance issues

On Mongoid 4 / Moped 2 stable, I am seeing crazy performance issues when $orderby: { _id: 1 } is used on a where statement that is doing an $or and looking for a single record. With $orderby its 40 seconds, without it, the same query goes to 1ms.
I checked all of my code and I am not setting a default scope to order by _id anywhere.

Running into the same issue. The problem is pretty simple: mongoid adds $orderBy: { _id: 1 } when I call #first and do not specify order myself. When we do somthing like `MyModel.first

Running into the same issue. The problem is pretty simple: mongoid adds $orderBy: { _id: 1 } when I call #first and do not specify order myself. When we do somthing like MyModel.first that's OK (probably), but when I do some query that uses some index other than _id_1 that's not OK anymore.
To outline issue, imagine we have a pretty stupid model:
class Sample
  include Mongoid::Document
  field :idx, :type => Fixnum
  index :idx => 1
end

Now, querying :idx => { :gt => 1_000_000 } without explicitly specified sort will execute following command:
{ $query: { idx: { $gt: 1000000 } }, $orderby: { _id: 1 } }

On a fairly large collection this leads into a pretty slow query. Here's some numbers:
# Let's pollute collection with some stuff this will take some time ;))
1_000_000.times.to_a.shuffle.each { |i| Sample.create :idx => i }

# Selector from the above
selector = { :idx => { :$gte => 1_000_000 } }

# Now let's try `#first` without explicit `sort`:
puts Benchmark.measure { Sample.where(selector).first }
# 0.010000   0.000000   0.010000 (  1.230106)

Now the same with explicit `sort`
puts Benchmark.measure { Sample.where(selector).sort(:idx => 1).first }
# 0.000000   0.000000   0.000000 (  0.278825)

This is monkey-patch I use for Mongoid 3.x:
require "mongoid"

module Mongoid
  module Contextual
    class Mongo
      def with_sorting
        if !criteria.options.key?(:sort) && criteria.selector.empty?
          query.sort(:_id => 1)
        end

        yield
      ensure
        apply_option(:sort)
      end

      def with_inverse_sorting
        if criteria.options.key?(:sort)
          query.sort(Hash[criteria.options[:sort].map { |k, v| [k, -1 * v] }])
        elsif criteria.selector.empty?
          query.sort(:_id => -1)
        end

        yield
      ensure
        apply_option(:sort)
      end
    end
  end
end

We should remove the orderby from first and not guarantee the returned order.

@durran :bow:


updated

Issue addresed
Enter the issue number of the bug(s) that this PR fixes
main.go:
Added a new route /search to handle task search requests.
Introduced a new handler function searchHandler, which:
Retrieves tasks from the database.
Filters tasks based on the search query passed in the URL (q parameter).
Displays only tasks that match the search term (case-insensitive search).
Updated existing functions (indexHandler, addHandler, and deleteHandler) to ensure tasks are rendered correctly.
index.html:
Added a search bar allowing users to type in a query to search for specific tasks.
The search bar sends a request to the /search endpoint and displays the results dynamically using HTMX.

Template should be strictly Closes <issue_number>
Example: Closes #1

closes #4


Code suggestion: limit recreation scenarios of bigip vm

I am suggesting these changes to the module to stop redeployment of the VM because of 'known after apply' behavior in terraform.
We observed frequent destructive changes to the infra when rerunning identical code. We suspect this being due to certain values not being predictable by terraform forcing recreation of the VM.
Values that were marked as 'known after apply' include resource_group_name and location, both of which were derived from the data source azurerm_resource_group.bigiprg. This data source seemed to serve no purpose other than provide access to these mostly static values.
Furthermore admin_password and custom_data also have similar behavior because they are 'sensitive values'. To prevent these properties from causing a redeployment of the VM I added these as lifecycle properties with ignore_changes
I do see some merit to having custom_data changes trigger a redeploy as the data in this block is not as static however it might be better to force a user to consciously remove the resource via terraform destroy or deletion of the VM; an update in place is not supported either way and there are more downsides to the unneeded redeployment that these fixes remediate than there are upsides to having modification to custom_data trigger an automated redeploy of the entire VM.

Consider also hostname could cause a recreation. Seems that this module is based for only creating F5 instances but not manage lifecycle. Pheraps it is better to accept a map where you insert hostname and type of machine, because sometimes you want only to upgrade size of VM. Consider also extensions ("custom data") to manage with lifecycle because them are managed by DO or Cloud-Init.


apply for Best Practices badge

Describe what problem your feature request solves:
the OpenSSF run a Best Practices Badge Program, it is very likely that Threat Dragon follows these best practices
Describe the solution you'd like:
Inspect the criteria for gaining the badge and apply
Additional context:
Consider the OWASP Project Committee Good Practice
Consider linking directly to OWASP from the initial home page
Consider applying for OWASP Production status

The best practices project has been created and partially complete for Threat Dragon, standing at 94% as of August 2024


NotebookProfileViewer - ability to choose regions in plots in Histograms

Summary
For numeric features, it is not uncommon to have uncleaned outliers that distort the double_histogram plot. If that happens, plot becomes useless, as there is no way to control the region of the plot to be viewed, or to predefined a range to be visualized when calling double_histogram
What is the expected correct behavior?
User should either:

Be able to define a range to view the histograms, like: visualization.double_histogram(feature_names="total sulfur dioxide", range=[0,100])
Interactively select the plot's region to zoom in in the UI

Relevant logs and/or screenshots

hmm either of options could be done. I'd lean more towards the putting in the range. What do you think, @FelipeAdachi ?

I think adding a range parameter would mean less effort, so I vote for that. But it doesn't need to be named range, maybe there's a better name for it?

This is really a question on do we want to have it keep the same bins but remove the outliers from that range, or recalculate the bins after excluding those outside the range.
To me, I'd prefer that latter as it seems more correct as the outliers would change the shape of the bins. @bernease, @jamie256  and @FelipeAdachi  Thoughts?


Change the name of the Add-in

Please make a change to the add-in description and name

@topowright What will the new name and description be?

@kgonzago This is not a real issue. This is an issue we will be creating for DevSummit. This is to share to the community how we work.


Comments: VR46 M1 Illustration

Hi, if you have any question or suggestion please leave your comment.

test

this is a test comment

2nd comment

test comment


testing string methods after game has been won

Current coverage is 88.46% (diff: 100%)

Merging #53 into master will increase coverage by 2.44%

@@             master        #53   diff @@
==========================================
  Files             8          8
  Lines           286        286
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits            246        253     +7
+ Misses           40         33     -7
  Partials          0          0

Powered by Codecov. Last update 5c7d01f...969b830


feat: Add IAM role output

Description
Adds an output for IAM role arn and updates the README.md to reflect this new output.
Motivation and Context

This allows you to reference the role used by the module.  Since the module might create a new role you don't know what the new role's ARN is unless you look it up in some other way.  This makes it easy to reference.
Breaking Changes

No, it shouldn't break anything.
How Has This Been Tested?

[x] I have updated at least one of the examples/* to demonstrate and validate my change(s)
[x] I have tested and validated these changes using one or more of the provided examples/* projects

[x] I have executed pre-commit run -a on my pull request

thanks - we should have exported this from the beginning. Can we copy this https://github.com/terraform-aws-modules/terraform-aws-ecs/blob/f2befd85b08c588ab2df4b515af187d2bb70c9e2/modules/cluster/outputs.tf#L57C1-L70C2 and use it here (update to match the correct name) to ensure the standard properties for the IAM role are exported

thanks - we should have exported this from the beginning. Can we copy this https://github.com/terraform-aws-modules/terraform-aws-ecs/blob/f2befd85b08c588ab2df4b515af187d2bb70c9e2/modules/cluster/outputs.tf#L57C1-L70C2 and use it here (update to match the correct name) to ensure the standard properties for the IAM role are exported

I have updated the PR to include role name and unique_id to match what is being done in ECS

This PR is included in version 3.1.0 :tada:


Segmentation fault with ns-process-images

I am getting a Segmentation fault (core dumped) towards the end of the ns-process-images using hloc.
To Reproduce
ns-process-data images --data images/ --output-dir with_hloc --sfm-tool hloc --refine-pixsfm --feature-type any --matcher-type any --use-sfm-depth --include-depth-debug --verbose --num-downscales 1

Below the last lines before the error:
Bundle adjustment report
------------------------
    Residuals : 78064
   Parameters : 28484
   Iterations : 6
         Time : 0.457851 [s]
 Initial cost : 0.851016 [px]
   Final cost : 0.850298 [px]
  Termination : Convergence

  => Completed observations: 8
  => Merged observations: 8
  => Filtered observations: 5
  => Changed observations: 0.000538
  => Filtered images: 0

Elapsed time: 8.471 [minutes]
[2023/05/10 18:32:16 hloc INFO] Reconstructed 1 model(s).
[2023/05/10 18:32:16 hloc INFO] Largest model is #0 with 100 images.
[2023/05/10 18:32:16 hloc INFO] Reconstruction statistics:
Reconstruction:
        num_reg_images = 100
        num_cameras = 1
        num_points3D = 9294
        num_observations = 39035
        mean_track_length = 4.20002
        mean_observations_per_image = 390.35
        mean_reprojection_error = 1.31182
        num_input_images = 100
[2023/05/10 18:32:16 pixsfm INFO] Extracting references and costmaps.
Segmentation fault (core dumped)

Ubuntu 20.04
nerfstudio 0.2.2
100 images, around 40Mb png each (7920x6004)
Thanks!

I dont know my answer will solve your problem. But, I think you can do this:

Try the command with less custom tags. For example, just run with the basic one:
ns-process-data images --data images/ --output-dir with_hloc

Try another dataset which has smaller size and use same syntax as you already use

I think the problem caused by OOM CUDA or OOM RAM.

Thanks for your reply. I ended up doing the pixel-perferct-sfm separately, and it worked fine. Actually, really good for images this size, with a reprojection error of just 1.1 pixels.


Remove deprecated warnings from console

What this PR does / why we need it:
This PR changes the default versions of CRDs, MutatingWebhookConfigurations, and Ingresses. Versions that are below the current one will be bumped in an octant view. For example, all ingresses are extensions/v1beta1 when shown in Octant.
Which issue(s) this PR fixes

Fixes #1596

Let's officially adopt this policy: https://kubernetes.io/docs/setup/release/version-skew-policy/#kubectl

Let's officially adopt this policy: https://kubernetes.io/docs/setup/release/version-skew-policy/#kubectl


Payload data empty when getting push notification

Hello there!,
I am sending push notification payload data to GCM endpoint and I am getting notification to corresponding browser but my payload data is empty in service workers JS.
Any ideas are appreciated, Thank you!

I have the same problem...

I have the same problem...

same problem, also in firefox (it returns {isTrusted: true})

i send such data:
Object {statusType: "chatMsg", name: "Bob", msg: "test", endpoint: "https://android.googleapis.com/gcm/send/endpoint", key: "key", auth: "auth key"}
server code
$notifications = array(
array(
'endpoint' => $_POST['endpoint'],
'payload' => '{data: "'.$_POST['msg'].'"}',
'userPublicKey' => $_POST['key'] ,
'userAuthToken' => $_POST['key_auth'],
),
);
$apiKeys = array(
'GCM' => 'GCM_API_KEY',
);
$webPush = new Minishlink\WebPush\WebPush($apiKeys);
foreach ($notifications as $notification) {
$webPush->sendNotification(
$notification['endpoint'],
$notification['payload'],
$notification['userPublicKey'],
$notification['userAuthToken']
);
}
this code returns TRUE, and i receive "push response", but data is empty
chrome: {}
firefox: {isTrusted: true}

After many research(as much as I can do due to time), I ended up sending notification from server(PHP) and fetch payload from service worker JS. It is works well for me.
Scenario:

Send push notification to specific client(firefox, chrome and safari) using PHP.
Our service worker JS triggered when new notification comes to browser.
Send request to server again for getting payload data with unique tag/id.
Show dynamic payload to client browser.

Since firefox, safari doesn't support payload fully, so I have updated my code like fetching payload data from service workers JS.
I have reviewed all major website service workers, they are all done in same way. You can check it out most popular website's service workers for getting clear idea about fetching payload and showing into client browsers.
Thank you!

Since firefox, safari doesn't support payload fully, so I have updated my code like fetching payload data from service workers JS.

Firefox does support push payloads (it's actually the first that implemented it).
Safari is not using the Web Push standard at all, it's a different, Apple proprietary, protocol.
Your solution is common (there's an example here: https://serviceworke.rs/push-get-payload.html), but it's heavier for the server.

Yes, you are right, what I meant was payload in web push notification are in devel stage.
I am open to accept any solution which send payload from server(PHP) and to service workers end.

Yes, you are right, what I meant was payload in web push notification are in devel stage.

I wouldn't say push payload is in the development stage, they're supported by the only two browsers that support Web Push (Firefox and Chrome).
There must be some problem in this library or in the way you're using it.
Are you sure the parameters you're using (userPublicKey and userAuthToken) are correct?

Yes, as recommended I have used the following set,
'userPublicKey' => 'some-unique-value'; // base 64 encoded, should be 88 chars
'userAuthToken' => 'some-unique-value' // base 64 encoded, should be 24 chars
Is there any specific format of value do we need to send with these? but these are optional values, so I have tried with 'null' value too.

Sorry for jump-in,
This library can send payload data successfully (I'm already test it on FF),

Is there any specific format of value do we need to send with these? but these are optional values, so I have tried with 'null' value too.

You doesn't need pass public token and auth key if you just want notify the subscriber there is update on your server, and need both (public token and auth key) if you want send payload data.
Can you share bit of your code on ServiceWorker.js? and how do you send the payload?

Yes I need to send payload from server to service worker, so I need to send public token and auth key.
Can you please give us an idea about how do you generate public token and auth key?
My server code look like @amadeyIllusix 's example.
My service worker looks like,
`self.addEventListener('push', function(e) {
console.log(e); //which contains 'data' and it is empty
var data = {};
if (e.data) {
data = e.data.json();
}
var title = data.title || 'Title';
var message = data.message || 'Message.';
var icon = 'icon-192x192.png';
var notification = new Notification(title, {
body: message,
icon: icon,
tag: 'push'
});
return notification;
});`

                                                                                When your subscriber successfully registered the push notification service.  It will return promise that have endpoint, public/browser token (p256dh) and auth key.                                                                                I believe you already get the endpoint. So you can get the other 2.                                                                                From: Anand SelvaduraiSent: Thursday, July 14, 2016 19:19To: Minishlink/web-pushReply To: Minishlink/web-pushCc: Febryan Y. Andika; CommentSubject: Re: [Minishlink/web-push] Payload data empty when getting push notification (#20)Yes I need to send payload from server to service worker, so I need to send public token and auth key.

Can you please give us an idea about how do you generate public token and auth key?
My server code look like @amadeyIllusix 's example.
My service worker looks like,
`self.addEventListener('push', function(e) {
console.log(e); //which contains 'data' and it is empty
var data = {};
if (e.data) {
data = e.data.json();
}
var title = data.title || 'Title';
var message = data.message || 'Message.';
var icon = 'icon-192x192.png';
var notification = new Notification(title, {
body: message,
icon: icon,
tag: 'push'
});
return notification;
});`
—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or mute the thread.

When your subscriber successfully registered the push notification service.  It will return promise that have endpoint, public/browser token (p256dh) and auth key. I believe you already get the endpoint. So you can get the other 2.

Yes, we can do it in js file, when initialize the worker:
//    ..... some code
navigator.serviceWorker.ready.then(function(reg) {
        reg.pushManager.getSubscription()
            .then(function(subscription) {
                var endpoint = subscription.endpoint;
                var key = subscription.getKey('p256dh');
                var key_auth = subscription.getKey('auth');
//.......

Then I send them on the server side, and put it into php array.
I found the condition in the webPush class:
WebPush.php#L194
then i checked all the variables
$endpoint
string(188) "https://gcm-http.googleapis.com/gcm/VERY_LONG_KEY"
$headers
array(7) {
  ["Content-Length"]=>
  int(4096)
  ["Content-Type"]=>
  string(24) "application/octet-stream"
  ["Content-Encoding"]=>
  string(6) "aesgcm"
  ["Encryption"]=>
  string(44) "keyid="p256dh";salt="random_salt""
  ["Crypto-Key"]=>
  string(107) "keyid="p256dh";dh="generated_key""
  ["TTL"]=>
  int(2419200)
  ["Authorization"]=>
  string(43) "key=GOOGLE_API_KEY"
}
// encoded payload
$content
string(4096) "nh8������-�|�fg݈�QU^$����,A���lX��ӱ6� ...."

this line WebPush.php#L223 returned TRUE, and notification was send.
But worker.js receive {} empty.
Mayby I do something wrong?
worker.js
self.addEventListener('push', function(event) {
    console.log('push');
    console.log(JSON.stringify(event)); // {} or {isTrusted: true}
});

self.addEventListener('push', function(event) {
console.log('push');
console.log(JSON.stringify(event)); // {} or {isTrusted: true}
});

Maybe this reference to this issue#46? or can you try using node.js version for sending with payload? this library is inspired by node.js version.
Unfortunately, I'm still working on FF version. Can't help you further on test this library. I'm back using node.js version because there is problem on dependency when using this library on my projects.

My problem is that the gcm server doesn't send me the subscription keys.
After  the subscription:
navigator.serviceWorker.ready.then(function(reg) {
        reg.pushManager.getSubscription()
            .then(function(subscription) {
                var endpoint = subscription.endpoint;
                var key = subscription.getKey('p256dh');
                var key_auth = subscription.getKey('auth');

I get the endpoint but  the key and key_auth ArrayBuffers are empty.... any idea why this happens ?

                                                                                This is happened because google already not support that chrome version, same happen on mozilla.Can you try using newer version?                                                                                From: arreumbSent: Wednesday, July 27, 2016 17:20To: Minishlink/web-pushReply To: Minishlink/web-pushCc: Febryan Y. Andika; CommentSubject: Re: [Minishlink/web-push] Payload data empty when getting push notification (#20)My problem is that the gcm server doesn't send me the subscription keys.

After  the subscription:
navigator.serviceWorker.ready.then(function(reg) {
reg.pushManager.getSubscription()
.then(function(subscription) {
var endpoint = subscription.endpoint;
var key = subscription.getKey('p256dh');
var key_auth = subscription.getKey('auth');
I get the endpoint but  the key and key_auth ArrayBuffers are empty.... any idea why this happens ?
—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or mute the thread.

Thanks a lot for your help.
Actually this is my chrome version: Versione 52.0.2743.82 (64-bit)
The code I have used is a bit different from that I have posted:
console.log('Notification Service Worker');
    if ('serviceWorker' in navigator) {
        navigator.serviceWorker.register('manager-service-worker.js', {scope: './'}).then(function () {
            return navigator.serviceWorker.ready;
        }).then(function (reg) {
            console.log('Richiesta endpoint:');
            reg.pushManager.subscribe({userVisibleOnly: true}).then(function (sub) {
                console.log('auth:', sub.getKey('auth'));
                 console.log('p256dh:', sub.getKey('p256dh'));
                console.log('endpoint:', sub.endpoint);

this is the output:

It seems you guys have problems with the client-side implementation of Web Push. Please look at examples here. May I close this?

Yes, you are right, it was my mistake, my problem is solved now.
Thanks.

Il giorno 29 lug 2016, alle ore 00:19, Louis Lagrange<EMAIL_ADDRESS>ha scritto:
It seems you guys have problems with the client-side implementation of Web Push. Please look at examples here https://github.com/Minishlink/web-push/blob/master/README.md#client-side-implementation-of-web-push. May I close this?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/Minishlink/web-push/issues/20#issuecomment-236042377, or mute the thread https://github.com/notifications/unsubscribe-auth/ACevL0u86u3McVNIdyqQHHH7BOkDYfuNks5qaSrggaJpZM4I9tNn.

@arreumb can you please give us some idea about how you resolved you problem? Did you updated chrome version?
Thanks

console.log('auth:', sub.getKey('auth'));
console.log('p256dh:', sub.getKey('p256dh'));
console.log('endpoint:', sub.endpoint);

My console for reference, can anyone help to get p256dh and auth key values?

I made a mistake: the ArrayBuffers in chrome debugger seemed empty but after converting them to string I saw they actually contain a value.
navigator.serviceWorker.register('manager-service-worker.js', {scope: './'}).then(function () {
           return navigator.serviceWorker.ready;
       }).then(function (reg) {
           console.log('Richiesta endpoint:');
           reg.pushManager.subscribe({userVisibleOnly: true}).then(function (sub) {
               var rawKey = sub.getKey ? sub.getKey('p256dh') : '';
               var key = rawKey ? Base64EncodeUrl(btoa(String.fromCharCode.apply(null, new Uint8Array(rawKey)))) : '';
               var rawAuthSecret = sub.getKey ? sub.getKey('auth') : '';
               var authSecret = rawAuthSecret ? Base64EncodeUrl(btoa(String.fromCharCode.apply(null, new Uint8Array(rawAuthSecret)))) : '';
               console.log("Key: " + key);
               console.log("AuthSecret: " + authSecret);
               console.log('endpoint:', sub.endpoint);

function Base64EncodeUrl(str){
    return str.replace(/\+/g, '-').replace(/\//g, '_').replace(/\=+$/, '');
}

function Base64DecodeUrl(str){
    str = (str + '===').slice(0, str.length + (str.length % 4));
    return str.replace(/-/g, '+').replace(/_/g, '/');
}

Now I managed successfully to send a payload from the server in java using this library: https://github.com/MartijnDwars/web-push

@arreumb Thank you! so much, it worked.
I can able to get those data from client and send push notification with payload using this PHP library. :thumbsup:

I have the same problem on service-worker side I get user token and key,
but if I send it with web-push PHP Api I have to str_replace the  keys - all good so far:
'userPublicKey' => str_replace(['_', '-'], ['/', '+'], 'BB0hBntOwrG-E9wqZZ5ZhZDru4bCubBd3iqiQMaDTNP714w96nraw-SX-c--aRTTPEcXXXXX38WWXRSqU='), // base 64 encoded, should be 88 chars         'userAuthToken' => str_replace(['_', '-'], ['/', '+'], 'Dn4XXXXXXXXXR2bw=='), // base 64 encoded, should be 24 chars
but on push-Listener in service-worker I do not get the push Data (on Chrome 54)
self.addEventListener('push', function(event) {     console.log('Push message', event);     console.log('push data', event.data.json()); });
JS-Error: unexpected token r in JSON at position 1

@arreumb  can you help me out

unable to get values by subscription.getKey('p256dh')  and subscription.getKey('auth') in chrome
i am able to get these values in an array bysubscription.toJSON(); in firefox

@Minishlink

while using minishlink library for php and sending message i get an error:
Fatal error: Class 'Minishlink\WebPush\Encryption' not found in on line 40

i am using Ubuntu16.04 as localhost with php7

@anandselva but i do not have the notification id to get its content from the server because i can not send any data in the notification data. so how do you get the notification's content by its id


Breaks with ZF 2.4?

I cannot seem to get this to work with ZF 2.4. I can go back to ZF 2.3 ok. It seems all my programmatically generated SQL statements broke.

@iskylut i cannot get your version to work with 2.4 Can you?

I got it to run with the ZF 2.4 Skeleton app, which errors are you getting?


Call out number inside code block replaced by not glyph

Hello,
I'm struggling to render conum inside coderay code blocks. I read almost every glyph issues I could find in this repo but I'm still missing something

As you can see conum are ok outside code blocks but not inside.
Using
<asciidoctorj.pdf.version>1.5.0-alpha.16</asciidoctorj.pdf.version>
<asciidoctorj.version>1.5.6</asciidoctorj.version>
<jruby.version><IP_ADDRESS></jruby.version>

And my theme with arial ttf extracted from a Win10 install:
font:
  catalog:
    Arial:
      normal: arial.ttf
      bold: arialbd.ttf
      italic: ariali.ttf
      bold_italic: arialbi.ttf
    # Noto Serif supports Latin, Latin-1 Supplement, Latin Extended-A, Greek, Cyrillic, Vietnamese & an assortment of symbols
    Noto Serif:
      normal: notoserif-regular-subset.ttf
      ...skipped...
    # M+ 1mn supports ASCII and the circled numbers used for conums
    M+ 1mn:
      normal: mplus1mn-regular-ascii-conums.ttf
      ...skipped...
    # M+ 1p supports Latin, Latin-1 Supplement, Latin Extended, Greek, Cyrillic, Vietnamese, Japanese & an assortment of symbols
    # It also provides arrows for ->, <-, => and <= replacements in case these glyphs are missing from font
    M+ 1p Fallback:
      normal: mplus1p-regular-fallback.ttf
     ...skipped...
  fallbacks:
    - M+ 1mn
    - M+ 1p Fallback

conum:
  font_family: M+ 1mn
literal:
  font_family: M+ 1mn
base:
  font_family: Arial

Thanks for your help

Ok I managed to fix this by using M+1mn font for code
code:
  font_family: M+ 1mn

Indeed, the M+ 1mn font has been prepared to provide the number glyphs. If you use your own font, you have to add these glyphs to the font so that Prawn can find them. The alternative is to use a fallback font that provides them.

Well as you can see in my first message, I added the font as fallback but it was not used until I explicitly use m+1mn for conum and code.

Yes, but the fallbacks you specified don't provide these characters. The font that has these characters is mplus1mn-regular-ascii-conums.ttf. You either have to use the font directly that provides these characters, or the fallback font you specify has to provide the characters. One way or anther, the glyph has to be there. And you actually have to check the fallback font that the characters are there.

Based on your screenshot, it seems like you were using Courier for the code. That's an AFM font. AFM fonts don't fallback to the fallback font (for reasons I will never understand, but that's just Prawn behavior). So you have to use a TTF font if you want the fallback to kick in. See https://github.com/asciidoctor/asciidoctor-pdf/blob/master/docs/theming-guide.adoc#fallback-fonts

Ok now I understand. I was not specifying any code font and I see that default code font is Courrier.
So this is weird to choose a default font that does not fallback. Or is it only when you specify a fontdir ?
I also was thinking that conum font would be used even inside code block. Maybe the documentation is not clear enough on that.
Thanks for your explainations and time :)

The default font choices are all AFM because that's what PDF provides by default. We can't assume there are any other fonts to choose from unless you specify them. This is just something you have to know when making a custom theme. I think I've made this pretty clear if you read the documentation about fonts (https://github.com/asciidoctor/asciidoctor-pdf/blob/master/docs/theming-guide.adoc#fonts). But if you can make it more clear, feel free to submit a revision.

Yes the explainations about AFM fonts is clear, but it does not seem so clear that the default font settings are not providing a full experience.
As soon as you start customizing fonts you need to customize ALL font settings.
Providing a base and a fallback like in the given example is not enough.

For a ease of use, is it possible that all Courrier references are changed to use $literal_font_familly by default?
This way we only need to redefine base, literal and fallback fonts to start a theme.

As soon as you start customizing fonts you need to customize ALL font settings.

Correct. We could add that statement to the docs.

For a ease of use, is it possible that all Courrier references are changed to use $literal_font_familly by default?

If you create your own theme, you have 100% control over the references. If you start with the provided default theme, this is already the case. See https://github.com/asciidoctor/asciidoctor-pdf/blob/f7337023eb9aa60671c85528f5a95c9b80513715/data/themes/default-theme.yml#L165

Yes, I saw that. But this is not the case if you start from a blank theme : you need to define every font familly (either by hand or variables ref)
In my case, this was confusing me : I was expecting that settings that I was not (re)defining were correct by default.
I agree this is a personnal biais and I don't expect you to change the way it works only for my personnal preference. I'm just giving my POV as a new user.
Maybe I'm missing something but can we configure our theme to inherit from default theme?

You're describing how the theme is designed. I think the docs are clear about this (aside from the revision we discussed). So I really don't see what would be changed anyway. You have to fill in all the values when creating a custom theme. That's the contract. The values are exactly correct by default, they just aren't what you want. So you're trying to paint a contract that isn't there.

Maybe I'm missing something but can we configure our theme to inherit from default theme?

Not yet, that's still an open issue. I agree that would make things simpler. See #367.

I suppose I could leave the code_font_family unset in the base theme and fallback to literal_font_family when resolved at runtime. The theme values are resolved up front, so the variable is already resolved by the time you set literal_font_family in your own theme. However, we could just defer until runtime. Please file an issue if you'd like that proposal to be implemented.

This is what I was suggesting. But I supose it would be less important when #367 is done.

The values are exactly correct by default, they just aren't what you want.

Now, after our discussion and after re-reading the doc admonitions again, I see what you mean by "it's in the doc".
I suggest that the AFM font part in the introduction would be in a separated Warning clearly stating that some glyph like callout number WILL be missing unless we explicitly define a code/callout font that supports them.
Let me see if I can come up with a doc PR that clears things.

Thanks for updating the docs.
I decided to wire the code and conum font family to the literal font family by default if you don't specify it in your theme. This reduces the number of fonts you have to set down to 2 (base and literal). That should eliminate some of the confusion when creating a new theme.

Perfect ! Thanks for your support.


Consider use and construction of content IDs for standards and other content

Standards currently have unique IDs of construction SEGAS-xxxxx
We may want all content to have permanent and unique IDs
We may want to use a different prefix (as SEGAS is not publicly understood)
We may want to use different prefixes for different types of content
Whichever construction and usage is selected, we will want some mechanism to allow users to know which content ID they should use for their new content so that they are permanent and unique.

Also need to ensure that content IDs are correctly displayed on content pages

We may wish to consider a unit test that verifies no duplicate ID has been used across documents.

Closing issue as not needed. Standard content IDs are working as is and other content doesn't require them. Issue is vague so any future issues can be raised to directly meet a use case


include preamble files generated by mylatexformat package

@Konfekt any reference material to help us understand why we should be ignoring these file extensions?

Hello,
How about
ftp://ftp.tex.ac.uk/ctan%3A/macros/latex/contrib/mylatexformat/mylatexformat.pdf
It is a file compiled from your TeX file preamble and as such reproducable from your source; around 10Mb big over here where the  source code weighs 10 Kb.

Hello,
Anything more detailed missing for this pull request to be reconsidered ?

Thank you!


Update Set-VMProcessor.md

Removed the additional "," at the end of -ExposeVirtualizationExtensions description per https://github.com/MicrosoftDocs/windows-powershell-docs/issues/3241
PR Summary

PR Checklist

[x] Descriptive Title: This PR's title is a synopsis of the changes it proposes.
[x] Summary: This PR's summary describes the scope and intent of the change.
[x] Contributor's Guide: I have read the contributors guide.
[x] Style: This PR adheres to the style guide.

Looks like it was fixed here https://github.com/MicrosoftDocs/windows-powershell-docs/commit/9de8e0b4e54e011c4d042f15c880d0801aef5188


Upgrade reporting cluster to ES 8

Tracking this here so I don't forget - this should be a simple change but the automatic rolling upgrade failed, so Elastic support are working on it

Still in correspondence with Elastic, progress is being made


Add links to devpost

Added links to devpost submissions page for HackCU III (2017) and HackCU 2016 in the footer.

Looks great! Unfortunately I'm not able to Review / Merge this from the mobile website. I'll merge this when I can get on a computer if it's still open.

Oh. Duh. Requests Desktop Version on phone -_-


Collections Dev Guide Update

Description
Updates the dev guide with the new role paths and filters changed with the collections work
Fixes # (issue)
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Just a docs update, needs a read thru by reviewer
Checklist:

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] Any dependent changes have been merged and published in downstream modules
[ ] Any variable changes have been validated to be backwards compatible

@erikgb good point! I need to add these dev steps somewhere
mkdir -p ansible_collections/confluent/platform
git clone https://github.com/confluentinc/cp-ansible.git ansible_collections/confluent/platform

without this molecule wont work!


Module version is not treeshakable and has side-effects on import

if you import either contentWindow or iframeResize from the 'iframe-resizer' module, for example like so
import { iframeResize } from 'iframe-resizer';

…you'd expect the reuslting browser bundle to only contain the iframeResize function.
What happens, however, is that both the host script and the contentWindow script get bundled (and executed!).
This result is completely un-obvious, until you open and carefully inspect the output bundle (which most people never do), or if you inspect the window of the webpage running the bundled code and notice that it has both the client and host scripts running at the same time.
This happens beacuse both modules (src/iframeResizer.js, and src/iframeResizer.contentWindow.js) contain an IIFE with immediate side-effects (creating global variables, adding window event hanlers, etc.) and thus the code can't be safely excluded from the output bundle.
...
There exist a few ways to resolve this.
I guess the simplest would be to stop exporting both modules from index.js, and instead tell people they must import from 'iframe-resizer/js/iframeResizer.js' and from 'iframe-resizer/js/iframeResizer.contentWindow.js'.
An even better solution would be to refactor the source files to export an init function, and then create separate output files for browsers and node.js import usage.
The browser files would then immediately invoke their respective init() function, whereas the node.js module would simply export the init functions and have developers decide when and if to run them.

BTW, my current workaround is to import the host function like this
import iframeResizer from 'iframe-resizer/js/iframeResizer.js';

which solves the problem, I guess, but only as long as the script path doesn't change.
A formal documentation of this would be nice.

Hi @SamMousa  trying to implement your approach but I have a few questions.

Do you still placing a script in the external page containing the frame as specified in the docs (https://github.com/davidjbradshaw/iframe-resizer#usage)

How are you using the <iframe/> element on svelte component?

Yes, external page is a classic php monolith that renders html in which we include the script.
The svelte app is deployed separately so it has a URL

For version 5 I am breaking the library up into separate modules. If your interested the draft docs are now online at https://iframe-resizer.com and their is an early alpha release on NPM. Would be interested in any feedback you have.

Fixed in V5.0.0
Please see https://iframe-resizer.com


Craft 5

nay plans to bring this to 5

@wmdhosting no plans of right now, will re-open if that changes so you get an update.


[BUG] EKS Service Role(s) not fully populated with all values stored in AWS Console in RancherUI dropdown

Rancher Server Setup

Rancher version: v2.7-ea6b86c79434dfc020c61227528f51ceaf4339b6-head
Installation option (Docker install/Helm Chart): Docker
Proxy/Cert Details: self-signed

Information about the Cluster

Kubernetes version: 1.23 or 1.24
Cluster Type (Local/Downstream):
Downstream EKS hosted cluster

User Information

What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom)
admin

Describe the bug
EKS Service Role(s) not fully populated with all values stored in AWS Console in RancherUI dropdown.  Only grabs first alphabetical grouping of results. [A-D, in my case]
To Reproduce

Fresh install of rancher v2.7-head
Create a cluster, and select EKS
Navigate to Cluster Options > Service Role > Custom
Reproduced:

not fully populated with all values stored in AWS Console in RancherUI dropdown
Only grabs first alphabetical grouping of results

Result

Not fully populated with all values stored in AWS Console in RancherUI dropdown
Only grabs first alphabetical grouping of results

Expected Result
All compatible values stored in AWS Console should populate in dropdown and be selectable
Additional context
Also seen w/ Node Instance Role dropdown; https://github.com/rancher/dashboard/issues/7755

Ticket #7756 - Test Results - ✅
With Docker on a single-node instance:
Verified on rancher v2.7-e54432ebcfba5cfdf92cf055b89a7ebff2537ef7-head:

Fresh install of rancher v2.7-head
Begin to create a downstream EKS cluster
Observe Service Role dropdown
Verified - All applicable roles from AWS console are populated in dropdown


how can I only let jester escape the facility

Go into config and disable all the monsters besides jester and also set the spawn outside chance to zero on them


Update GitHub Actions CI packages and command

Issue
N/A
Description
This change updates GitHub Actions packages actions/checkout to v4, actions/setup-go to v5, and actions/upload-artifacts to v4 to resolve NodeJS 16 deprecation warnings. It also updates GitHub Actions set-output command which is also deprecated.
Testing
CI passes without deprecation warnings
Additional references

https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/

Need to verify changes to release workflow. Will try to have an update by 2/13.


[RFE] More control over rancher/webhook updates

When we update Rancher and Kubernetes version on a large number of downstream clusters, that change must pass through internal validation steps (automatic tests, manual tests, deployment plan, approvals, etc...).
For us, the main benefit of using Rancher + RKE instead of using a managed Kubernetes service offered by a cloud provider is to have control on what is deployed on our environments and to have more control on when these deployments are done.
We are used to controlling Rancher and Kubernetes updates.
We don't understand how rancher-webhook has been updated to 0.3.4 last night without allowing us to test and approve that change. We didn't find documentation on that automatic update process.
We would like to have more control over the release process of rancher-webhook.
From my point of view (i'm a rancher user), I would expect rancher-webhook to be updated at the same time as Rancher.
Would it be possible to link webhook releases to Rancher ones ?
E.g. : The update of rancher 3.7.5 would trigger the rancher-webhook 0.3.6 update.
Optionally, we would be able to force a specific version of the rancher-webhook without updating Rancher (to be able to push a hotfix of rancher-webhook quickly if needed).

i close this issue because it is duplicated : https://github.com/rancher/rancher/pull/41302


iroh-net: implement portmapping

[x] UPNP: https://github.com/n0-computer/iroh/pull/1117
[ ] PMP
[ ] PCP

Ref: https://github.com/tailscale/tailscale/blob/main/net/portmapper/portmapper.go

Are we going to land PMP and/or PCP before the v0.5.0 release?

probably one but not both

progress update:
Both probes are working but the code needs love. Also, implementing mapping after this looks pretty simple now that most of the work is done and the road is paved
2023-07-12T23:25:47.601149Z DEBUG iroh_net::hp::portmapper: portmap starting
2023-07-12T23:25:47.604372Z TRACE iroh_net::hp::portmapper: tick: msg Some(Probe { result_tx: _ })
2023-07-12T23:25:47.604475Z DEBUG iroh_net::hp::portmapper: Starting portmapping probe
2023-07-12T23:25:47.604513Z DEBUG iroh_net::hp::portmapper::pmp: Starting pmp probe
2023-07-12T23:25:47.604579Z DEBUG iroh_net::hp::portmapper::pcp: Starting pcp probe
2023-07-12T23:25:47.636458Z DEBUG iroh_net::hp::portmapper::pmp: received pmp response Ok(PublicAddress { epoch_time: 1028171, public_ip: <sensored> })
2023-07-12T23:25:47.636590Z TRACE iroh_net::hp::portmapper: tick: pmp probe ready
2023-07-12T23:25:47.675048Z DEBUG iroh_net::hp::portmapper::pcp: received pcp response Response { version: Pcp, opcode: Announce, result_code: Success, lifetime_seconds: 0, epoch_time:
1028171 }
2023-07-12T23:25:47.675156Z TRACE iroh_net::hp::portmapper: tick: pcp probe ready
2023-07-12T23:25:48.672150Z TRACE iroh_net::hp::portmapper: tick: upnp probe ready
2023-07-12T23:25:48.672275Z TRACE iroh_net::hp::portmapper: tick: probe ready
2023-07-12T23:25:48.672349Z DEBUG iroh_net::hp::portmapper: upnp gateway changed "None" -> "http://<IP_ADDRESS>:58089/ctl/IPConn"
2023-07-12T23:25:48.672379Z DEBUG iroh_net::hp::portmapper: probe output portmap={ UPnP: true, PMP: true, PCP: true }

will need to be moved to 0.5.1 indeed


Singularity Update

Support for last Stable update 3.2.1 looks like something changed in Google Go version

@ashleychew , maybe we should use conditionals to decide on the Go version depending on the Singularity version we need to install?
Or are you willing to maintain a cygnet file only for the latest Singularity version?

@ashleychew , maybe we should use conditionals to decide on the Go version depending on the Singularity version we need to install?
Or are you willing to maintain a cygnet file only for the latest Singularity version?

I thought about it when I initially looked at it, the google go version is backward compatible ie forward compatible with singularity (well 3.x+) so the highest revision of google go will work against all singularity build at this stage


[CLOSED] Allow searching Specialist Documents by Slug and Title

 Issue by kalleth
Wednesday 22 October 2014 at 12:30 GMT
Originally opened as https://github.com/alphagov/specialist-publisher/pull/286

Trello ticket

Some of our document formats (AAIB and Drug Safety Updates, for example) have many documents, and our publishing tool doesn't easily allow users to find a document to amend other than by paging through the
results.
This change allows users to search by either slug or title using a form on the list page.
I have made the decision to truncate the results if there are more than the Kaminari pagination limit, and show a warning to the user. The alternative to this is to write another Paginator or modify the existing one to gracefully handle the searching case, which was a level of complexity I didn't feel was required for this task.
This behaviour was run past Holly.
I'm not entirely happy about the tight ties the tests have with AAIB reports, but as they're a subset of specialist documents I don't think it really matters.
Reviewing
Please review commit by commit.

kalleth included the following code: https://github.com/alphagov/specialist-publisher/pull/286/commits

 Comment by tommyp
Wednesday 22 October 2014 at 12:59 GMT

Maybe also show a flash with 'N results for "search term"'?

 Comment by tommyp
Wednesday 22 October 2014 at 12:58 GMT

Should probably say '0 results' if they aren't any, or at least show a flash to the user.

 Comment by kalleth
Wednesday 22 October 2014 at 15:21 GMT

@tommyp I've added a flash notice to tell the user if there's no results found. Showing info for 'n results for.. ' seems a bit much given it's only ever going to be one page of results -- also I'm not sure it should be a flash message, so I'll have a think about that tomorrow.

 Comment by kalleth
Thursday 23 October 2014 at 09:10 GMT

Having spoken to holly, we'd definitely like to add some search information ("Showing 41 results with slug matching: "cessna-a-"") but it's not required for this MVP. We've added a card to pick that up and she's happy for this to go through as-is.


silence|alerts: add metrics about current silences and alerts

This adds metrics that look like this:
alertmanager_alerts_count{state="active"} 6
alertmanager_alerts_count{state="suppressed"} 0
alertmanager_silences_count{state="active"} 1
alertmanager_silences_count{state="expired"} 1
alertmanager_silences_count{state="pending"} 0

This can be used to monitor alertmanager's usage and validate that
alertmanagers in a mesh have a similar number of silences and alerts.

count removed, tests fixed

Should be good now

I'm unsure about the performance implications the locking for counting may have

Should not be worse than accessing the API or the UI.

Fair enough. Thanks for the contributions!


Feature: Add Currently Open File to Task

Ideally, there would be some way to quickly add the currently open file to a task. When I click the Add Task option from the menu, it still gives me the file explorer. Maybe another option should be added that is "Add current file to task" or something.

Give me more ways to add it 😆

I gave it to you ❤


fix: Incrementing Duration #21

When seconds or minutes are added or removed via the Edit View,
minutes and hours should be adjusted accordingly. That is, we should
never allow 62 seconds or -5 minutes or any similar nonsense to be
displayed and stored as such.

Accidentally merged before ready. Hard-reverted on main.


Breaking/landing zone construction kit

This PR also comes with a draft release announcement as set out below, also appreciate your comments on this one:
Release Announcement
Welcome to the next iteration of collie! Whereas collie was a great tool to help cloud shepherds run their clouds before, collie will now also help you build them.
One consistent theme of feedback we got about collie from professional cloud shepherds like enterprise architects and platform engineers is that their key challenge lies in building and operating landing zones for their cloud customers. With its unified model to list tenants, tags, costs and IAM across all major cloud platforms, collie has been a great help to shepherds already running a flock of clouds. However, building up landing zones involves a myriad of different tools and technologies, as well as a complex landscape of functional requirements from your customers as well as compliance requirements from security stakeholders.
Staying true to the spirit of building a convenient operator experience on top of established CLI tools (like aws, az, gcloud), collie now also offers utilities to build and deploy modular landing zones leveraging with a GitOps approach on top of terraform and terragrunt. Our vision is that collie will empower cloud shepherds to build and operate their own Landing Zone Construction Kit as a means to rapidly assemble compliant landing zones across different clouds.
This change in scope of course means that we are introducing a number of breaking changes in this release. Experienced collie users should however have no problem finding all the existing capabilities under the collie tenant command. Here's an overview of the new command structure:
  Commands:

    init          - initialize a new collie repository in the current directory
    foundation    - Create, list and deploy cloud foundations covering all your cloud platforms
    tenant        - List tenants in your cloud foundations and manage tags, cost and IAM
    kit           - Manage a kit of reusable modules to build landing zones in your cloud foundation
    compliance    - Manage compliance frameworks and audit their implementation in your cloud foundation
    docs          - Generate end-user friendly documentation for your cloud foundation
    interactive   - Experimental interactive mode to explore your cloud foundations

The most important change is that collie now assumes you're working from a git repository ("collie repository") to manage one or more foundations. A foundation manages a set of cloud platforms (e.g. an AWS Organization, an AAD Tenant or a GCP Organization) as an intuitive tree of configuration files in your repository. This allows cloud shepherds to easily switch between e.g. a dev and prod environment, while building their landing zones from a common set of IaC modules. We will provide updated documentation and tutorials for this new approach together with our Landing Zone Construction Kit soon.
This release addresses a number of existing issues, all of which I propose to close with this PR

collie now uses a new process runner subsystem with consistent logging and observability capabilities, closing https://github.com/meshcloud/collie-cli/issues/143
collie version now shows installed and detected cli's, closing https://github.com/meshcloud/collie-cli/issues/131. The new process runner system also verifies cli installation status lazily, significantly improving collie invocations times
the new command structure solves previous issues https://github.com/meshcloud/collie-cli/issues/129 https://github.com/meshcloud/collie-cli/issues/96
Azure platforms are now filtering tenants on an AAD Tenant level, which is equivalent to the root-management group, closing https://github.com/meshcloud/collie-cli/issues/31

Review Notes
I've aligned on this change and conducted f2f code review with @tfelix

Really nice! The press statement is also well written. I just found one issue:
to build and deploy modular landing zones leveraging with a GitOps approach -> remove 'with'


Changed parent poms from operaton-database-settings to root/parent pom's

Changed parent pom's from operaton-database-settings to root/parent poms for some submodules where database is not relevant

Thanks a lot! I approved the workflow because you are a first-time committer, this won't be necessary for future PRs ;)  Will take a closer look later
It would be nice if you could change your commit message so it follows our conventions: https://github.com/operaton/operaton/blob/main/CONTRIBUTING.md#commit-message-conventions


Lists and Keys initial translation

@ImranMH Let's remove all the commented lines from this PR. It would be more cleaner and easy to review 😃

@ImranMH You are adding extra 8 lines in your PR. Can you please re check?

We are expecting to have same number of line changes.
Can you please undo these 3 missing lines

@nutboltu , @shakib609  would you guys have a check again

@ImranMH  Any update on this PR?

I think it already fully updated. You can have a final check


Connect to Community 'unable to generate' error styling

Very basic job of just adding padding here, otherwise the Marketplace library's 'unable to generate...' message on marketplace connect failure sits in here without any padding whatsoever. Looks kindof broken!
Added a touch more inline CSS to adding padding to the ccm-error div that gets returned here (and ONLY the error div...didn't want to add back into the ccm-pane-body class as i'm guessing you forcibly removed the padding inline for a reason!)

Sorry...should add that this is only for an error-generating Marketplace request from within the dashboard. In other words, the smaller marketplace error dialog that's generated from following a 'more themes' link from the dashboard > themes page, or from the 'extend concrete5' links in the footer of the dashboard mega-panel interface... NOT from within the 'more add-ons' button in the front-end, 'add block' dialog (this, larger, error page, has it's default .ui-dialog padding intact).


Only the last entity was being found by 'revup'.

If a function returns more than one entity, only the last one is found.

Fixed  by using "contains" instead of "startswith".


fix: remove unicodes non-needed in python3

Coverage remained the same at 90.23% when pulling 5f86c46a0b2b5041d658e6e6e4c769cd234ca360 on fix/unicode into b5c37636baf1eab9b34a2f821fbdc767a438ce35 on master.


DWI: report the small delta/capital delta

Related to #181, once we have the final, clean research protocol, we should keep track of the small delta/capital delta (they are not saved in the protocol itself, but we can read them at the scanner).

Yes, I had forgotten about this and indeed, it is very important.

In our current protocol (I refer here to the piloting session on July 14th):
delta=19.4 ms
DELTA=42.6 ms
To double-check if we change to Ines' DWI sequence

You can use latex on GitHub :)
$\delta$ = 19.4 ms
$\Delta$ = 42.6 ms
(which was created with the following markdown text)
$\delta$ = 19.4 ms
$\Delta$ = 42.6 ms

BTW, these I suspect will change if we change the TR of the DWI.

@acionca @esavary let's not forget to write these down for today's protocol (and keep doing it until we have the four of them written down).

From today's AP session I read:
$\delta$ = 23.3ms and $\Delta$ = 47.5ms

Pinning this for when I consolidate metadata in BIDS -- thanks @acionca and @helenelajous !


Can't turn on or off switches after update.

System Health details
.
Checklist

[X] I have enabled debug logging for my installation.
[X] I have filled out the issue template to the best of my ability.
[X] This issue only contains 1 issue (if you have multiple issues, open one issue for each issue).
[X] This issue is not a duplicate issue of currently previous issues..

Describe the issue
after the last update I can't turn on or of any switches.
I get the Error message:

Reproduction steps

restart livisi
restart livisi integration in home assistant
turn on/ off any livisi switch
...

Debug logs
Source: components/websocket_api/commands.py:240
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 12:00:36 (2 occurrences)
Last logged: 12:14:05

[546852822848] Failed to turn on None
[547382106560] Failed to turn on None
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 240, in handle_call_service
    response = await hass.services.async_call(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2279, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2316, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 892, in entity_service_call
    single_response = await _handle_entity_call(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 962, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/config/custom_components/livisi/switch.py", line 86, in async_turn_on
    raise HomeAssistantError(f"Failed to turn on {self._attr_name}")
homeassistant.exceptions.HomeAssistantError: Failed to turn on None

Diagnostics dump
...

If you have 1.8.0, update to 1.8.1
solve your issue


MalwareBytes' AI engine marks ModernFlyouts as malware

Hi, today my MalwareBytes started showing ModernFlyouts as malware. It is detected by the AI engine, so i don't know if this (hopefully) false positive has to do with that or with the new <IP_ADDRESS> update, because as soon as i updated the app, malwarebytes gave me this detection.

Thanks for letting us know @toineenzo.
Yes the detection is false . I have contacted Malwarebytes.

Malwarebytes has reported this as fixed.
Please reopen if this is not resolved.

@Samuel12321,

I have contacted Malwarebytes.

How?????

on their website they have a way to report false positives


DOC/BUG: Incorrect copyright in conf.py produces wrong copyright in official docs

Incorrect copyright in conf.py is producing wrong copyright on official pandas docs.
https://github.com/pandas-dev/pydata-bootstrap-sphinx-theme/blob/0a8df6ce84a1d1240e57c0ba747f5aba83a00e4b/pandas-docs/source/conf.py#L309

That's actually an issue for pandas itself (the pandas-docs directory is only an (outdated) patched subset to test the theme), but also solved recently, I see; https://github.com/pandas-dev/pandas/pull/31085 (so in the dev docs it is correct now: https://dev.pandas.io/docs/index.html)


Add Watcher attributes for the instance and class being watched on

This makes the Watcher more self-contained, providing all the information needed to subsequently unwatch it.

LGTM

Looks good to me as well. I'll merge as soon as one of the test sets goes green..


Support singer.io schema

Its python
Some has written a golang schema for it.
https://github.com/pbegle/singer-go
Here is one implementation of it:
https://github.com/transferwise/pipelinewise
doc: https://transferwise.github.io/pipelinewise/
k8 operator: https://github.com/dirathea/pipelinewise-operator
there are so many useful taps, and it makes it easy to do those integrations that takes huge amounts of code checked for data changes not being ordered. It seems to use a log based approach.
Is can do CDC off databases also, but also it seems other APIS.

Hey @gedw99, it'd be interesting to see if we can integrate with this. From the looks of it we'd probably need to add this as some sort of processor.

sorry about not responding. I got heavily side tracked with other stuff.
to manage expectations, i realised that as cool as Singer is, its does not meet my needs.
Please close or leave open @Jeffail


Launch fails with Python 3.10, Fix strict parameters for gui.move

Tor Browser Launcher
By Micah Lee, licensed under MIT
version 0.3.5
https://github.com/micahflee/torbrowser-launcher
Downloading Tor Browser for the first time.
Downloading https://aus1.torproject.org/torbrowser/update_3/release/Linux_x86_64-gcc3/x/en-US
Traceback (most recent call last):
  File "/usr/bin/torbrowser-launcher", line 30, in <module>
    torbrowser_launcher.main()
  File "/usr/lib/python3.10/site-packages/torbrowser_launcher/__init__.py", line 92, in main
    gui.move(
TypeError: arguments did not match any overloaded call:
  move(self, QPoint): argument 1 has unexpected type 'float'
  move(self, int, int): argument 1 has unexpected type 'float'

Adding int() wrappers around the parameters in question fixed it.

This PR solves https://github.com/micahflee/torbrowser-launcher/issues/628

I created #644. It solves it for me. This PR would, too.

Though, it doesn't seem to do a whole lot. torbrowser-launcher within a terminal gives the following:

It does it. It is another issue that should be adressed in another PR.

Please merge #542 to fix this.

This bug fix from Ubuntu's packaging is likely to resolve this in at least 1 week before this is acted upon here.

@micahflee can you take a look and merge this?


Support for LUKS disk encryption for local-storage

Support for LUKS encryption for local-storage can make hwameistor an even more complete solution. I'd like to be able to say; here's a PVC, encrypt it with LUKS with the key from this secret. And it'd create a logical volume encrypted with the given secret.
I think it might be a bit more challenging for local-disk stuff so I'd like to separate those two out into separate feature requests.

@wokalski Thanks for your feedback, data volume encryption is really important, especially inside scenarios with high data security requirements.
This feature is currently in our planning, it would be great if you could contribute to this feature, and if you have any questions during this process, please feel free to contact us, we will provide as much help as possible!

I think i might be able to contribute it but I'd need some information how you'd like it implemented:

Example how would it be configured
The encryption method on the lvm level - which commands etc would be used to perform encryption and decryption
Any "tricky situations" that come to your mind.

Thank you!

Example how it would be configured

I think this has to do with how users use it. Perhaps both StorageClass and LocalVolume should reflect whether encryption is applied and how it is done.

LocalVolume is our custom resource, which corresponds one-to-one to PV. Generally speaking, the information of the data volume will be included in this resource.
See https://github.com/hwameistor/hwameistor/blob/main/pkg/apis/hwameistor/v1alpha1/localvolume_types.go for more details.

As for the encryption key information, it might be associated through secrets. This relationship can be maintained in the StorageClass.
These are some of my current ideas about this. If there is a better way, please feel free to communicate~

Any "tricky situations" that come to your mind.

Actually, I'm not sure whether all the physical volumes (PVs) in a volume group (VG) should have the same encryption policy. If a data volume spans across multiple PVs with different encryption policies, is there any security risk?

The encryption method on the lvm level - which commands etc would be used to perform encryption and decryption

cryptsetup  can achieve LVM encryption, but I am thinking that having an LV level encryption method may be more flexible.I'm not sure if there is such a tool.
Ref: https://linuxhint.com/encrypt-lvm-volumes-luks/

cryptsetup  can achieve LVM encryption, but I am thinking that having an LV level encryption method may be more flexible.I'm not sure if there is such a tool.

cryptsetup can be used for LV level encryption as well so it's the right tool for the job 👍.

I will think about the rest and get back to you soon.

Same requirement.

@wokalski Hi, are you still working on this issue?

Unfortunately I'm not working on it actively. It is in my backlog but untouched.

I don't think it's stale - the maintainers expressed interest and there's no dedicated issue 😄

FYI: Data encryption is planned, this feature will come up in 1~1 months.
Anyone who has interest with this feature can follow https://github.com/hwameistor/hwameistor/issues/1581 :).

Closing this then


[CORE] LocalParitionWrriter causes OOM during mergeSpills

Backend
VL (Velox)
Bug description
LocalParitionWrriter::mergeSpills use arrow::io::MemoryMappedFile to read all spill files. But it is not munmap in time, which results in a significant consumption of RssFile.
Spark version
None
Spark configurations
No response
System information
No response
Relevant logs
No response

Would you like to post the OOM error message you have seen? Or it's only an abnormal consumption of rss files?

The error message is same as https://github.com/apache/incubator-gluten/issues/6947, killed by yarn. But the reasons are different.

CC @marin-ma

So the root cause is that the memory is unmapped until the file is close. When we merge the spills it eventually mapped all the spill data to memory. "Killed by yarn" error makes sense here.
Let's see if the ReadableFile performance is the same as MemoryMappedFile. It's an easy fix if so. Otherwise we need to manually unmap the file.

@jinchengchenghh, what's the way Velox used to merge spill file?
Most efficient way should be mmap + MADV_SEQUENTIAL + manually munmap

Would you like to post the OOM error message you have seen? Or it's only an abnormal consumption of rss files?

@zhztheplayer  we caught this issue by dumping /proc/self/status when the executor is killed. It shows that RssFile is almost 3G when VmRss is 3.5G. This issue affects a number of our large-scale ETL processes.
/proc/self/status

So the root cause is that the memory is unmapped until the file is close. When we merge the spills it eventually mapped all the spill data to memory. "Killed by yarn" error makes sense here.
Let's see if the ReadableFile performance is the same as MemoryMappedFile. It's an easy fix if so. Otherwise we need to manually unmap the file.

@FelixYBW
The Arrow MemoryMappedFile does not support a method to get the underlying mmap address. However, MemoryMappedFile::MemoryMap has a method to get the region data pointer, so we might add a method for MemoryMappedFile to return head() or data().
Another approach is in mergeSpills(), where we open the file for each partition and close it when finished. We do some internal performance test, it shows ReadableFile may have some regression.
@ccat3z Can you commit a PR and trigger the community performance benchmark by adding the comment /Benchmark Velox?

benchmark velox doesn't have spill. How much performance regression? Is the regression compared with mmap file or vanilla spark?

@marin-ma see if we can have any way to call madvise(MADV_SEQUENTIAL), it should be able to release physical memory quickly.

@marin-ma see if we can have any way to call madvise(MADV_SEQUENTIAL), it should be able to release physical memory quickly.

I tested the following patch on arrow, it seems that it will not be released immediately. It might still require explicit 'MADV_DONTNEED' in this case.
diff -ru apache-arrow-15.0.0.orig/cpp/src/arrow/io/file.cc apache-arrow-15.0.0/cpp/src/arrow/io/file.cc
--- apache-arrow-15.0.0.orig/cpp/src/arrow/io/file.cc   2024-11-09 11:39:58.497266369 +0800
+++ apache-arrow-15.0.0/cpp/src/arrow/io/file.cc        2024-11-09 14:50:37.035869206 +0800
@@ -41,6 +41,7 @@
 #include <sstream>
 #include <string>
 #include <utility>
+#include <iostream>

 // ----------------------------------------------------------------------
 // Other Arrow includes
@@ -575,6 +576,12 @@
       return Status::IOError("Memory mapping file failed: ",
                              ::arrow::internal::ErrnoMessage(errno));
     }
+    int madv_res = madvise(result, mmap_length, MADV_SEQUENTIAL);
+    if (madv_res != 0) {
+      return Status::IOError("madvise failed: ",
+                             ::arrow::internal::ErrnoMessage(errno));
+    }
+    std::cerr << "madvise success: " << result << " " << mmap_length << std::endl;
     map_len_ = mmap_length;
     offset_ = offset;
     region_ = std::make_shared<Region>(shared_from_this(), static_cast<uint8_t*>(result),
@@ -720,6 +727,27 @@
   return ::arrow::internal::MemoryAdviseWillNeed(regions);
 }

return ::arrow::internal::MemoryAdviseWillNeed(regions);
}

Does it call another madvise later? it will override previous one.

MemoryMappedFile has Region, when region is destructed, it called munmap. See how we can take use of it.

Another approach is in mergeSpills(), where we open the file for each partition and close it when finished.
@kecookier Not sure how much performance impact this might introduce. This approach requires invoking mmap and munmap for each partition, and some partitions in a single spill file may be quite small.

We do some internal performance test, it shows ReadableFile may have some regression.
How much performance regression do you see? Could you share some results? Thanks!

Velox use SpillReadFile to read the file, it uses FileInputStream to read the file and simd::memcpy to copy the bytes, It will output batch RowVector one by one. FileInputStream uses velox::LocalReadFile pread or preadv to read the file.
As I see, it reads bufferSize_  which is controlled by QueryConfig kSpillReadBufferSize (default 1MB) one time. Note: if file system supports async read, read double bufferSize_ one time.
@FelixYBW
readBytes = readSize();
      VELOX_CHECK_LT(
          0, readBytes, "Read past end of FileInputStream {}", fileSize_);
      NanosecondTimer timer_2{&readTimeNs};
      file_->pread(fileOffset_, readBytes, buffer()->asMutable<char>());

uint64_t FileInputStream::readSize() const {
  return std::min(fileSize_ - fileOffset_, bufferSize_);
}

/* Read data from file descriptor FD at the given position OFFSET
   without change the file pointer, and put the result in the buffers
   described by IOVEC, which is a vector of COUNT 'struct iovec's.
   The buffers are filled in the order specified.  Operates just like
   'pread' (see <unistd.h>) except that data are put in IOVEC instead
   of a contiguous buffer.

   This function is a cancellation point and therefore not marked with
   __THROW.  */
extern ssize_t preadv (int __fd, const struct iovec *__iovec, int __count,
		       __off_t __offset) __wur;

How much performance regression do you see? Could you share some results? Thanks!

@marin-ma After internal re-testing last weekend, no noticeable performance regression was found.

In io_util.cc, madvise(willneed) is called again. You may disable it.

The new patch is here, but it's not working either. There might still be some codes that haven't been found, I'll recheck it if required.
diff -ru apache-arrow-15.0.0.orig/cpp/src/arrow/io/file.cc apache-arrow-15.0.0/cpp/src/arrow/io/file.cc
--- apache-arrow-15.0.0.orig/cpp/src/arrow/io/file.cc   2024-11-09 11:39:58.497266369 +0800
+++ apache-arrow-15.0.0/cpp/src/arrow/io/file.cc        2024-11-09 18:09:06.540567675 +0800
@@ -41,6 +41,7 @@
 #include <sstream>
 #include <string>
 #include <utility>
+#include <iostream>

 // ----------------------------------------------------------------------
 // Other Arrow includes
@@ -575,6 +576,12 @@
       return Status::IOError("Memory mapping file failed: ",
                              ::arrow::internal::ErrnoMessage(errno));
     }
+    int madv_res = madvise(result, mmap_length, MADV_SEQUENTIAL);
+    if (madv_res != 0) {
+      return Status::IOError("madvise failed: ",
+                             ::arrow::internal::ErrnoMessage(errno));
+    }
+    std::cerr << "madvise success: " << result << " " << mmap_length << std::endl;
     map_len_ = mmap_length;
     offset_ = offset;
     region_ = std::make_shared<Region>(shared_from_this(), static_cast<uint8_t*>(result),
@@ -660,8 +667,8 @@
   ARROW_ASSIGN_OR_RAISE(
       nbytes, internal::ValidateReadRange(position, nbytes, memory_map_->size()));
   // Arrange to page data in
-  RETURN_NOT_OK(::arrow::internal::MemoryAdviseWillNeed(
-      {{memory_map_->data() + position, static_cast<size_t>(nbytes)}}));
+  // RETURN_NOT_OK(::arrow::internal::MemoryAdviseWillNeed(
+  //     {{memory_map_->data() + position, static_cast<size_t>(nbytes)}}));
   return memory_map_->Slice(position, nbytes);
 }

If ReadableFile doesn't have significant perf loss, let's use the quick fix and optimize it to mmap in future. @marin-ma can you test #7861 on our jenkins with spill?

Velox use SpillReadFile to read the file, it uses FileInputStream to read the file and simd::memcpy to copy the bytes, It will output batch RowVector one by one. FileInputStream uses velox::LocalReadFile pread or preadv to read the file.

The optimal way should be we map like 1M each time, unmap it once accessed. Looks MemoryMappedFile::Region can implement it.

In io_util.cc, madvise(willneed) is called again. You may disable it.

The new patch is here, but it's not working either. There might still be some codes that haven't been found, I'll recheck it if required.
diff -ru apache-arrow-15.0.0.orig/cpp/src/arrow/io/file.cc apache-arrow-15.0.0/cpp/src/arrow/io/file.cc
--- apache-arrow-15.0.0.orig/cpp/src/arrow/io/file.cc   2024-11-09 11:39:58.497266369 +0800
+++ apache-arrow-15.0.0/cpp/src/arrow/io/file.cc        2024-11-09 18:09:06.540567675 +0800
@@ -41,6 +41,7 @@
 #include <sstream>
 #include <string>
 #include <utility>
+#include <iostream>

 // ----------------------------------------------------------------------
 // Other Arrow includes
@@ -575,6 +576,12 @@
       return Status::IOError("Memory mapping file failed: ",
                              ::arrow::internal::ErrnoMessage(errno));
     }
+    int madv_res = madvise(result, mmap_length, MADV_SEQUENTIAL);
+    if (madv_res != 0) {
+      return Status::IOError("madvise failed: ",
+                             ::arrow::internal::ErrnoMessage(errno));
+    }
+    std::cerr << "madvise success: " << result << " " << mmap_length << std::endl;
     map_len_ = mmap_length;
     offset_ = offset;
     region_ = std::make_shared<Region>(shared_from_this(), static_cast<uint8_t*>(result),
@@ -660,8 +667,8 @@
   ARROW_ASSIGN_OR_RAISE(
       nbytes, internal::ValidateReadRange(position, nbytes, memory_map_->size()));
   // Arrange to page data in
-  RETURN_NOT_OK(::arrow::internal::MemoryAdviseWillNeed(
-      {{memory_map_->data() + position, static_cast<size_t>(nbytes)}}));
+  // RETURN_NOT_OK(::arrow::internal::MemoryAdviseWillNeed(
+  //     {{memory_map_->data() + position, static_cast<size_t>(nbytes)}}));
   return memory_map_->Slice(position, nbytes);
 }

Thank you for your test. Can you try posix_fadvise? We have a case to show madvise doesn't work on file mapping.

Just found some test:

Could you introduce what each columns in stdout represents?

Could you introduce what each columns in stdout represents?

Not sure :( It's a test two years ago.

some link about mmap we referred before.
http://xoyo.space/2017/11/mmap-performance-analyzing-and-tuning/
https://github.com/xoyowade/mmap_benchmark
https://stackoverflow.com/questions/30470972/using-mmap-and-madvise-for-huge-pages
https://lwn.net/Articles/590693/

summary:

I wrote a simple gist to dump memory usage during reading file. Both MADV_DONTNEED and munmap can release RssFile immediately. And MAP_POPULATE significantly improve performance. However, MAP_POPULATE will use hundreds of MB of untrackable RssFile, which is not acceptable in this case.
So I believe that the combination of manual MDAV_WILLNEED and MADV_DONTNEED is the best solution. I will test it in gluten later.


Add Prometheus metrics for the controller

We could add metrics for the following:

kubeless_controller_function_create_total{namespace, function_name, topic_name, success}
kubeless_controller_function_create_duration_seconds_sum{namespace, function_name, topic_name, success} and _count

The same for delete and update

sg. We might have to combine create and update, since the two are intertwined in the controller code.
Something that counted individual operation failures and/or queue length would be good too, since we should expose the distinction between the count/duration of an individual update attempt, and the overall count/duration of an object update including all the retries.


fix(schedule): hide publish schedule when not connected to congregation account

:tada: This PR is included in version 2.3.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Update license to match the PDF.js license

Update to match the PDF.js license - Apache.

@donal-tobin-sap @devinea @jlongie: Could you please review?

👍

:+1:

:-1: use correct license https://spdx.org/licenses/

@detobin: Re-review please? 😺

:+1:


fix: re-enable semantic release

My concern with semantic-release is this: how does it interact with Registrator.jl/TagBot?
My understanding is that semantic-release creates Git tags. But the preferred workflow for registering Julia packages is to first register the release with Registrator, and then after the pull request is merged into the registry, then TagBot creates the tag.
In particular, I have noticed the Julia General registry maintainers specifically ask that we not make a tag until they have merged the pull request into the registry.
Is there a way to configure semantic-release to not create tags?
The conversations in these two threads might be helpful:

https://github.com/JuliaRegistries/General/pull/670
https://github.com/JuliaRegistries/General/pull/1103

Good point and that's reasonable.

Also, perhaps you could email Fernando Gelin (he’s here at Brown) and ask him what they’ve ended up doing on their semantic release repositories?
Certainly it’s a nice workflow and makes releases convenient. It’d be nice if we could make it play well with the Julia workflow.

I guess the other thing we’ll need to figure out is how semantic-release interacts with Bors. I think semantic-release makes commits to master. But for Bors to be most effective, no one except Bors should be able to commit to master - that’s how we ensure the 100% guarantee that master is always green.
So we’d have to figure out how semantic-release would work in a repository with Bors enabled.
But first we have to figure out how semantic-release can work with the Registrator/TagBot workflow. Once we have that solved, we can figure out how semantic release will work in the setting of Bors.

It looks like the ideal workflow is to use something like the semantic-release tool to automatically update the version number in Project.toml and generate release notes, based on structured commits. While neither the Python nor JS semantic-release version support the first part, it wouldn't be hard to extend them to do so. We'd also probably need some kind of bot to automatically add the comment to Julia to register the release.
The harder part is integrating this with bors. Ideally, we want the version to be generated just prior to bors merging into master, after tests are run, which means Project.toml needs to be updated right before the merge is done. This might be possible from Travis, but it would seem to require pushing a commit to the staging branch, which... I'm not sure is a good idea.
Basically, I'm imagining a flow like this:
PR -> bors [test] -> bors [staging] -> Update Project.toml (new part) -> bors[merge] -> Registrator -> TagBot
This might be a bit cleaner if bors RFC0322 ever materialises.

Confirmed with Fernando that they haven't found a solution and were just dropping semantic release.