introduce custom schedulers for parallel iterators

This PR introduces the notion of "custom schedulers". The idea is that you can invoke, for any parallel iterator iter, iter.in_scheduler(S), and it will run with the scheduler S. The scheduler basically takes the place of the (currently hard-coded) "bridge" functions -- it connects a parallel producer and a parallel consumer.
We introduce a Scheduler associated type into the ParallelIterator trait. This is used to ensure that the user cannot invoke in_scheduler() twice on any given parallel iterator. For implicity, we also forbid customized schedulers from the "inputs" where mutiple parallel iterators are "joined" (like zip() or chain()). Hence a.in_scheduler().zip(b) is illegal, but a.zip(b).in_scheduler() is fine. We could alter these rules to permit both (but we want to avoid a.in_scheduler().zip(b.in_scheduler())).
In this PR, Rayon exposes three schedulers:

DefaultScheduler -- what you get if you don't say anything; also encodes the fact that in_scheduler has never been called. Currently, this delegates to AdaptiveScheduler.
AdaptiveScheduler -- the default strategy of "split to depth N, and more if stolen"
FixedLenScheduler -- like above, but where we adjust N to try and respect the user's min/max requests

Originally, the idea here was to allow you to swap out the Rayon runtime completely. In fact, you can still see this in the Runtime trait introduced by the first commit, though I've removed it by the end -- basically because I don't feel we're ready to commit to this trait yet (and in principle no flexibility is lost). I imagine we'll probably add this back at some point in some form.
Some unknowns:

[ ] Module structure. I feel like we should move most of these things to iter::scheduler.
[ ] What should be done with the Scheduler associated type in ParallelIterator?

It's kind of weird right now. It sounds like it specifies the scheduler, but really it's always one of two values: DefaultScheduler or CustomScheduler. This is just used to prevent multiple in_scheduler() calls. I think I'd rather just give a semantics to multiple calls in some useful way. This interacts with the next question.

[ ] Can we support "layering" schedulers?

Right now, we prohibit multiple calls to in_scheduler, but I was thinking that it'd also be possible to permit "combining" schedulers (e.g., with_min and with_max). I have to play around with this a bit more.

[ ] What to do with chain() and other ad-hoc calls to join()?

@cuviper recently modified the drive() impl of chain() so that it will drive in parallel, using join(). This kind of goes against the grain of this PR, so we probably want to either extend the scheduler interface with a join method or reframe that as a parallel producer consumer, or something like that.

cc #93 -- this permits a user-provided "backend", though I think ultimately more flexibility might be desired. (This only works for parallel iterators, for one thing.)

OK, I see I was not running all the tests and some broke. Bah. Well, I've got to run now, but I'll get them working.

Rebased and fixed the broken tests. (Well, let's see what travis has to say.)

Yet another custom scheduler (code and diff). The design is not finalized and we wish to get feedback early.
We tried another design which diverges in following ways:

Scheduler is not an associated type in the ParallelIterator trait, but only belongs to WithScheduler, which is created by iter.with_scheduler(S).
WithScheduler forces the bridge of producer and consumer to happen in this part. That require the iterator on the left to work in producer mode and iterator on the right to work in consumer mode.
Distinguish between Scheduler and UnindexedScheduler, since their behaviour are so different.

Some consequences of the design:

Non breaking change (at least for indexed parallel iterator). No need to change the implementation of other existing tools like Map, Zip and MinLen.
User-provided "backend" is supported.
Scheduler can't be applied before zip. So a.in_scheduler().zip(b) and a.zip(b.in_scheduler()) is illegal. That is because after in_scheduler(), iter can only work in consumer mode, but Zip works in producer mode, and iter on it's left can only work in producer mode.
Scheduler can't be applied twice, so there is no "layering" schedulers. The reason is similar to above.

Some minor designs that could change:

Use empty link hack to forbid using iter.with_scheduler(S) further in producer mode. When users do that, the program simply won't compile and output an ugly link error.
Some alternatives:

Use panic!(). Program will compile but panic when bridge happens.
Use more rigours type system to seperate with_producer() and drive() into two different trait. So we can tell whether a ParallelIterator support working in producer mode or consumer mode. This would be a breaking change.

Some obstacle we are facing:

Current ParallelIterator doesen't have with_unindexed_producer() method, so we can't explicitly get the producer. We believe breaking change is needed for support of unindexed scheduler.
Benchmark is hard...
We find that merely using scheduler in a program will affect the performance of other part of the program where just default bridge function is used. We have no idea where this gloabl effect comes from.

Features we wish to implement in the future:

UnindexedScheduler
We believe breaking change is needed to obtain the producer from ParallelIterator.
Thread Affinity
We wish to allow users to assign which core they want to use to for a task and a slice of ParallelIterator.

@xiaoniu-578fa6bff964d005

Benchmark is hard...
We find that merely using scheduler in a program will affect the performance of other part of the program where just default bridge function is used. We have no idea where this gloabl effect comes from.

A lot of performance depends on sufficient inlining -- it could just be that your additional code changed some thresholds in the module that caused LLVM to make different inlining decisions.
Apart from raw performance, you could also try to evaluate your changes at a higher level, like @wagnerf42 has been doing with his rayon-adaptive. He has logging infrastructure that can show exactly how the jobs were divided, so differences can be compared and justified.
(Those API changes are more invasive than what you seem to be suggesting.)

@xiaoniu-578fa6bff964d005

Current ParallelIterator doesen't have with_unindexed_producer() method, so we can't explicitly get the producer. We believe breaking change is needed for support of unindexed scheduler.

Perhaps a breaking change can be avoided by adding a new trait for scheduling purposes.
Anyway, I'd suggest opening your own issue or pull request to discuss your design further.

I'm going to close this PR as being obviously stalled and outdated.


Writing to file+S3 does not work

With this config:
    resourceLog:
      json:
        file: 'resources.json'
        s3:
          bucket: anz-innovation-revolver-bucket
          path: output/resources.json
      csv:
        file: 'resources.csv'
        s3:
          bucket: anz-innovation-revolver-bucket
          path: output/resources.csv
        reportTags: ['Name', 'Schedule']
      console:
        reportTags: ['Name', 'Schedule']
    auditLog:
      console:
      csv:
        file: audit.csv
        s3:
          bucket: anz-innovation-revolver-bucket
          path: output/audit.csv
        append: false

Looks like the code assumes (by returning early) that both will not be true at the same time:
  process(): any {
    if (this.options.file) {
      return this.writeFile();
    }
    if (this.options.s3) {
      return this.writeS3();
    }
  }

If I disable the file output, the S3 fails anyway:
2024-02-14 10:23:10.668 INFO    /lib/objectLog.js:110   revolver        Writing data to s3://anz-innovation-revolver-bucket/output/resources.json
2024-02-14 10:23:41.134 ERROR   /lib/accountRevolver.js:142     revolver        failed to write resourcesLog json: Region is missing

With PR #180 and following config:
      csv:
        file: 'resources.csv'
        s3:
          bucket: anz-innovation-revolver-bucket
          region: ap-southeast-2
          path: output/resources.csv
        reportTags: ['Name', 'Schedule']
      console:
        reportTags: ['Name', 'Schedule']
    auditLog:
      console:
      csv:
        file: audit.csv
        s3:
          bucket: anz-innovation-revolver-bucket
          region: ap-southeast-2
          path: output/audit.csv
        append: false


Asynchronous JavaScript

Description
Hello everyone !!
This issue will help readers in acquiring all the knowledge that one needs to know about Asynchronous JavaScript!

Introduction (What is Asynchronous Javascript?)

Promise, Callbacks

Event Loop

Async/await

Domain
Frontend Dev HTML/CSS/JS
Type of Contribution
Documentation
Code of Conduct

[X] I follow Contributing Guidelines & Code of conduct of this project.

/assign


script in layers needed for qgis

they look like this

cd qgis
make
ha

solution is to
mkdir .make first before running make
maybe best to just add this to the readme?
leaving open for now

Leave it open, the male file should create that folder
On Sat, May 14, 2022 at 8:07 AM Allan Irvine @.***>
wrote:

solution is to
mkdir .make first before running make
maybe best to just add this to the readme?
leaving open for now
—
Reply to this email directly, view it on GitHub
https://github.com/smnorris/bcfishpass/issues/232#issuecomment-1126702950,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKDCHKUN3T6FWLGGJBVH53VJ6JPVANCNFSM5V4T7Q2A
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


Variant number is not increased when new rpms of the same build is published

Platform: CentOS 7 (but probably affects others)
Architecture: x86_64 (but probably affects others)
When changes are released to the rpm packages release branch (not nightly) without changing the version of the JRE or JDK, the variant number on the rpm package should be increased so that the fact that there have been changes is evident and each rpm in the release repo is an immutable artifact.

As @aahlenst this has been built in


PER-447 - Provisioning EKS in EDB-Terraform

Adds support for AWS Elastic Kubernetes Service
Updated 'README.md' to reflect Azure Kubernetes Service and AWS Elastic Kubernetes Service

Note: the Kubernetes VPC's from each provider is not connected with region peering at the moment and this will eventually need to be handled so that we do not need to use the public interface, vpn or kubectl when using virtual machines, or other, to simulate workloads within a provider's private network.


Fix HTTP/1.0 connections not closing without keep-alive

Fixes #18
What was happening under the hood is that ab expects the connection to close, after a request and only to stay open when using -k (that is sending the Connection: Keep-Alive Header). ab is using HTTP/1.0 and our logic of handling it was missing the case of an HTTP/1.0 request without Connection header.
Also when using the keep-alive feature of HTTP/1.0 ab expects the server to acknowledge the keep-alive by sending Connection: Keep-Alive in the response. I think this is kindof valid behaviour and should happen that way. I updated the examples to work with this assumption, but didn't manage to get it into the library and out of users responsibility.
I honestly only used curl and wrk and my browser to interface with this lib yet.

Hi @mfelsche,
The changelog - fixed label was added to this pull request; all PRs with a changelog label need to have release notes included as part of the PR. If you haven't added release notes already, please do.
Release notes are added by creating a uniquely named file in the .release-notes directory. We suggest you call the file 19.md to match the number of this pull request.
The basic format of the release notes (using markdown) should be:
## Title

End user description of changes, why it's important,
problems it solves etc.

If a breaking change, make sure to include 1 or more
examples what code would look like prior to this change
and how to update it to work after this change.

Thanks.

I am ignoring the failing lint check, as it is complaining about the release notes md file not having a level 1 heading on first line, although main just suggested me to do so. I will obey main, not any random linter. All hail ponylang-main!


upgrade sbt 1.0.0 -> 1.0.1

(for the limited number of projects that are even on sbt 1.0.x at all)
hopefully this will make #597 go away

test run https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-community-build/1898/consoleFull


Jquery .val() undefined

Whenever i call $.val() im getting it as undefined, I believe the javascript is getting executed before the dom has loaded, but I'm not sure why its happening and how to solve it

@LongJeongS would you mind posting this on Stack Overflow and then you can link to it back here if you want.  We're trying to get things run through SO for these sorts of coding questions.
Also, when you post there, if you could post a code example that will us provide the feedback you need.  :smile:

ahh ok, will do. Thanks :+1:


Pixie review

Initial doc commit
:thumbsup:
Signed-off-by: Ed Robinson<EMAIL_ADDRESS>

Great work, /approve


Keep annotation source field Description

When OP is "self" or "first", keep the annotation source's field Description instead of a message like "transfered from matched variants in..."

https://github.com/ousamg/vcfanno/pull/2


Segmentation fault (core dumped) happened when I use more than one session.

Hello. I'm new to Keras. I'm using TensorFlow as a backend. I'd like to load many models and use those in different threads. For different threads, I have to use different graphs, so I have to use different sessions. (Am I right?)
Here is a trimmed version of my code that can reproduce this error. What am I doing wrong?
I am using Keras-2.2.0 and tensorflow 1.8.0 without GPU on Ubuntu 16.04.4.
import tensorflow as tf
import numpy as np

from keras.models import Sequential
from keras.layers import Masking

def predict():
    model = Sequential()
    layer = Masking(batch_input_shape=[None, 1])
    model.add(layer)

    x = np.array([[0]], dtype='int32')

    model.predict(x, batch_size=1)

if __name__ == '__main__':
    with tf.Session():
        predict()

    with tf.Session():
        predict()

The error disappeared using tensorflow 1.9.0.


Create the logs_cdn machine

I think the ELB security group needs to allow access from Fastly IPs to rsyslog ports, as configured in govuk-provisioning, instead of 443

this should mount a persistent volume; it's important we keep a history of our CDN logs

@surminus Yep, I was going to look into that now. Dean told me to copy Bouncer's config because it had access from Fastly and external IPs in it, hence why this was a bit wrong for the cdn-logs case.

The IAM policy needs to be attached.

@surminus I added an IAM policy.
When I know I've done everything correctly, I'll squash the commits.


Add ROS2 version of Stopwatch

🎉 New feature
Summary
Add a ROS2 version of math::Stopwatch that can make use of rclcpp::Clock and sim time via /clock
Test it
Check test/test_stopwatch.cpp for examples
Checklist

[x] Signed all commits for DCO
[x] Added tests
[ ] Added example and/or tutorial
[ ] Updated documentation (as needed)
[ ] Updated migration guide (as needed)
[ ] Consider updating Python bindings (if the library has them)
[ ] codecheck passed (See contributing)
[x] All tests passed (See test coverage)
[ ] While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

Linters are failing

moving to gazebo-style with public: on each member caused the linters to fail


Module helpers  cannot import name 'highlight'

Describe the bug
Module helpers  cannot import name 'highlight'
To Reproduce
Steps to reproduce the behavior:

git clone --recursive https://github.com/byt3bl33d3r/CrackMapExec
cd CrackMapExec
sudo pip3 install -r requirements.txt
sudo python3 setup.py install
sudo crackmapexec -h

Expected behavior
Print help
Screenshots
None
Crackmapexec info

OS: Debian GNU/Linux bullseye/sid x86_64
Version of CME: master
Python: 3.8.3rc1

Additional context
None

Strange, cannot reproduce the bug

With the same version of python ?

yes python3.8

I do not know thanks to what witchcraft, I no longer have the error.
Thanks @mpgn for your replies.


Changelog ?

Hi there,
I'm deeply sorry if this information is available elsewhere but I failed to find it.
Could we add it in a dedicated CHANGELOG.md, in the README.mdor via github tags. ?
I'd like to update my package to the latest version but I need to ensure no breaking changes occurred. Thanks in advance

Hi @adrien-may!
That's a really good idea! I'll create a CHANGELOG.md in the next update.
Right now, we are in the "get people to try it out and give feedback" stage of development.
There haven't been any breaking changes. Mostly adding things people suggest.
What version are you using now?
How are you finding using the Web Components so far?

Using 0.0.3 right now. Working ok so far, we’ve been using the react
deprecated sdk for quite a while, this web component approach seems to work
better !
Le mar. 1 août 2023 à 18:11, Dwane Hemmings @.***> a
écrit :

Hi @adrien-may https://github.com/adrien-may!
That's a really good idea! I'll create a CHANGELOG.md in the next update.
Right now, we are in the "get people to try it out and give feedback"
stage of development.
There haven't been any breaking changes. Mostly adding things people
suggest.
What version are you using now?
How are you finding using the Web Components so far?
—
Reply to this email directly, view it on GitHub
https://github.com/opentok/web-components/issues/20#issuecomment-1660642529,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APPMGHDE6GJKRX6J6NUMPYTXTETB5ANCNFSM6AAAAAA2Z2D72Y
.
You are receiving this because you were mentioned.Message ID:
@.***>

@adrien-may
Sounds good. As you can see we are still in v0.0.x and looking for people's feedback before we publish v1.0.0 with everything stable.
I was just reminded from another issue that we went from setting the width and height on the elements, i.e.:
 <video-publisher witdh="320px" height="240px"></video-publisher>

to setting a properties object (see: https://tokbox.com/developer/sdks/js/reference/OT.html#initPublisher) because folks said they wanted to have more properties to change.
So if you wanted to set the properties, it would be the same as setting the session and token.
publisher.current.properties = {width: '320px', height: '240px'}

I'm going to update the sample docs and code to be more explicit.

Hi @adrien-may
Nothing has changed since your message except some documentation updates. When you say the old version, do you mean you are setting the height and width like:
 <video-publisher witdh="320px" height="240px"></video-publisher>
instead setting the properties like
publisher.current.properties = {width: '320px', height: '240px'}
I don't see anything major changing for a while. We've just released a beta of Video API to be used with Vonage applications for a more unified experience with everything in one dashboard.
As for the Web Components, nothing functionally will change. The only difference will be in the way you generate the session and token. So regardless if you are still using OpenTok creds or Vonage creds, you just need to pass in the session and token for the Web Components to work just like normal.
I'm going to be converting and creating some applications using the Web Components with Vonage credentials to make sure everything is the same and will start a change log announcing being able to use Vonage credentials with the current versions of the components and update the examples and docs.
In conclusion, aside from testing using Vonage credentials. I don't see the Web Components changing (functionality) for a while. I'm still trying to get more feedback before anything will change. If it's working for folks, we'll leave it as it is for the foreseeable future and release a v1 early next year.
Hope that helps. Let me know if you need anything more.
How do you find using the Web Components in your application? Any feedback?


A fix for #207 — Signal Generator distorts over long periods of time

This fix changes SignalGenerator such that its internal time value to wrap-around for every period of the generated waveform.  This requires more testing but is posted here for visibility.
(This PR was posted just before and closed due to an error by me, I was basing off the wrong HEAD.)

I think at large values of i the cast to float loses precision. f32 can store integers up to 16,777,216 exactly. At 44.1k sample rate distortion will start after 380 seconds, growing more extreme as time goes on.

@dvdsk also floats' precision is relative while period is fixed, hence generator timing precision will drop gradually over time. f64 may have enough digits to make it unnoticeable, though.
I see wrapping may only be a problem with rather small periods (with frequencies that are close to the sample rate). Rounding error there may become comparable to sampling period. That  means that rounded remainder may introduce slight timing error. If signal period is exact multiply of sampling rate the wrapping version is exact, and definitely better than the current one.

i needs to wrap around but I'm not sure how to negotiate this considering I've promised a f32 period in the interface, do I have to find a least common multiple of I and period or can I just dump out of i after it gets sufficienly high?

listing all related issues/pr's (so we can close em once this is done)

https://github.com/RustAudio/rodio/pull/202
https://github.com/RustAudio/rodio/issues/208

Current approach using sample counter will not work. You have to switch to adding an increment.

period = 1/freq
in 1s we have sample_rate samples

thus every sample advances we advance (1/sample_rate) * (1/period) of a full cycle (between 0 and 1)
Example:
sample-rate = 10, period = 10 seconds, each sample takes 1/100 of a full cycle

now we want the full cycle to be between 0 and 2pi instead of 0 and 1 -> multiply by 2p
we do not want the period but the frequency -> replace period by 1/frequency

gives: cycle_increment = freq/sample_rate * 2pi which is what #202 uses. To prevent cycle_increment (or a better name if you can think of it) from blowing up over time (and getting to the part where floats lose precision) just do: cycle_increment =  cycle_increment mod 2pi

It did not occur to me that floats support modulo too. Yes, float increment is better, since wrap-around will be calculated exactly. As I understand this requires to change trait Function.  Also try_seek implementation will have to be adjusted accordingly.
So the implementation would:
next (&mut self) {
  let sample = self.f(self.t); //  t is cycle_pos
  self.t += self.dt;
  if self.t >= 1.0 { self.t -= 1.0; }  // t =  t % 1.0
  sample
}

Then, sin generator reduces to
fn render(t: f32) -> f32 {
  (t * 32::consts::TAU).sin()
}

This limits functions to be periodic, but it looks like fancier versions are not anticipated.
Regarding API, I am not sure it makes sense to do match Function on every iteration unless we want to switch those dynamically. Each function can be its own Source (maybe that is too verbose) or generator can be parametrized (SignalGenerator<Function>).

Regarding API, I am not sure it makes sense to do match Function on every iteration unless we want to switch those dynamically. Each function can be its own Source (maybe that is too verbose) or generator can be parametrized (SignalGenerator).

Making the function generic makes it a chore to pass the source around. Any struct the source is stored in, or function its passed too now must be generic too. Unless you use Box, but that is slower then the match that is there now.

The latter also lets users to provide their own functions for generation.

Are there enough other functions that you could want be passed in for this to make sense? If we miss a few then adding them to signal-generator makes more sense to me. Since that does not require the code to become more complex.

We could implement the function as a fn, they're all simply enough. When the SignalGenerator is created could just use the function enum to pick which one and then we save this in a slot on the struct.
Could then also add a second initializer to the struct that would allow a client to write their own fn,

Could then also add a second initializer to the struct that would allow a client to write their own fn,

wouldn't that make the Signalgenerator generic over the function? Thats my main issue the custom function approach. Having generics in Source could become annoying for end users.

No, because the fn would always have the same type. Essentially the struct becomes:

#[derive(Clone, Debug)]
pub struct SignalGenerator {
    sample_rate: cpal::SampleRate,
    period: f32,
    function: fn(f32) -> f32, //this was Function,
    i: u64,
}

Then in the initializer we set function to whatever it needs to be, probably with a closure that captures the period variable.

No, because the fn would always have the same type.

ahh yes I forgot fn is a function pointer. Wonder what that does to perf, we shall see. Given how fast signal generator is already I do not think it matters at all.
Kinda fun since you can throw f32::sin in there if you want a sine wave :)
We could ship standalone square and sawtooth wave functions too. Then the API would be something like: Signalgenerator::new(rodio::signals::square)

Having a lot of cruft in type parameters is a price for not having dyn dispatch, if I understand this. I see existing Sink types may already get long. This is not strictly necessary unless we want absolutely top performance where most functions can be inlined.
User-provided function could be a wavetable synth? :) It does not have to be some analytically simple stateless function. But if making this generator flexible requires too much of a hassle its not worth it unless there is actual need for that.
If only a plain function is supported does it mean it cannot be a closure? A plain (stateless) fn would be enough for existing cases, though.

in order of my personal preference we can have

sine/sawtooth etc as seperate Sources, possible just aliasing SignalGenerator with F a function. This is as performant as it can be and pretty discoverable but not extendable.
(the current situation) a SignalGenerator which you pass an enum selecting the signal you want to generate. Its fine I think users might look for the words sine in the sources list and mis this but a search will turn up Function::Sine, I guess that could get a docs link to SignalGenerator.
a SignalGenerator which you pass a signal function (fn(f32) -> f32) as paramater. A sine wave is made with SignalGenerator::new(std::f32::sin). More signal functions are provided in a rodio module. This will be slower but maybe marginally so since the fn can not be inlined as its size is unknown to the compiler. Its fun :) I kinda like it, especially passing in std::f32::sin just feels fun. It will confuse users though so lets not do it. Also YAGNI.
a SignalGenerator which takes something implementing FnMut(f32) -> f32 and stores and uses it through dynamic dispatch (Box<dyn FnMut(f32) -> f32>). Can we provide ready made structs that implement FnMut? Or do we provide a lot of example code? Slower maybe more then the previous option, way harder to use as you need to know what closures are (rodio attracts quite a lot of novices). I see no reason for this since no one has presented a usecase and it just introduces complexity. Very much YAGNI.
a SignalGenerator taking a generic argument FnMut through static dispatch (impl<F: FnMut(f32) -> f32> SignalGenerator<F>) super versatile and can be fully optimized just like seperate sources (the first option). Still has the disadvantages of the previous option and also requires the generics to be named. Those generics spread through the users option. Honestly we should never implement this unless a very clear usecase is presented. Peak YAGNI

Separate sources will have some duplicating code. But likely those can be extracted into helper functions. I would not object this, though. We can have other implementations later when needed.

Thanks all, let me get to work on this and I'll post changes. Per @dvdsk 's note I'll continue to implement (2) but it's easy to add interfaces along the lines of (1), either with some boilerplate or maybe a macro if we're being really nerdy.

This addresses the guts of everything discussed here I think:

SignalGenerator now uses an f32 to keep track of elapsed time, as a proportion of the completed period. This variable wraps around the interval of (0.0,1.0) to prevent loss of precision when the generator runs for long periods of time.
We no longer perform a match for every rendered sample of the signal generator, the selected test signal is now generated by a function pointer that's selected at init-time.
I have created some convenience sources for generating triangle, square and sawtooth waves to make these test signals more visible to new users.

Looking good to me.

Looking good to me.

very clean nice :+1:

Looking good to me.

very clean nice 👍
I guess the little artifacts come from resampling or the Fourier itself?

The artifacts are mostly from the windowing function, here's the cos3 window of the same test.

There are odd harmonics but the first one is -80 dB below the fundamental, that's something like one part in 10 million, well below the f32::EPSILON and is probably spectral leakage in the fft anyways. (FFTs are very sensitive to window selection depending on what you're trying to do.)

Looking good to me.

very clean nice 👍
I guess the little artifacts come from resampling or the Fourier itself?

The artifacts are mostly from the windowing function, here's the cos3 window of the same test.

You know what? When I make a test signal with iZotope or Pro Tools it's perfect, I'm only getting from the signal generated by SignalGenerator. I'm not sure exactly what's going on here.

I figured it out, I was recording rodio's output through a virtual device and there was a sample-rate conversion happening. Once I set everything to 44.1 it cleans up.

let me know once this is ready (I think it still needs a note in the change-log it supports a custom function now?)

Okay all ready I think.

very nice! thanks for all the hard work!

Always a pleasure and very educational!

Since SignalGenerator with a function callback option was chosen, and it already implements Source then separate structs for various function shapes are not necessary. Those differ only in initialization parameters. Functions can be passed directly. Do we need the function enum compatibility? So SawtoothWave could be implemented for example as
pub fn sawtooth(frequency: f32, sample_rate: u32) -> SignalGenerator {
        SignalGenerator::with_function(cpal::SampleRate(sample_rate), frequency, sawtooth_signal)
}

OK, can refine this later...

your right, its a little bit more complex then it needs to be. Feel free to address that directly on main.

Since SignalGenerator with a function callback option was chosen, and it already implements Source then separate structs for various function shapes are not necessary. Those differ only in initialization parameters. Functions can be passed directly. Do we need the function enum compatibility? So SawtoothWave could be implemented for example as
pub fn sawtooth(frequency: f32, sample_rate: u32) -> SignalGenerator {
    SignalGenerator::with_function(
        cpal::SampleRate(sample_rate), frequency, sawtooth_signal)
}

OK, can refine this later...

Yeah I didn't do this because it'd be a breaking change to remove the enum but please feel free if you want.

I might have misunderstood @PetrGlad so let me clarify my stance. I agree the individual sources (structs) Sine, Sawtooth and Square may use SignalGenerator::with_function() and that would be slightly neater. I do not think it really needed. We can adress it when we remove SignalGenerator::new. What we can do is mark SignalGenerator::new as deprecated.


[UT] add more special resolution and slice-mode tests

https://rbcommons.com/s/OpenH264/r/1686/

Are all the new tests necessary here, or could some of them be removed?
The merged of this pull request made the runtime of the test suite significantly longer. On Travis, the runtime went from around 18 minutes (https://travis-ci.org/cisco/openh264/builds/157794575) to 40 minutes (https://travis-ci.org/cisco/openh264/builds/156316540). When running tests on ARM, each of the new tests take around 180 seconds, and there's 4 of them.
I understand that you want to test things thoroughly, but when the tests get this big (for not much added value), it can at worst force people to stop running the tests, just because it takes way too long.

Also, when running the unit tests in valgrind (in order to make sure that there are no use of uninitialized memory, no memory leaks etc), the runtime of the unit test suite went from around 20-30 minutes to 65-80 minutes.

@mstorsjo thx for the info, we will shorten these new test first and see if there is other things we can do to save some more time somewhere else.


spark-k8s service account is not created in kubeflow user's namespaces

When deploying the operator alongside Charmed Kubeflow, the service account is correctly created in the kubeflow namespace. However it is not created in the user's own namespace. This means sparkapplications do not run when created in the user's namespace.
As I see it, we need to either (a) have an alternative service account already created with the correct permissions in the user's namespace ready to go, (b) automatically create the spark service account in users' namespaces when the spark-k8s operator is deployed (which looks like a problematic can of worms to me) or (c) provide instructions to the user about creating a suitable service account (least desirable option).

The code for a workaround - copy the spark-k8s service account from kubeflow namespace:
kubectl get sa spark-k8s -n kubeflow -o yaml | sed 's/namespace: kubeflow/namespace: <NEW>/'  | kubectl create -f -

Replace  with your namespace name.

This workaround will only work for anyone who has access to the kubeflow namespace, so an admin or similar would have to apply it for general users.

Hi @Barteus this is something we'd like to fix by providing the right RBAC to users, which is something the charm code has to figure out. If we copy the sa into each user namespace we are providing access to resources and actions that are not necessarily related to spark, and thus a better workaround would be to create sa with just access to spark objects (those defined in the CRDs). We can leave this bug open until we provide a good fix.

Do we know which of the following is the issue?

that a service account called spark-k8s is required in the user namespace, but that we currently don't have one
that users do not have the proper RBAC to enable them to create instances of the CRs that the Spark Operator deploys (eg: SparkApplication or ScheduledSparkApplication)?
something else?

I ask because, unless the problem is (1), I'm not sure how copying the ServiceAccount alone fixes.
If this is actually an RBAC issue (2), I agree with @dnplas that we can be a bit more specific with our fix (and I'm confused how copying the SA alone would fix anything, but I might be misunderstanding something).  If we need RBAC assigned to users, I think the appropriate way to give users this RBAC would be either to:
a. (if deploying spark on its own) make role bindings and a service account for these permissions and put them in spark's namespace
b. (if deploying alongside kubeflow, enabling all kubeflow users to use spark) create ClusterRoles for with the desired user RBAC and use Kubeflow's role aggregation procedure to get them attached to all users.
To do (b), we create ClusterRoles with the rbac.authorization.kubeflow.org/aggregate-to-kubeflow-* label as we do here.  Ideally, these ClusterRoles would be created and managed by the charm operator itself, but I think that the barrier for pod spec charms (including this Spark Operator) is that pod spec does not let us create arbitrary ClusterRoles.  As a workaround, we created the kubeflow-roles-operator to (if I recall correctly...) manage roles for any legacy pod spec charms, and as we decommissioned those charms we'd pull those roles back out.
So assuming this is an RBAC issue, I see two possible actions:

short term hack: give users guidance on creating their own ClusterRoles that get aggregated into the regular kubeflow roles
short term fix: add a spark ClusterRole to the kubeflow-roles-operator
long term fix: whenever this gets converted to sidecar, move the ClusterRole creation into the charm code

Spark integration is not supported. A new design is being introduced and integration with Spark will be different.
This will need to be revisited and spec'ed out from the beginning.
Closing.


Code: 67, Error: Bad Net Name

Hi,
I am trying to use SMB, but it throws exception Code: 67, Error: Bad Net Name
Code: 67, Error: Bad Net Name
Have anyone run into this issue?
Thanks

This issue is likely to occur when you provide a local path instead of a UNC path. The expectation (of both Steeltoe and the underlying libraries) is that you will provide the path in the form \\server\path\to\share
How are you describing the location of the SMB share?

Hi Tim,
I tried the following, but none of them work.
\servername\path
\server address\path\
Thank you

This error message is coming from Windows, it's only passing through Steeltoe. Have you validated that you can access the share from the location the app is running? Could there be networking issues if you're 100% sure the path is correct?
https://stackoverflow.com/questions/9705535/how-can-i-resolve-a-error-bad-net-name-error-in-a-call-to-wnetaddconnection2

This issue should have been resolved with 2.4.1, please re-open if it persists


Selecting "Ready Now" as a Map filter returns no results

Is this because we just don't have any right now? Or is it a bug?
https://main--hubblehomes-com--aemsites.hlx.page/available/quick-move-ins

I currently see one

According to the spread sheet there is only one home which got scraped from the site, so I guess there is only one?!

Code is working as designed.


CRDs upgrade problem with Helm

By default, helm won't upgrade our CRDs. This behavior might introduce some unexpected result.
There is no support at this time for upgrading or deleting CRDs using Helm.
This was an explicit decision after much community discussion due to the
danger for unintentional data loss. Furthermore, there is currently no community
consensus around how to handle CRDs and their lifecycle. As this evolves,
Helm will add support for those use cases.

https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations
https://github.com/helm/helm/issues/5871

We have fixed it in https://github.com/kubesphere/ks-installer/pull/1931
/close


AsyncDuplexStreamingCall hang/deadlock on net48 on linux

What version of gRPC and what language are you using?
Grpc.Core 2.35.0 (latest)
Grpc.Tools 2.35.0 (latest)
Google.Protobuf 3.14.0 (latest)
C# (net48)
What operating system (Linux, Windows,...) and version?
Linux (Ubuntu 18.04)
What runtime / compiler are you using (e.g. python version or version of gcc)
mono <IP_ADDRESS>
What did you do?
Please see https://github.com/shaan1337/async-duplex-streaming-call-hang for a minimal repro of the issue or
async-duplex-streaming-call-hang.zip
What did you expect to see?
The application should exit.

The application does not hang when the target framework is changed to netcoreapp3.1 or net5.0
The application does not hang when AsyncDuplexStreamingCall is changed to another type of call (e.g AsyncClientStreamingCall)
Attaching a debugger shows that the C# gRPC thread pool is getting stuck during the shut down phase (during application exit)

What did you see instead?
The application hangs.
Anything else we should know about your project / environment?
I'm available if you need any additional information. I'm attempting to find the cause of the deadlock and will share updates if I make any progress.

Thanks for the reproduction. I ran it and got the same results with the application hanging using target framework net48 but passing for net5.0 or netcoreapp3.1.

I can also reproduce using your example, but at the same time there seem to be problem with your example itself:

After setting GRPC_VERBOSITY=debug env variable I got this warning (which indicates that you're probably not waiting for the client call to finish as you should)

W0315 11:30:59.219262 Grpc.Core.Channel Channel shutdown was called but there are still 1 active calls for that channel.

the hang seems to be caused by the TestInterceptor logic. TBH I don't understand what the interceptor's logic is supposed to do (it first invokes the ReadWrite handler, but then returns a new instance of AsyncDuplexStreamingCall for no apparent reason.

you never invoke await _channel.ShutdownAsync() and await _server.ShutdownAsync() before exiting from the app as you should  -  even though that doesn't seem to be actually causing the behavior you're seeing, it's a contributing factor.

See https://github.com/grpc/grpc/blob/d589f4e6ca88168c306ebc7a883a615dc46d3f2b/src/csharp/Grpc.Core.Tests/Interceptors/ClientInterceptorTest.cs#L169 for an example of a meaningful DuplexStreamingCall interceptor.

Since the logic in the TestInterceptor serves no clear purpose an is probably broken and removing the interceptor makes the problem go away, I'm going to close this issue since it doesn't seem to be a gRPC bug.


Use existing Kafka cluster

I'm a maintainer of https://github.com/Yolean/kubernetes-kafka, so naturally we have a cluster already :) Also https://github.com/kubernetes/charts/tree/master/incubator/kafka is quite widely adopted, judging by the number of pulls from https://hub.docker.com/r/solsson/kafka/.
I disagree with https://github.com/kubeless/kubeless/issues/32. Kafka is an excellent choice of events backend, with semantics that fit nicely with serverless function execution (can strive for exactly once). Also it makes it easy to integrate with other services in a streaming platform.
Can Kubeless use an existing Kafka cluster? You'd basically only need to specify any requirements on kafka config, and decouple kafka from the rest of Kubeless through a bootstrap brokers config. Maybe have some naming convention for topics.

Hi @solsson good to see you here, I have checked out your Yolean images of Kafka :)
I am glad you agree with the use of Kafka. I think it is a matter of expertise and taste.
To your issue, yes you can totally reuse an existing kafka cluster. The coupling is very lite.
Currently when you install kubeless with the manifest we generate in the release https://github.com/kubeless/kubeless/releases/download/0.0.16/kubeless-0.0.16.yaml you will see that we launch a very simple kafka setup with single nodes statefulesets.
The coupling happens in two locations:

the cli which provides a convenience wrapper kubeless topic to create,ls,delete topics. This is very much a kubectl exec into the kafka pod and execution of the kafka bash script to manage topics. This is hackish and needs to be improved. https://github.com/kubeless/kubeless/blob/cba55476923ab38d5454b07be37396d9885fb4f3/cmd/kubeless/topicCreate.go#L30

the runtime for events. It is basically a kafka consumer. This kafka consumer listens for events on kafka.kubeless. See: https://github.com/kubeless/kubeless/blob/master/docker/runtime/python-2.7/event-trigger/kubeless.py#L38

Currently, events are only supported for the python runtime. We should be able to get to the Node.js within the next two weeks.
So say you have an existing kafka cluster, if you could expose it within k8s as kafka.kubeless dns entry that you are up and running.
We should actually make that kafka endpoint a bit more generic to avoid hard-coding the dns name.

Thanks for pointing out the built manifest. Without it the setup was a bit opaque to me.
I hope to get time to test this as a branch from https://github.com/Yolean/kubernetes-kafka/pull/30, now that I have vacation :) I haven't yet considered how/if we'll use Kubeless in production, but it'll be interesting to see where discussions in #186 and #148 are heading as we already have build infrastructure - and obviously events - in our clusters.

the cli which provides a convenience wrapper kubeless topic to create,ls,delete topics

I think this mechanism can be in Kubeless independent of how Kafka is hosted. It too needs name+port to brokers. For example we used a simple java client  at first, but more often these days we use the shell script in separate Jobs using a /bin/bash -c command with the Kafka image. To Kafka it makes no difference, and I think execing into the production image has no advantage over using a separate container.

the runtime for events. It is basically a kafka consumer. This kafka consumer listens for events on kafka.kubeless

I guess a service+endpoint could map to the actual service, for example kafka.kafka. However, due to https://github.com/Yolean/kubernetes-kafka/issues/21 I've decided to remove the Service from kafka (see referencing commits).
Maybe the kafka.kubeless service could use arbitrary broker(s) as endpoint if Kubeless uses it as "bootstrap", i.e. to get the actual names of online brokers. Thanks to StatefulSet those are cluster-global already, for example kafka-0.broker.kafka.svc.cluster.local:9092.

Would love to help you if you want to try this out, and any feedback would be very welcome.
and definitely using your existing kafka configuration should not be a huge hurdle

I'll be on vacation for two weeks. Will give this a shot after that.

We've released kubernetes-kafka v2.0.0 now, and I just confirmed how easy it is to set up Kubeless on top of it. Summarized in https://github.com/Yolean/kubernetes-kafka/pull/48.
As you predicted @sebgoa the CLI's topic support fails with FATA[0000] Can't find any kafka pod. Maybe it could use the kafka.kubeless service to find brokers, or does it need an equivalent zookeeper service? Anyway users of kubernetes-kafka will likely be proficient in creating topics so it's not a big deal.
I suggest that you release a separate manifest, like you've done with openshift and rbac, for Kubeless with existing Kafka. I've only deleted from the release manifest, not edited anything. The only thing that needs to be configurable is externalName field, https://github.com/Yolean/kubernetes-kafka/pull/48/files#diff-7c5fe8f6bc7c0bdce8d75d7387f4cd08R9. This assumes that the kafka setup will provide actual listener resolvable names through the service.
I guess the service can also rely on a kind: Endpoints resource, for Kafka outside kubernets, so maybe it's best left to the user to define kafka.kubeless.

@solsson great, thanks for trying this.
If I recall correctly the kafka topic find the kafka broker pod via a label query with labels kubeless:kafka so assuming you label your broker with that, the command should work...I think...
cc/ @ngtuna see comment about creating a manifest with a kafka setup kubeless-no-kafka.yaml ? and probably we need to be able to point to a different DNS name than kafka.kubeless

Tested with the label in https://github.com/Yolean/kubernetes-kafka/pull/48/commits/fe3a6b3a0943366168deda25abce21cdf5234385, but still getting FATA[0000] Can't find any kafka pod. Does it assume that the label is in the kubeless namespace?

@ngtuna can you check this ? thx

Took the time to investigate this myself now :) Yes it assumes kafka is in the kubeless namespace, and it also assumes an install path with /opt/bitnami/kafka/bin/kafka-topics.sh. But as I suggested before we can scope this out because those who have a kafka cluster already will be comfortable with creating topics in some other way.

@solsson sorry for a bit delay in response. So I just wrap up this:

the current deployment places kafka and zk statefulset into kubeless namespace and hard-coding the dns to kafka.kubeless
kubeless topic executes this command /opt/bitnami/kafka/bin/kafka-topics.sh in the kafka pod. It finds the kafka pod by label kubeless: kafka

So:

that's easy to make a kubeless-without-kafka/zk manifest and we should do that (in upcoming release)
the external kafka should have this label kubeless:kafka
We should not fixing the namespace,  so the pubsub runtime can find it via the injected envvar KUBELESS_KAFKA_NAMESPACE
the external kafka will handle topic creation and publishing, the kubeless runtime will do the consumption.

Are you fine with that ? @sebgoa @solsson

the external kafka should have this label kubeless:kafka
We should not fixing the namespace, so the pubsub runtime can find it via the injected envvar KUBELESS_KAFKA_NAMESPACE

This means that there wont be a Service with name:kafka in the kubeless namespace, but that pubsub will look up brokers directly? I think that would be great, because k8s services obscure actual broker listeners.

This means that there wont be a Service with name:kafka in the kubeless namespace

Even if there is a kafka in kubeless namespace it's still be fine because the pubsub runtime is prioritized to get the KUBELESS_KAFKA_NAMESPACE first.

Adding a label to your kafka pods is a convenient way to qualify them for Kubeless.

Yeah. But for kubeless topic, there still remains a tricky point that someone deploys kafka and label it as kubeless: kafka. What we are doing is we just pick up the first pod in the list.

@ngtuna is this available in the latest release?

@nikhita this one should be relatively easy to start with.
You should run a kafka deployment outside kubeless...then figure out how to start kubeless to use that specific kafka. Verify that pubsub functions work.
It should give you a smooth intro to running kubeless and using pubsub functions.

hello,
i am running kafka on a different port: 9094
is there any config where i can change that ?

Hello,
Unfortunately it is not possible to configure Kafka in a different port and make it work with Kubeless yet. I am curious about why that port setup is a best practice, could you elaborate?

because you want different security levels,
https://github.com/wurstmeister/kafka-docker/blob/master/README.md#listener-configuration

Pretty sure we fixed this cc/ @andresmgot @ngtuna

Not really, PubSub functions still expect Kafka in the kubeless namespace listening in the port 9092

No we have fixed this already. The pubsub func read KUBELESS_KAFKA_SVC and KUBELESS_KAFKA_NAMESPACE envvars which are defaulted to kafka and kubeless. So if the func is deployed with --env flag to provide those env vars it will work. The missing bit is that the port 9092 is fixed. We should consider to support @lghinet 's case.

I thought that the idea was to support an external Kafka system (so it doesn't even need to be in the same cluster).

In the case of Kafka being on the same cluster, do we have documentation on how to use it?

It is not. We need to document it.

I have created https://github.com/kubeless/kubeless/issues/587


Deploy from image not working, results in: git:from-image: command not found

Description of problem
How reproducible
Use deploy_docker_image to deploy an docker image from ghcr.io
Steps to Reproduce

Create a workflow with the following step

  deploy-to-dokku:
    runs-on: ubuntu-latest
    needs: build-and-push-image
    steps:
      - name: Push to dokku
        uses: dokku/github-action@master
        with:
          git_remote_url: 'ssh://dokku@MY_IP_ADDRESS:22/my-app'
          ssh_private_key: ${{ secrets.SSH_PRIVATE_KEY }}
          deploy_docker_image: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}

Actual Results
bash: line 1: git:from-image: command not found

Expected Results
deployed 🎉

Environment Information
dokku report MYAPP output
Details

-----> uname: Linux racknerd-5b1c8c 5.15.0-89-generic #99-Ubuntu SMP Mon Oct 30 20:42:41 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
-----> memory:
                      total        used        free      shared  buff/cache   available
       Mem:            7937        2339         399          38        5198        5252
       Swap:           2303         531        1772
-----> docker version:
       Client: Docker Engine - Community
        Version:           24.0.7
        API version:       1.43
        Go version:        go1.20.10
        Git commit:        afdd53b
        Built:             Thu Oct 26 09:07:41 2023
        OS/Arch:           linux/amd64
        Context:           default

       Server: Docker Engine - Community
        Engine:
         Version:          24.0.7
         API version:      1.43 (minimum version 1.12)
         Go version:       go1.20.10
         Git commit:       311b9ff
         Built:            Thu Oct 26 09:07:41 2023
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.6.25
         GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
        runc:
         Version:          1.1.10
         GitCommit:        v1.1.10-0-g18a0cb0
        docker-init:
         Version:          0.19.0
         GitCommit:        de40ad0
-----> docker daemon info:
       Client: Docker Engine - Community
        Version:    24.0.7
        Context:    default
        Debug Mode: true
        Plugins:
         buildx: Docker Buildx (Docker Inc.)
           Version:  v0.11.2
           Path:     /usr/libexec/docker/cli-plugins/docker-buildx
         compose: Docker Compose (Docker Inc.)
           Version:  v2.21.0
           Path:     /usr/libexec/docker/cli-plugins/docker-compose

       Server:
        Containers: 7
         Running: 7
         Paused: 0
         Stopped: 0
        Images: 17
        Server Version: 24.0.7
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Using metacopy: false
         Native Overlay Diff: true
         userxattr: false
        Logging Driver: json-file
        Cgroup Driver: systemd
        Cgroup Version: 2
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
        Swarm: inactive
        Runtimes: io.containerd.runc.v2 runc
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
        runc version: v1.1.10-0-g18a0cb0
        init version: de40ad0
        Security Options:
         apparmor
         seccomp
          Profile: builtin
         cgroupns
        Kernel Version: 5.15.0-89-generic
        Operating System: Ubuntu 22.04.3 LTS
        OSType: linux
        Architecture: x86_64
        CPUs: 4
        Total Memory: 7.751GiB
        Name: racknerd-5b1c8c
        ID: **********************************
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
         File Descriptors: 72
         Goroutines: 66
         System Time: 2024-03-03T22:20:25.628655117Z
         EventsListeners: 0
        Experimental: false
        Insecure Registries:
         <IP_ADDRESS>/8
        Live Restore Enabled: false

-----> herokuish version:
       herokuish: v0.7.2
       buildpacks:
         heroku-buildpack-multi     v1.2.0
         heroku-buildpack-ruby      v259
         heroku-buildpack-nodejs    v224
         heroku-buildpack-clojure   v91
         heroku-buildpack-python    v237
         heroku-buildpack-java      v73
         heroku-buildpack-gradle    v39
         heroku-buildpack-scala     v96
         heroku-buildpack-play      v26
         heroku-buildpack-php       v239
         heroku-buildpack-go        v181
         heroku-buildpack-nginx     v24
         buildpack-null             v3
-----> dokku version: dokku version 0.32.3
-----> dokku-event-listener version: 0.15.0build+5268732
-----> dokku-update version: dokku-update 0.7.2
-----> docker-container-healthchecker version: 0.7.2
-----> docker-image-labeler version: 0.6.1build+c6e15a9
-----> git version: git version 2.34.1
-----> lambda-builder version:        0.5.0
-----> netrc version: 0.7.1build+677c033
-----> plugn version: plugn: 0.13.0build+fd5297a
-----> sigil version: 0.10.1build+e443be0
-----> sshcommand version: sshcommand 0.17.1
-----> dokku plugins:
         00_dokku-standard    0.32.3 enabled    dokku core standard plugin
         20_events            0.32.3 enabled    dokku core events logging plugin
         app-json             0.32.3 enabled    dokku core app-json plugin
         apps                 0.32.3 enabled    dokku core apps plugin
         builder              0.32.3 enabled    dokku core builder plugin
         builder-dockerfile   0.32.3 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.32.3 enabled    dokku core builder-herokuish plugin
         builder-lambda       0.32.3 enabled    dokku core builder-lambda plugin
         builder-nixpacks     0.32.3 enabled    dokku core builder-nixpacks plugin
         builder-null         0.32.3 enabled    dokku core builder-null plugin
         builder-pack         0.32.3 enabled    dokku core builder-pack plugin
         buildpacks           0.32.3 enabled    dokku core buildpacks plugin
         caddy-vhosts         0.32.3 enabled    dokku core caddy-vhosts plugin
         certs                0.32.3 enabled    dokku core certificate management plugin
         checks               0.32.3 enabled    dokku core checks plugin
         common               0.32.3 enabled    dokku core common plugin
         config               0.32.3 enabled    dokku core config plugin
         cron                 0.32.3 enabled    dokku core cron plugin
         docker-options       0.32.3 enabled    dokku core docker-options plugin
         domains              0.32.3 enabled    dokku core domains plugin
         elasticsearch        1.35.0 enabled    dokku elasticsearch service plugin
         enter                0.32.3 enabled    dokku core enter plugin
         git                  0.32.3 enabled    dokku core git plugin
         haproxy-vhosts       0.32.3 enabled    dokku core haproxy-vhosts plugin
         letsencrypt          0.20.3 enabled    Automated installation of let's encrypt TLS certificates
         logs                 0.32.3 enabled    dokku core logs plugin
         mongo                1.36.6 enabled    dokku mongo service plugin
         network              0.32.3 enabled    dokku core network plugin
         nginx-vhosts         0.32.3 enabled    dokku core nginx-vhosts plugin
         openresty-vhosts     0.32.3 enabled    dokku core openresty-vhosts plugin
         plugin               0.32.3 enabled    dokku core plugin plugin
         ports                0.32.3 enabled    dokku core ports plugin
         postgres             1.36.0 enabled    dokku postgres service plugin
         proxy                0.32.3 enabled    dokku core proxy plugin
         ps                   0.32.3 enabled    dokku core ps plugin
         redis                1.37.1 enabled    dokku redis service plugin
         registry             0.32.3 enabled    dokku core registry plugin
         repo                 0.32.3 enabled    dokku core repo plugin
         resource             0.32.3 enabled    dokku core resource plugin
         run                  0.32.3 enabled    dokku core run plugin
         scheduler            0.32.3 enabled    dokku core scheduler plugin
         scheduler-docker-local 0.32.3 enabled    dokku core scheduler-docker-local plugin
         scheduler-null       0.32.3 enabled    dokku core scheduler-null plugin
         shell                0.32.3 enabled    dokku core shell plugin
         ssh-keys             0.32.3 enabled    dokku core ssh-keys plugin
         storage              0.32.3 enabled    dokku core storage plugin
         trace                0.32.3 enabled    dokku core trace plugin
         traefik-vhosts       0.32.3 enabled    dokku core traefik-vhosts plugin
=====> my-admin app-json information
       App json computed selected:    app.json
       App json global selected:      app.json
       App json selected:
=====> my-admin app information
       App created at:<PHONE_NUMBER>
       App deploy source:             docker-image
       App deploy source metadata:    ghcr.io/myapp/lungo-prod:latest
       App dir:                       /home/dokku/my-admin
       App locked:                    false
=====> my-admin builder information
       Builder build dir:
       Builder computed build dir:
       Builder computed selected:
       Builder global build dir:
       Builder global selected:
       Builder selected:
=====> my-admin builder-dockerfile information
       Builder dockerfile computed dockerfile path: Dockerfile
       Builder dockerfile global dockerfile path: Dockerfile
       Builder dockerfile dockerfile path:
=====> my-admin builder-herokuish information
       Builder herokuish computed allowed: true
       Builder herokuish global allowed: true
       Builder herokuish allowed:
=====> my-admin builder-lambda information
       Builder lambda computed lambdayml path: lambda.yml
       Builder lambda global lambdayml path: lambda.yml
       Builder lambda lambdayml path:
=====> my-admin builder-nixpacks information
       Builder nixpacks computed nixpackstoml path: nixpacks.toml
       Builder nixpacks global nixpackstoml path: nixpacks.toml
       Builder nixpacks nixpackstoml path:
       Builder nixpacks computed no cache: false
       Builder nixpacks global no cache: false
       Builder nixpacks no cache:
=====> my-admin builder-pack information
       Builder pack computed projecttoml path: project.toml
       Builder pack global projecttoml path: project.toml
       Builder pack projecttoml path:
=====> my-admin buildpacks information
       Buildpacks computed stack:     gliderlabs/herokuish:latest-22
       Buildpacks global stack:
       Buildpacks list:
       Buildpacks stack:
=====> my-admin caddy information
       Caddy image:                   lucaslorentz/caddy-docker-proxy:2.8
       Caddy letsencrypt email:
       Caddy letsencrypt server:      https://acme-v02.api.letsencrypt.org/directory
       Caddy log level:               ERROR
       Caddy polling interval:        5s
       Caddy tls internal:            false
=====> my-admin ssl information
       Ssl dir:                       /home/dokku/my-admin/tls
       Ssl enabled:                   true
       Ssl hostnames:                 my-admin.upto11.nl lungo.myapp.com
       Ssl expires at:                May 18 07:41:39 2024 GMT
       Ssl issuer:                    C = US, O = Let's Encrypt, CN = R3
       Ssl starts at:                 Feb 18 07:41:40 2024 GMT
       Ssl subject:                   subject=CN = my-admin.upto11.nl
       Ssl verified:                  verified by a certificate authority
=====> my-admin checks information
       Checks disabled list:          none
       Checks skipped list:           none
       Checks computed wait to retire: 60
       Checks global wait to retire:  60
       Checks wait to retire:
=====> my-admin docker options information
       Docker options build:          --link dokku.postgres.my-admin-pgsql:dokku-postgres-my-admin-pgsql
       Docker options deploy:         --link dokku.postgres.my-admin-pgsql:dokku-postgres-my-admin-pgsql --restart=on-failure:10 -v /var/lib/dokku/data/storage/my-admin-uploads:/opt/app/public/uploads
       Docker options run:            --link dokku.postgres.my-admin-pgsql:dokku-postgres-my-admin-pgsql -v /var/lib/dokku/data/storage/my-admin-uploads:/opt/app/public/uploads
=====> my-admin domains information
       Domains app enabled:           true
       Domains app vhosts:            my-admin.upto11.nl lungo.myapp.com
       Domains global enabled:        true
       Domains global vhosts:         upto11.nl
=====> my-admin git information
       Git deploy branch:             main
       Git global deploy branch:      master
       Git keep git dir:              false
       Git rev env var:               GIT_REV
       Git sha:                       f1961ce
       Git source image:              ghcr.io/myapp/lungo-prod:latest
       Git last updated at:<PHONE_NUMBER>
=====> my-admin haproxy information
       Haproxy image:                 byjg/easy-haproxy:4.3.0
       Haproxy letsencrypt email:
       Haproxy letsencrypt server:    https://acme-v02.api.letsencrypt.org/directory
       Haproxy log level:             ERROR
=====> my-admin letsencrypt information
       Letsencrypt active:            true
       Letsencrypt autorenew:         false
       Letsencrypt computed dns provider:
       Letsencrypt global dns provider:
       Letsencrypt dns provider:
       Letsencrypt computed email<EMAIL_ADDRESS>
       Letsencrypt global email:
       Letsencrypt email<EMAIL_ADDRESS>
       Letsencrypt expiration:<PHONE_NUMBER>
       Letsencrypt computed graceperiod: 2592000
       Letsencrypt global graceperiod:
       Letsencrypt graceperiod:
       Letsencrypt computed lego docker args:
       Letsencrypt global lego docker args:
       Letsencrypt lego docker args:
       Letsencrypt computed server:   https://acme-v02.api.letsencrypt.org/directory
       Letsencrypt global server:
       Letsencrypt server:
=====> my-admin logs information
       Logs computed max size:        10m
       Logs global max size:          10m
       Logs global vector sink:
       Logs max size:
       Logs vector sink:
=====> my-admin network information
       Network attach post create:
       Network attach post deploy:
       Network bind all interfaces:          false
       Network computed attach post create:
       Network computed attach post deploy:
       Network computed bind all interfaces: false
       Network computed initial network:
       Network computed tld:
       Network global attach post create:
       Network global attach post deploy:
       Network global bind all interfaces:   false
       Network global initial network:
       Network global tld:
       Network initial network:
       Network static web listener:
       Network tld:
       Network web listeners:                <IP_ADDRESS>:1337
=====> my-admin nginx information
       Nginx access log format:
       Nginx access log path:         /var/log/nginx/my-admin-access.log
       Nginx bind address ipv4:
       Nginx bind address ipv6:       ::
       Nginx client max body size:
       Nginx disable custom config:   false
       Nginx error log path:          /var/log/nginx/my-admin-error.log
       Nginx global hsts:             true
       Nginx computed hsts:           true
       Nginx hsts:
       Nginx hsts include subdomains: true
       Nginx hsts max age:            15724800
       Nginx hsts preload:            false
       Nginx computed nginx conf sigil path: nginx.conf.sigil
       Nginx global nginx conf sigil path: nginx.conf.sigil
       Nginx nginx conf sigil path:
       Nginx proxy buffer size:       4096
       Nginx proxy buffering:         on
       Nginx proxy buffers:           8 4096
       Nginx proxy busy buffers size: 8192
       Nginx proxy read timeout:      60s
       Nginx last visited at:<PHONE_NUMBER>
       Nginx x forwarded for value:   $remote_addr
       Nginx x forwarded port value:  $server_port
       Nginx x forwarded proto value: $scheme
       Nginx x forwarded ssl:
=====> my-admin openresty information
       Openresty access log format:
       Openresty access log path:     /var/log/nginx/my-admin-access.log
       Openresty bind address ipv4:
       Openresty bind address ipv6:   ::
       Openresty client max body size:
       Openresty error log path:      /var/log/nginx/my-admin-error.log
       Openresty global hsts:         true
       Openresty computed hsts:       true
       Openresty hsts:
       Openresty hsts include subdomains: true
       Openresty hsts max age:        15724800
       Openresty hsts preload:        false
       Openresty image:               dokku/openresty-docker-proxy:0.6.0
       Openresty letsencrypt email:
       Openresty letsencrypt server:  https://acme-v02.api.letsencrypt.org/directory
       Openresty proxy buffer size:   4096
       Openresty proxy buffering:     on
       Openresty proxy buffers:       8 4096
       Openresty proxy busy buffers size: 8192
       Openresty proxy read timeout:  60s
       Openresty x forwarded for value: $remote_addr
       Openresty x forwarded port value: $server_port
       Openresty x forwarded proto value: $scheme
       Openresty x forwarded ssl:
=====> my-admin ports information
       Ports map:                     http:80:1337 https:443:1337
       Ports map detected:            https:1337:1337
=====> my-admin proxy information
       Proxy enabled:                 true
       Proxy type:                    nginx
=====> my-admin ps information
       Deployed:                      true
       Processes:                     1
       Ps can scale:                  true
       Ps computed procfile path:     Procfile
       Ps global procfile path:       Procfile
       Ps procfile path:
       Ps restart policy:             on-failure:10
       Restore:                       true
       Running:                       true
       Status web 1:                  running (CID: 97bc1cb4384)
=====> my-admin registry information
       Registry computed image repo:      myapp/lungo-prod
       Registry computed push on release: true
       Registry computed server:          ghcr.io/
       Registry global push on release:
       Registry global server:
       Registry image repo:               myapp/lungo-prod
       Registry push on release:          true
       Registry server:                   ghcr.io
       Registry tag version:
=====> my-admin resource information
=====> my-admin scheduler information
       Scheduler computed selected:   docker-local
       Scheduler global selected:     docker-local
       Scheduler selected:
=====> my-admin scheduler-docker-local information
       Scheduler docker local disable chown:
       Scheduler docker local init process: true
       Scheduler docker local parallel schedule count:
=====> my-admin storage information
       Storage build mounts:
       Storage deploy mounts:         -v /var/lib/dokku/data/storage/my-admin-uploads:/opt/app/public/uploads
       Storage run mounts:            -v /var/lib/dokku/data/storage/my-admin-uploads:/opt/app/public/uploads
=====> my-admin traefik information
       Traefik api enabled:           false
       Traefik api vhost:             traefik.dokku.me
       Traefik basic auth password:
       Traefik basic auth username:
       Traefik dashboard enabled:     false
       Traefik image:                 traefik:v2.10
       Traefik letsencrypt email:
       Traefik letsencrypt server:    https://acme-v02.api.letsencrypt.org/directory
       Traefik log level:             ERROR
       Traefik priority:

How (deb/make) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:
Using the instructions on the website. Nothing fancy.
Additional information
My guess would be that the deploy action is invalid. If I look in https://github.com/dokku/ci-docker-image/blob/master/bin/dokku-deploy#L100 I see that the cli dokku isn't being called.
So, maybe I am missing somehting in my configuration, or, should the line read as follows?
if [ -n "$DEPLOY_DOCKER_IMAGE" ]; then
  log-info "Deploying image to Dokku Host"
  ssh "$ssh_remote" -- dokku git:from-image "$remote_app_name" "$DEPLOY_DOCKER_IMAGE" "$DEPLOY_USER_NAME" "$DEPLOY_USER_EMAIL"
else
  # OMITTED
fi

Can you paste the contents of the following file from your server? It should only contain public keys and is therefore fine to paste as is.
/home/dokku/.ssh/authorized_keys

Can you paste the contents of the following file from your server? It should only contain public keys and is therefore fine to paste as is.
/home/dokku/.ssh/authorized_keys

Sure:
command="FINGERPRINT=SHA256:jHj9xDySI7uBCM2T+eMfOPuEQHfULAooStr+fQ5fCoE NAME=\"admin\" `cat /home/dokku/.sshcommand` $SSH_ORIGINAL_COMMAND",no-agent-forwarding,no-user-rc,no-X11-forwarding,no-port-forwarding ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE1wsLVxhl28w4geftcdUZwvjGAHu6sGP3GGnDcpW2xp<EMAIL_ADDRESS>command="FINGERPRINT=SHA256:O5+lHW8LXKYQuzvmXeDhKqKFBl3e2bcRGE7MUB7tM/4 NAME=\"michael\" `cat /home/dokku/.sshcommand` $SSH_ORIGINAL_COMMAND",no-agent-forwarding,no-user-rc,no-X11-forwarding,no-port-forwarding ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGDNTacMUFmrWSbbgAal+VpioBEBiNaxtxE2wTwQ/rQR github
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDmt1/ERqRJfMNyYBJFYtgXaboIhbY3GWIsMAgC/Pfxr7EZ1R0TZ7OylIVlU9eZOiuOTlNg6YiDC8UhhCOyur5FY16aw4BaVSfCsyEvEBhUyqQ4fVVjfXh6srMy6pdXtLw3NADLXss7c2m8XEgkHFXbu8IGUbY3vNnSLpIyBNrG6UudaDRXP4HXG4RFql8MUDTT+r0Q07C40EZJWxKYyFRGNJ3bMriqFvINFH0zgWkjn3DQ48GjmI/jW4uJxbfBy8kJ2dB89LS1dnHImjHxmS5DX3LiapFwV08bfzPxDpDkg+oXutIKnOmbEoUDg2VBiXoEpp2yB76sPD1iyAWd5xf0Rgno8PhvKbNfXDMBxcJUVU9iG2y8vGprY0VlclB0XpvJWiqEjaPRxYdyrHxQkeJ3LlDNYPY/1kd2dqX+k2L7OLcahtKNpXX4Vn373cxYCnzG2SJT8ZeRl9zPu7fThtpKywU0XfXJRXO2NAGUDDEBdH+lgVCSrU3IVWWYYFlc2JmHJvg6298RsS/CETcVDNcJ8eMPd72c6bNMCYCrVK4WScAQQCPM4zRtoS9P+/H9jNhq3hj8YV9D/QG1szOE31BVwikyUQL55eF16KRSfY0dtTf7mrJ/2kJjrKXOthb/0g5OshqXzhZbDLzMHMi4Tjo6HS0ayvAq2wjpdtvmj4rpPw==<EMAIL_ADDRESS>=

Those last two lines are bad entries (one is just an equals sign). Remove them. That file is meant to be managed by the ssh-keys plugin. Please see the user management docs on the dokku site for more details.

Those last two lines are bad entries (one is just an equals sign). Remove them. That file is meant to be managed by the ssh-keys plugin. Please see the user management docs on the dokku site for more details.

Hero! I indeed messed up here. Thanks for your fast response and help.


Desktop E2E Test: MM-T1288 Keyboard Shortcuts in Settings Page

This ticket is to create a test script to automate some testing for the Desktop App.
The testing framework makes use of a few frameworks to successfully automate Desktop App tasks:

Playwright: An E2E testing framework provided by Microsoft that currently has “experimental” support for Electron. Using Playwright, you can access individual BrowserWindows or BrowserViews and automate most functionality provided within those windows. You can also get access to the main Electron process and verify/check functionality in the main process. See here for docs: https://playwright.dev/docs/api/class-electron
RobotJS: A Node.JS Desktop Automation tool that allows us to arbitrary emulate mouse and keyboard inputs. This is used for automation around Electron keyboard shortcuts and the menu bar. It is useful to cover cases that aren’t automatable directly through Playwright (since the menu is not an HTML control). See here for docs: http://robotjs.io/docs/syntax

The test should be written in the file e2e/specs/settings/keyboard_shortcuts.js
If the file doesn’t exist, feel free to create it.
If you’re looking for examples, see these PRs:
https://github.com/mattermost/desktop/pull/1841
https://github.com/mattermost/desktop/pull/1843
https://github.com/mattermost/desktop/pull/1846
The test script for this ticket is as follows:

Open the Settings Window
Type something in the spellchecker languages text box.
Use CtrlC and then CtrlA and then CtrlV. Verify that the text reappears.

Expected:
See above.

If you're interested please comment here and come join our "Contributors" community channel on our daily build server, where you can discuss questions with community members and the Mattermost core team. For technical advice or questions, please  join our "Developers" community channel.
New contributors please see our Developer's Guide.
JIRA: https://mattermost.atlassian.net/browse/MM-39673

I would like to take this up. Just a short query @devinbinnie. Is this the spellchecker languages text box in settings (At the bottom of the image the last button)?

@shadowshot-x You'll want to use the Settings window provided by the Desktop App, so if you go to 3-dot menu > File > Settings (or top menu > Mattermost > Preferences on Mac) you'll see where the spellchecker box is.
The test here is to validate that keyboard shortcuts like Cut/Copy/Paste are working correctly in that window.

Oh Okay. Found it on Mac, I guess I will have to write separate tests using if-else for darwin and other processes. I will take this up.

@shadowshot-x Are you still interested in working on this?

Sorry @devinbinnie, cant find time as of now :(


linux/common-config: restrict access to dmesg

Description of changes
SECURITY_DMESG_RESTRICT is enabled by default by a lot of other distributions for a quite a while now, NixOS is a bit of an outlier. The main justification to enable it is that kernel log might leak kernel pointers which can then be used by exploits to defeat KASLR (NixOS also enables kernel.kptr_restrict by default since 2013).

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Sounds good in principle.
common-config.nix advises:
# WARNING/NOTE: whenever you want to add an option here you need to either
# * mark it as an optional one with `option`,
# * or make sure it works for all the versions in nixpkgs,
# * or check for which kernel versions it will work (using kernel
#   changelog, google or whatever) and mark it with `whenOlder` or
#   `whenAtLeast`.
# Then do test your change by building all the kernels (or at least
# their configs) in Nixpkgs or else you will guarantee lots and lots
# of pain to users trying to switch to an older kernel because of some
# hardware problems with a new one.

Thoughts on how this applies to this PR?

It is fine, CONFIG_SECURITY_DMESG_RESTRICT exists since 2.6.37 and the oldest supported kernel in nixpkgs is 4.19.

Also, should this PR target staging rather than master, in light of rebuild-linux: 2501-5000?

I

There is a merge conflict, feel free to ping me when resolved to merge.
Thanks for contributing :3

Thanks for the conflict resolution @JohnRTitor :heart:

I agree with this change, but doesn't this mean we also need to do something about journalctl -k as an unprivileged user?

AFAIK it is already the case unless you are in the adm, systemd-journal or wheel group.


submodularize and update gyp for libsodium 1.0.10

this basically deletes the libsodium dep and turns it into a git submodule
we can checkout the upstream as part of configure step (if submodule is already checked out locally, git ignores that update command)
However, the PR is incomplete, since i think some stuff has to be changed in sodium.cc before the module will load symbols correctly
Maybe we could checkout a release version instead? Like libsodium 1.0.9? I could change this PR if there's any interest in that and wonder whether you think this sort of thing would make it easier to manage upstream version changes

oh cool, the 1.0.10 release is here :)
and woah this diff is like impossible to read on github, sorry about that..

Thank you for the contribution. I've never favored this approach because changes to Libsodium that cause any compatibility issues will break node sodium. With a fixed version we can keep it working while we update, change, test newer versions of libsodium.

I've never favored this approach because changes to Libsodium that cause any compatibility issues will break node sodium

that makes sense. btw i was inspired by your work on defines.js and basically took that idea slightly further using a generic gyp.am file, here's the basic idea, which I learned from node-sodium:

automatic gyp generation of c sources, cflags, xcode_settings, defines

the rationale for such a thing would be to ease node binding compiler upkeep for upstream changes
an example gyp.am:
{
    'targets': [
        {
            'target_name': '{LIB}',
            'variables': { 'target_arch%': '{ARCH}' },
            'type': 'static_library',
            'dependencies': [],
            'defines': [{DEFINES}],
            'include_dirs': [{INCLUDES}],
            'xcode_settings': { 'OTHER_CFLAGS': [{CFLAGS}], },
            'cflags': [{CFLAGS}],
            'ldflags': [{LDFLAGS}],
            'sources': [{SOURCES}],
        }
    ]
}

an example configure.js based on node-sodium's defines.js:
/**
 * node configure
 * --------------
 *
 * automatic gyp generation of c sources, cflags, xcode_settings, defines
 * some machine specifics available in post ./configure file:
 *   • config.log
 *   • others found in Makefile.am
 *
 */

const readFileSync  = require('fs').readFileSync
const readdirSync   = require('fs').readdirSync
const writeFile     = require('fs').writeFile
const readFile      = require('fs').readFile

ø=[
  'libsodium',
  'libmill'
].map(lib => {
  readFile(`${lib}/config.log`, 'utf8', (err, l) => {
    var f = l.split(/## Output variables. ##/)[1].split(/## confdefs.h. ##/)

    var gyp = readFileSync('gyp.am', 'utf8')
    var gypi = require(`./${lib}.gypi`)
    for (var i in gypi) {
      var I = i.toUpperCase()
      var reg = new RegExp('\\{'+I+'\\}', 'gi')
      gyp = gyp.replace( reg, '{'+I+'}'+', \''+ gypi[i].join('\',\'') + '\'' )
    }

    var s = sources(lib)

    switch (lib) {
      case 'libmill':
        if (process.platform !== 'darwin')
          s[1] += `'-lanl', '-lrt',`
        break
    }

    gyp = gyp.replace(/{LIB}/g, lib)
    gyp = gyp.replace(/{ARCH}/g, require('os').arch())
    gyp = gyp.replace(/{INCLUDES}/g, includes(lib))
    gyp = gyp.replace(/{LDFLAGS}/g, ldflags(f[0]))
    gyp = gyp.replace(/{SOURCES}/g, s[0] )
    gyp = gyp.replace(/{CFLAGS}/g, (s[1] ? s[1] : '') + cflags(f[0]) )
    gyp = gyp.replace(/{DEFINES}/g, defs(f[1]))

    writeFile (`${lib}.gyp`, gyp )
  })
})

function sources (o) {
  var src = readdirSync(o), dir = {}
  if (src.filter(i => i === 'src').length) {
    dir.path = `${o}/src`
    src = readdirSync(dir.path)
  } else {
    dir.path = o
  }
  dir.src = src
  return make(dir)
}

function make (dir) {
  if (dir.src.filter(i => i === 'Makefile.am').length) {
    var file = readFileSync(`${dir.path}/Makefile.am`, 'utf8')
    var subdirs = file.match(/SUBDIRS/)
    if (subdirs) {
      var f = file.match(/\S+$/gm)
      f.shift()
      dir.path += `/${f}`
      file = readFileSync(`${dir.path}/Makefile.am`, 'utf8')
    }
    return readMakefile(file, dir.path)
  }
}

function readMakefile (f, root) {
  var read = 0, sources = '', cflags = '', rd, s
  var srcs = f.split(/la_SOURCES/)
  while (srcs.length - 1) {
    var src = srcs.pop()
    src = src.split(/\n/)
    while (++read) {
      rd = src[read]
      if (rd) {
        rd = rd.match(/(\S.*?)\.c/g)
        if (rd)
          sources += '\'' + root + '/' + rd + '\','
      } else {
        read = -1
      }
    }
  }

  var flags = f.split(/la_CFLAGS/)
  srcs = flags.length-1 ? flags[1].split(/\n/) : flags
  while (++read) {
    rd = srcs[read]
    if (rd) {
      rd = rd.split(/\\/)[0].replace(/^\s+/g,'')
      if (rd)
        cflags += '\'' + rd + '\','
    } else {
      read = -1
    }
  }
  return [sources, cflags]
}

function includes (lib) {
  var ret = '\'' + lib + '\''
  var f = readFileSync(`${lib}/configure.ac`, 'utf8').split(/AC_CONFIG_FILES/)
  if (f.length-1) {
    var line = f[1].match(/(\S.*?)include/gm)
    if (line) {
      ret += `,'${lib}/${line[0]}'`
      ret += `,'${lib}/${line[0]}/${lib.slice(3)}'`
    }
  }
  return ret
}

function cflags (o) {
  return '\'' + o.split('\n')
    .filter(i => /^CFLAGS/.test(i) || /^CPPFLAGS/.test(i) )
    .map(i => i.split("\'")[1].replace(/^\s+/g,''))
    .filter(i => i)
    .join('\',\'') + '\''
}

function ldflags (o) {
  return '\'' + o.split('\n')
    .filter(i => /^LDFLAGS/.test(i))
    .map(i => i.split("\'")[1].replace(/^\s+/g,''))
    .filter(i => i)
    .join('\',\'') + '\''
}

function defs (o) {
  return '\'' + o.split('\n')
    .filter(i => /define/.test(i))
    .map(i => i.slice(8).split(/\s/g).join('='))
    .filter(i => i)
    .join('\',\'') + '\''
}

Yesterday I began exploring the approach in experimental network library for distributed encrypted msgs:
https://github.com/reqshark/mill

With a fixed version we can keep it working while we update, change, test newer versions of libsodium

when you say "fixed version" are you talking about node-sodium or libsodium?

I mean lib sodium's version being fixed in time, and when we're sure a new version does not break anything move node-sodium to the new libsodium library.
The gyp generation tool is great. thanks for expanding upon it.

Oh I might have now stumbled upon the reason I couldn't finish this PR earlier...
some files contain opcodes like for symbol crypto_scalarmult_curve25519_sandy2x_ladder_base.
these sources are built from assembly files that end in .S, but gyp is able to compile them just fine, so I think it is now a matter of:

listing any .s .S files in gyp sources array, and
passing the AVX cflags, like -mavx or -mavx2 accordingly, I have to put them in xcode_settings: OTHER_CFLAGS on osx, and plain cflags on linux (cflags of course doesnt work on osx)

Oh by the way, lets close this PR
Ultimately while I think leaning on configure/make is a good way to build deps in a cross platform way as the library might have intended, it's still somewhat gnarly as a matter of npm installer business... So taking this stuff further might have been a step in the wrong direction. Also I'm not sure that npm supports git submodule inits in a clean way, or least that sort of thing seems to land outside of its preview as a subroutine of the install step


[HOTFIX] Resolve merge conflict preventing 1.15 build

Also fix maintenance command solr_prepare to test proper names of resource SOLR fields.

Build finished.

@alvacouch and @aphelionz This needs to get merged before I can make my hotfix for this RC.

@pkdash merged!


chore: deprecated kcl-openapi tool and just used as a normal go package

1. Does this PR affect any open issues?(Y/N) and add issue references (e.g. "fix #123", "re #123".):

[x] N
[ ] Y

chore: deprecated kcl-openapi tool and just used as a normal go package

Pull Request Test Coverage Report for Build<PHONE_NUMBER>1
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 67.774%

Totals

Change from base Build<PHONE_NUMBER>9:
0.0%

Covered Lines:
2305

Relevant Lines:
3401

💛  - Coveralls


return headers from .reply

It'd be really nice to also support mocked headers from the .reply callback return, like this:
  nock(serviceURI)
    .get('/api/route')
    .reply(function() {
      console.log('GOT NOCKED');
      return [200, mockDataObj, mockHeadersObj];
    });

Landed on v5.4.0

nice, ty!


@shopify/shopify-api/dist/auth/session does not exist

According to my local install @shopify/shopify-api/dist/auth/session doesn't exist.
here's the packages I have installed:
"@nestjs-shopify/core": "^3.1.0",
"@nestjs/common": "^10.2.1",
"@shopify/shopify-api": "^7.5.2",

I'm having issues with pretty much everything about creating session storage as written up in the readme file:
Type 'typeof MyRedisSessionStorage' is missing the following properties from type 'SessionStorage': storeSession, loadSession, deleteSession, deleteSessions, findSessionsByShopts(2739)
core.interfaces.d.ts(6, 5): The expected type comes from property 'sessionStorage' which is declared here on type 'ShopifyCoreOptions<any> & Partial<{}>'

This is my app.module.ts
import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { PrismaModule } from './prisma/prisma.module';
import { UsersModule } from './users/users.module';
import { HealthModule } from './health/health.module';
import { OrgsModule } from './orgs/orgs.module';
import { ShopifyCoreModule } from '@nestjs-shopify/core';
import { ApiVersion } from '@shopify/shopify-api';
import { MyRedisSessionStorage } from './my-redis-session-storage';

@Module({
  imports: [
    ShopifyCoreModule.forRoot({
      apiKey: process.env.SHOPIFY_API_KEY,
      apiSecretKey: process.env.SHOPIFY_API_SECRET,
      apiVersion: ApiVersion.Unstable,
      hostName: 'localhost:8081',
      isEmbeddedApp: true,
      scopes: ['test_scope'],
      sessionStorage: MyRedisSessionStorage,
    }),
    PrismaModule,
    UsersModule,
    HealthModule,
    OrgsModule,
  ],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

All I'm trying to do is use auth and web hooks.

@blazestudios23 Could you help me nail this down? 🙂
I would like to see the contents of your my-redis-session-storage.ts file. Curious to see which interface its implementing or which class its extending.

@blazestudios23 Actually, see commit a3ce7e5. The README snippet was outdated. Thanks for bringing this up! Can you let me know if you get it to work by following the new snippet!

I'm closing this due to inactivity. If you still find any problems, feel free to comment and I'll reopen!


Local setup 3

➜  teiserver_tei git:(master) ✗ mix create

Compiling 398 files (.ex)
Generated central app
** (Mix) The task "create" could not be found

➜  teiserver_tei git:(master) ✗ mix.create

zsh: command not found: mix.create

Guessing it should've been mix ecto.create there

Yes, it should have. I'll change it and push. Thanks :)


chore(cta): introduce dds-video-cta-container

Related Ticket(s)
This change sets the stage for fixing #4028.
Description
Introduces <dds-video-cta-container>, that takes care of populating video caption data as well as handles click action on video type of <dds-text-cta>, <dds-card-cta> and <dds-feature-cta>.
Also removes <dds-cta-container>. For rendering CTA, <dds-text-cta>, <dds-card-cta> and <dds-feature-cta> can be used directly.
Changelog
New

<dds-video-cta-container>, that takes care of populating video caption data as well as handles click action on video type of <dds-text-cta>, <dds-card-cta> and <dds-feature-cta>.
dds-cta-request-video-data custom event, fired from <dds-text-cta>, <dds-card-cta> and <dds-feature-cta>, so <dds-video-cta-container> can populate video caption data.

Changed

A CodeSandbox example of video CTA, updated with the new API.
The documentation of video CTA, updated with the new API.
Replaced type attribute with cta-type for <dds-text-cta>, <dds-card-cta> and <dds-feature-cta>, to fix the conflict with native <a>.

Removed

<dds-cta-container> for the sake of <dds-text-cta>, <dds-card-cta> and <dds-feature-cta>.
The feature of rendering video thumbnail in <dds-feature-cta>.

Looks good @asudoh! One thing though, looks like card and feature card video versions aren't rendering the video caption.

Great catch @kennylam - Fixed.


Other Output Formats doesn't work?

The output formats described in the docs (https://momentjs.com/docs/#/get-set/day/) doesn't work with node-red-contrib-moment?
For example:

Day of Week
ISO Day of Week

How do I get the last Friday as an output?

Have you tried the Humanizer node instead of the main moment node? I think that should do the job of giving you dates such as "Last Friday".

hm...
i thought i could use node-red-contrib-moment to calculate (output) the last friday from the input. so for example i enter today's date 14.08.2019 and get the answer date 09.08.2019 (the date of last friday)

Ah, I see. I'm afraid that you can currently only add/subtract in a very simple way.
I've always had a thought to create another node to the repo that let you do more complex calculations but you are the first person to raise an actual use-case :)
Good news though, this is a fairly common requirement and the easiest thing to do is to add a function node. The following gives some examples of JavaScript code that will do the calculation for you.
https://stackoverflow.com/questions/30322894/how-to-get-timestamp-of-last-friday-using-javascript

Added to the list for a future release.
I will add a calculation node that lets you work out things like "last Friday", "last day of the year" "days until" "Is the date a weekday/weekend/special day" and so on.


Formatting

makes sure to use const and let on as much as possible

Fixes JSDoc

Removes YUI Compressor (incompatible with es6)

Uses Terser to compress the js file (to support es6)

Updates README command to clean, build and compress the files

Removes the grunt task "grunt-closure-compiler" as it doesn't seem to be used (it was using YUI instead)

Removes the JSDoc documentation (compiled docs) as it seems that was never there.

@harrymoore @uzquiano @cloudcms
Sorry I needed to recreate this after running the tests

Hi Juan,
I'd like to get this merged in the near term.  You put a lot of work in here and it looks really good.  We hadn't updated the JS driver's build dependencies and I appreciate what you've done to improve that.
My sense is that we could merge this PR and then further address the NPM audit you mentioned in the other ticket.
Have you been using this version in your code?  Or building from master?  I'm just curious if the build product here passed tests or if you've tried it out and it seemed to work relatively well.  We'll have to run it through the full test suite but based on what I see, I don't see why it wouldn't pass in short order.
Cheers,
Michael

Hi,
I couldn't really run all the tests.
I followed the instructions but it has errors trying to connect to test.cloudcms.com and starting a server, so I changed the server configuration to localhost and the in has issues with the proxy server

Pulled the latest code from Master.

Changed the proxy server for the tests from test.cloudcms.com to test.cloudcms.net since the .com domain is not responding.

Still having issues connecting to the proxy server.

I was able to use Chrome's Local Override feature to load the changes onto CloudCMS and everything seems to be loading just fine.


Not working with latest home assistant

Fails to validate config with error.
Please help.
Thank you

Configuration invalid
CHECK CONFIGURATION
Platform error sensor.inkbird - Integration 'inkbird' not found.

Home Assistant 2021.6.3

2021-06-08 22:33:20 ERROR (SyncWorker_6) [homeassistant.loader] The custom integration 'inkbird' does not have a valid version key (None) in the manifest file and was blocked from loading. See https://developers.home-assistant.io/blog/2021/01/29/custom-integration-changes#versions for more details

To get it working had to ad a version line to the manifest.json

Can you provide more information? im having the same issue right now and I'm still unable to make it work


remove overwolf due to deprecation

[ ] Have you signed the Contributor License Agreement?
[ ] Have you checked that there aren't other open pull requests for the same manifest update/change?
[ ] Have you validated your manifest locally with winget validate --manifest <path>?
[ ] Have you tested your manifest locally with winget install --manifest <path>?
[ ] Does your manifest conform to the 1.2 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

details of deprecation : when installed and run, the app will show that the version of the app is no longer supported and encourages people to migrate to the new app, wich I have problems with the sha-265 validation