WIP: Render static resources #19

Codecov Report

Merging #23 into master will decrease coverage by 2.48%.
The diff coverage is 33.33%.

@@            Coverage Diff             @@
##           master      #23      +/-   ##
==========================================
- Coverage   86.07%   83.59%   -2.49%
==========================================
  Files           7        7
  Lines         316      323       +7
  Branches       40       39       -1
==========================================
- Hits          272      270       -2
- Misses         44       53       +9

Impacted Files
Coverage Δ

src/RequestContext.js
100% <ø> (ø)
:arrow_up:

src/HelixProject.js
98.37% <ø> (ø)
:arrow_up:

src/utils.js
44.44% <33.33%> (-55.56%)
:arrow_down:

src/HelixServer.js
54.66% <33.33%> (+0.71%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ebca91c...cdb8bc7. Read the comment docs.


Update unit test maven plugin version

Purpose
Update unit test maven plugin version to fix sample running issues
Security checks

Followed secure coding standards in http://wso2.com/technical-reports/wso2-secure-engineering-guidelines? yes
Ran FindSecurityBugs plugin and verified report? yes
Confirmed that this PR doesn't commit any keys, passwords, tokens, usernames, or other secrets? yes

Test environment

JDK 8 Mac OS

Closing this PR, since we have updated the version to 5.2.27, https://github.com/wso2/mule-conversion-samples/blob/master/examples/testing-apikit-with-munit/testing-apikit-with-munit-sample/pom.xml#L109


Performance after refactor

Hi Alister,
I've noticed that after the refactor in commit 05ac4282 the parsing performance dropped by orders of magnitude (from 0.4 s to 74 s for a rather big data file).
After line profiling, the clear offender are the very atomic add_line and add_newline methods in TextBuffer which add single lines to the buffer to be read into the DataFrames.
Bypassing the TextBuffer object altogether by replacing the parser._parse_loop_data method with its pre-refactor equivalent (i.e., keeping the text buffer a plain local variable) restores the performance, with the actual pd.read_csv requiring the most time. This is consistent with my previous experience with parsing complex (non-.star) text formats that putting too much abstraction around performance-critical parts might not be a good idea.
For now I'm going to stick to the old version, please let me know if I can be of any help.
Robert

As a side note, a marginally faster (arguably more elgant way) would be to use the StringIO as buffer itself, like this:
    def _parse_loop_data(self) -> pd.DataFrame:

        with StringIO() as buffer:
            while self.crawler.current_line_number <= self.n_lines:
                if self.crawler.current_line.startswith('data_'):
                    break
                buffer.write(self.crawler.current_line + '\n')
                self.crawler.increment_line_number()

            buffer.seek(0)
            df = pd.read_csv(buffer, delim_whitespace=True, header=None, comment='#')

        return df

Hey Robert,
Big thanks for the heads up and for taking the time to dig a little deeper, it's really much appreciated!
This is frustrating, I was quite happy with the TextBuffer and TextCrawler objects :laughing: - I like your StringIO implementation though, I wasn't exactly sure what was in memory and what was being written to disk with that which is why I went down the 'create your own' object route, should've spent time reading up
This highlights a few things

tests should be added for performance to enable detection of this type of problem during testing
the TextBuffer object should be scrapped at least for loop blocks, probably for everything (or we end up with multiple ways of doing things within the same object which feels bleh)

Do you have any experience with testing for performance related things like this? It's not something I've had to do before
Is this performance hit specifically an implementation issue?
    def add_newline(self):
        self.buffer += '\n'

    def add_line(self, line):
        self.buffer += f'{line}'
        self.add_newline()

Would removing the add_newline method and replacing add_line with
    def add_line(self, line):
        self.buffer = f'{self.buffer}{line}\n'

this seems to indicate that f strings should be faster? The other methods such as add comment could be refactored in a similar way
I'd really appreciate some help knocking this back into shape if you have time? I'm in PhD writing mode at the minute and although sometimes these things are a welcome distraction I should definitely try to focus, at least during the working day :octopus:

I've also yanked 0.4.0 from PyPi to stop others from running into the same issue, this means that the API in the readme is no longer valid... I'll hope nobody complains for now and hopefully if anyone has problems they'll open an issue

Hi Alister,
I've tried your suggestion already earlier (as first sanity check). Bypassing the add_newline function roughly halves the runtime, but the main issue is not solved (indeed f-strings are fast, though). My guess is that the very granular calls require making copies at various points, but I'm not deep enough into these things... TBH, my suggesting is to double-check whether pure container classes like TextBuffer and TextCrawler are really needed. It's a lot of boilerplate code without improving readability a lot, especially for such a focused tool. I can look into it a bit, but also will have to make sure not to get carried away too much from my actual data analysis (rabbit hole).
Anyway - good luck with thesis writing!

Really interesting, looks like a local variable is indeed the way to go then, the more you know!
Thanks, and I hope your data aren't giving you too much trouble!

@brisvag implemented a nice solution using deque which I didn't know about and which is faster than 0.3.3, really nice!
This let's us keep the crawler/buffer objects and the performance


util-linux: 2.39.3 -> 2.40.1

Description of changes
CVE-2024-28085 (low priority since NixOS doesn't make wall setgid by default).
Release notes

2.40
2.40.1

Don't see anything looking like a breaking change.
Things done

Built on platform(s)

[ ] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Security backport for stable: https://github.com/NixOS/nixpkgs/pull/309808

Is the liblastlog2 and pam_lastlog2 dependency on sqlite always necessary or only for with pamSupport?

If I'm reading correctly, it's possible to build liblastlog2 without building pam_lastlog2.  We could add an option to disable it, but I don't know how to decide whether that's worth it.

For reference, the parallel build is a bit flaky now:
/nix/store/bgcaxhhxswzvmxjbbgvvaximm5hwghz1-binutils-2.41/bin/ld: cannot find -llastlog2: No such file or directory
collect2: error: ld returned 1 exit status
libtool:   error: error: relink 'pam_lastlog2.la' with the above command before installing it

Time to switch to Meson?

It feels late for 24.05.  But fortunately adding meson+ninja wouldn't create an eval-time cycle here.

Reported, also.

And (aarch64-)darwin build regressed on this update with a different error now: https://hydra.nixos.org/build/260146685
libuuid/src/uuid_time.c:89:87: error: aliases are not supported on darwin

Oh, that darwin issue would even become a blocker for nixpkgs-unstable channel :-/

There's also a bug report here: https://github.com/util-linux/util-linux/issues/3011#issuecomment-2119153020

Man, x86_64-darwin sports yet another error:
libsmartcols/src/filter.c:194:14: error: call to undeclared function 'fmemopen'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        fltr->src = fmemopen((void *) str, strlen(str), "r");
                    ^

(IIRC older macOS doesn't provide this function yet)

Quite a bad luck with this release, and in the last moments before nixpkgs "freeze".

I really wish OfBorg was capable of getting passed LLVM on staging for Darwin…

If it was so easy, we perhaps wouldn't even need staging*.

It gets most of the way through — we just need to up the timeout a bit, and then I think OfBorg might even be less overloaded, because it wouldn't be wasting so much of its time trying the same futile LLVM builds over and over.

Maybe we should just roll back to util-linux 2.39.x for now and keep nixpkgs 24.05 that way?  (darwin now, linux on the next rebuild)

Sounds fine.

We should probably cherry-pick #309808 in that case.

Perhaps 2.39.4 instead, as it sounds minimal and should contain also that fix
https://github.com/util-linux/util-linux/blob/v2.39.4/Documentation/releases/v2.39.4-ReleaseNotes

Ah yes, that sounds good.  I missed that.

For reference, 2.40.1 also broke its static musl build:
https://hydra.nixos.org/build/259722977

That's likely easily fixed by just disabling the PAM plugin on static builds.  But is there any point fixing that, even on staging, or will we revert this there as well?

Right, not now.  I've given up on fixing 2.40.x for nixpkgs 24.05.

I mean what about staging?  i.e. post 24.04

I guess it'd just cause merge conflicts, since the gradual reversion on staging-next will propagate to staging.
So for future reference when we come back to 2.40, here's the diff for static:
diff --git i/pkgs/os-specific/linux/util-linux/default.nix w/pkgs/os-specific/linux/util-linux/default.nix
index 642480b670c7..169f8293ae3e 100644
--- i/pkgs/os-specific/linux/util-linux/default.nix
+++ w/pkgs/os-specific/linux/util-linux/default.nix
@@ -6,6 +6,7 @@
 , ncursesSupport ? true
 , ncurses
 , pamSupport ? true
+, pamLastlogSupport ? !stdenv.hostPlatform.isStatic
 , pam
 , systemdSupport ? lib.meta.availableOn stdenv.hostPlatform systemd
 , systemd
@@ -61,6 +62,7 @@ stdenv.mkDerivation rec {
     "--disable-makeinstall-setuid" "--disable-makeinstall-chown"
     "--disable-su" # provided by shadow
     "--with-tmpfilesdir=${placeholder "out"}/lib/tmpfiles.d"
+    (lib.enableFeature pamLastlogSupport "pam-lastlog2")
     (lib.enableFeature writeSupport "write")
     (lib.enableFeature nlsSupport "nls")
     (lib.withFeature ncursesSupport "ncursesw")

For context: This update also caused a regression with detecting the UUID of some special LUKS devices (maybe only if the LUKS device contains a LVM). I haven't reported this upstream yet, will do later.
Some more context is in https://discourse.nixos.org/t/nixos-24-05-upgrade-no-boot-device/46335, which is quite analogous to what I experienced.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/nixos-24-05-upgrade-no-boot-device/46335/3

bcachefs-tools on master needs https://github.com/util-linux/util-linux/pull/3001 to detect my disks correctly, not a huge issue, but worth noting for future potential issues if that makes it in to nixpkgs-unstable before this does.

We can backport that patch.


Only use UTF-8 strings when POSTing to the server

This occurs in 2 places:

taxii2client._HTTPConnection.post() should use requests.post(data=X) rather than requests.post(json=X). This function is called from taxii2client.Collection.add_objects(). X should be a UTF-8 encoded byte string.

If we use ‘data’ and X is a dict, it will be form-encoded, which we don’t want.
If we use ‘json=X’ and X is a dict, requests will call json.dumps(), which is OK but means we can’t pass dicts containing datetime objects or anything else that isn’t JSON-serializeable (such as _STIXBase objects).
Note that we control this, so we don’t need to do anything to address other types in this function.

taxii2client.Collection.add_objects(X) should prefer to accept a Unicode string, which it encodes as UTF-8 before passing to requests (urllib3, which requests uses internally, wants a bytes-like object).

This string should be a valid STIX 2 bundle, but we don’t want to do anything in the TAXII library to validate this. This will improve performance compared to validating ourselves. We prefer to receive this from somewhere that has already done the validation (python-stix2). The server should do its own validation, and we can catch errors and return them to the client, but we don’t want to try to validate ourselves ahead of time.
If X is already a bytestring, we pass it unmodified. We shouldn’t try to detect the encoding, but be aware that servers might reject the content if it’s not in UTF-8.
If X is a dict, we call json.dumps(), which returns a Unicode string, that we then encode as UTF-8. We can do some trivial conversions in a custom JSON encoder, like detecting datetimes and converting to isoformat(), but the TAXII client shouldn’t have any knowledge of STIX 2 when doing this encoding (which is part of why we prefer having a string that has already been converted from a dict to JSON.

Related to https://github.com/oasis-open/cti-python-stix2/issues/134.

On the third bullet of taxii2client.Collection.add_objects(X):
I don't like the custom JSONEncoder and isoformat() idea because we might introduce problems when converting data with different restrictions on the precision after the seconds. I could do json.dumps(bundle, encoding='utf-8') when isinstance(bundle, dict) is True

I'm fine with not supporting datetimes (which is the primary use case for a custom JSONEncoder). datetime objects are the most straightforward data type which is not JSON serializable by default, but has a pretty straightforward encoding anyway (isoformat). That doesn't mean we should burden cti-taxii-client with this complexity though. As we've said elsewhere, when coming from stix2 objects, the TAXIICollectionSink will handle serializing to a string. Other uses of the TAXII client outside python-stix2 will likely involve building dictionaries from scratch and/or parsing existing STIX 2 and making minor changes; in either case, it's not unreasonable to expect users of cti-taxii-client to handle proper JSON encoding for their STIX content.


matchLabelKeys in topologySpreadConstraints doesn't not print anything in kubectl describe

What happened?
I used kubectl describe on a pod with topologySpreadConstraints's matchLabelKeys applied. I have the featuregate MatchLabelKeysInPodTopologySpread applied
What did you expect to happen?
I expect some printed output that indicates that the config has been applied. I expect the code needs to be updated here https://github.com/kubernetes/kubernetes/blob/23a4920d83559ba803629512c8a02630e5fdc0a6/staging/src/k8s.io/kubectl/pkg/describe/describe.go#L895
How can we reproduce it (as minimally and precisely as possible)?
Use kubectl describe on a pod with topologySpreadConstraints's matchLabelKeys applied
Anything else we need to know?
No response
Kubernetes version
1.25
Cloud provider
k3s baremetal
OS version
No response
Install tools
No response
Container runtime (CRI) and version (if applicable)
No response
Related plugins (CNI, CSI, ...) and versions (if applicable)
No response

/assign

@liboz after investigating the issue, I noticed that matchlabelskeys are merged in labelselector(by ANDing) and we are already showing labelselector in describe command. As far as I understand, if matchlabelkeys is applied for your resource, it should be seen in Topology Spread Constraints field.

Here's a reproduceable test on GKE (I originally was using k3s):

Created a cluster on GKE with this command: gcloud container clusters create test-cluster --cluster-version=1.25 --release-channel=rapid --zone=us-central1-a --machine-type=e2-small --num-nodes=2  --enable-kubernetes-alpha --no-enable-autorepair --no-enable-autoupgrade
Run kubectl apply -f test.yaml where the contents of test.yaml are:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ubuntu
  labels:
    app: ubuntu
spec:
  replicas: 2
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: ubuntu
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: ubuntu
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          matchLabelKeys:
            - app
            - pod-template-hash
      containers:
        - name: ubuntu
          image: ubuntu
          command:
            - "sleep"
            - "604801"
          resources:
            requests:
              cpu: 50m
              memory: 50Mi
            limits:
              cpu: 50m
              memory: 50Mi

Run kubectl describe pod on one of the pods that get created

I get output like this:
Node-Selectors:               <none>
Tolerations:                  node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints:  kubernetes.io/hostname:DoNotSchedule when max skew 1 is exceeded

As you can see the parts about app, pod-template-hash do not display for me. The result of my kubectl version command is
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"windows/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0-gke.1100", GitCommit:"eac9e5e684229c0f820abb108048cff3b91440d4", GitTreeState:"clean", BuildDate:"2022-09-05T09:29:20Z", GoVersion:"go1.19 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}

so I'm on the right version for kubectl I believe.
Are you seeing something different?

I was able to confirm in that test that the pods do get scheduled on different nodes (as their kubernetes.io/hostname are different) so I believe the feature is working as intended.  A rolling update also didn't result in pods all in a single node, which is further evidence it's working.

Thanks @liboz. I tried this on local-cluster-up.sh(with alphafeatures are all enabled), but still matchLabelKeys are empty. For kubectl describe part, we can simply add print mechanism but I think we need to ensure prior that;

What is the working mechanism of matchLabelKeys?
Can we simply add print in describe or is there any direct relation between labelselector and matchLabelKeys?
Why above deployment yaml still has matchLabelKeys field empty?

Maybe before fixing the problem, scheduling folks can enlighten us.
/transfer kubernetes
/cli scheduling


How to Debug with pycharm

Hi, very excellent work. Thanks for the code, it's very helpful and instructive!I want to study your code structure, but I don't know how to use pycharm to debug your code.

Hey,
Thanks for your interest, and happy the code helps you :D I am not a Pycharm user myself unfortunately, but I know you could use pdb package to go into debug mode. I encourage you to have a look at #15 to setup pdb. Let me know if you need further help.
Best !

Thank you for your reply. Secondly, I noticed that you used compute FB, using the features and gt of query set, which means that the FB of each segmentation is known, and the network is used to process new images in the end. For features without gt, the segmentation effect will be greatly reduced. How do you solve this problem?

Hi, the FB segmentation is not known in advance in our main method (otherwise there would be no point in developing a method at all). Please have a look at issue #10 , and particularly my answer https://github.com/mboudiaf/RePRI-for-Few-Shot-Segmentation/issues/10#issuecomment-805967381 which should answer your question (if I understood it correctly). If it's still not clear, do not hesitate to ask me further clarification.

Thank you for your reply. I noticed that in the part of RePRI, you used gt_q of query set to get the high-level feature map of the downsampling , and then passed the valid pixels which you said that is not 255 in get_entropies to get d_KL, cond_entropies and marginal, which are all related to valid_pixels , can you tell me why you use gt_q for this part?

Please reference precise lines of the code so that I can help you better (copy the permalink of the line you're referring to in your comment). I'm really not sure I understand the question right now. As you said, there is no gt_q involved in computing the loss functions, only for the valid pixels. gt_q is only used for the following purposes:

Computing the valid_pixels in several parts of the code
Computing the Foreground/Background parameter self.FB_param in the case of the oracle method https://github.com/mboudiaf/RePRI-for-Few-Shot-Segmentation/blob/5900a52e27621c5772193599cfc85a730a82f327/src/classifier.py#L112
Computing the cross-entropy ce_q for metric visualization purposes only https://github.com/mboudiaf/RePRI-for-Few-Shot-Segmentation/blob/5900a52e27621c5772193599cfc85a730a82f327/src/classifier.py#L336


Adding image localization and force resolution

Would it be possible adding a config.xml read by the program where the country code can be set? This would make it possible to download the 'localized' image given by json from something like  "https://www.bing.com/HPImageArchive.aspx?format=js&idx=0&n=1&mkt=" + countryCode instead of the default one. E.g. I am german so it would be countryCode="de-DE".
Also it would be a great addition to force a specific resolution if the appropriate file exists instead of just checking the resolution of the device. This may be useful if the device just has a weird resolution and don't want to download a file with 1920x1080 pixels.
I may come up with a pull request but C# isn't my programming language so it would be a pleasure if you just have an idea for the implementation.
Also I could just change these things in the program and recompile it but it would be easier for those who are not in programming. :wink:
Nice tool though and keep up the good work ofc! :+1:

Yes! I will investigate this further when I get some free time, thank you for the notes!

Also, there are some more or less useful forks:
csmith1210
drasockalypse


Client side design

Working on it. So far finished log in screen

Working on it. So far finished log in screen


Switch to cdylibs

This seems like the right thing to do (cf. Rust RFC 1510) but when I tried it it didn't export my hook function.
$ nm -D examples/readlinkspy/target/debug/libreadlinkspy.so
                 w __cxa_finalize
                 w __gmon_start__
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
                 w _Jv_RegisterClasses
00000000000009d0 T rust_eh_personality
                 U _Unwind_GetDataRelBase
                 U _Unwind_GetIPInfo
                 U _Unwind_GetLanguageSpecificData
                 U _Unwind_GetRegionStart
                 U _Unwind_GetTextRelBase
                 U _Unwind_SetGR
                 U _Unwind_SetIP

it didn't export my hook function

same, but it works with LD_PRELOAD
symbol count, as reported by nm
3139 dylib
68 cdylib

redhook + crate_type = ["cdylib"] requires rust-nightly
[lib]
crate_type = ["dylib"] # works with rust-stable and rust-nightly
# release binary size
# 2.7M rust-stable
# 5.2M rust-nightly

[lib]
crate_type = ["cdylib"] # requires rust-nightly. silently fails with rust-stable
# release binary size
# 450 KB rust-nightly
#  20 KB rust-stable (broken)

compile options for my project
[profile.release]
# https://arusahni.net/blog/2020/03/optimizing-rust-binary-size.html
# https://github.com/johnthagen/min-sized-rust
opt-level = "s" # optimize for binary size
lto = true
codegen-units = 1
panic = "abort" # disable backtraces on panic


Adding of missing wavelets from the matlab list

Missing wavelets are added #165
1d cwt added (not complete)  #48

This is not finished by now.
What do you think? Should i proceed or did i make some design mistakes?

@kwohlfahrt has the most familiarity with the current c extensions, but this looks pretty reasonable to me.  I have not checked the implementation of the algorithm at all at this point, but the organization seems reasonable.
Are there particular design choices you would like feedback on?  From what I can tell, it seems the following properties were added to the Wavelet object:
dwt_possible, cwt_possible, complex_cwt, lower_bound, upper_bound.  Of these, most are specific to continuous wavelets.  It looks like lower_bound and upper_bound will return None for discrete wavelets which seems reasonable.  Maybe someone with more object-oriented programming experience can comment on whether from an API standpoint, it is preferable to having a separate ContinuousWavelet class that adds those properties?

Also, how does this compare to the CWT functionality in scipy?
http://docs.scipy.org/doc/scipy-0.17.0/reference/generated/scipy.signal.cwt.html
or to that in @aaren 's python-based package?:
https://github.com/aaren/wavelets

Thank you @grlee77 for the review.
What are new and not implemented are the wavelet funktions. I plan also to be able to use all DWT-Wavelets for cwt (e.g. db4).
I added now all remaining wavelets except the Meyer wavelet
We can exchange the CWT function and use either scipy or the one from  @aaren
At the moment, the cwt function i implemented is slow

Well, We've got enough requests for it that I think we should add the CWT. So I think this is a good plan in general. I might not have much time to review this until next Monday, but I've had a quick look. If I've missed something let me know please!
I have made a few in-line comments, my main criticism from a design point of view is what @grlee77 said. The new continuous wavelets and the discrete wavelets don't really have much overlap in their fields. Could you see what is possible to separate out? Both in terms of the C struct and the Python/Cython object.

For the meyer wavelet, we neeed ifft. This meens more dependencies. Should i add it?

Why is assert_allclose(np.imag(PSI), np.imag(psi)) failing for linux? It works for me (Windows x64)

In regards to the test failures, all of the tests in your PR are passing on my linux machine.  Looking at the Travis output for the failures, it seems like the results are very close, it is probably just a numerical precision issue causing it to not quite meet the default tolerances in assert_allclose.  Maybe try bumping up rtol and/or atol slightly?
As far as adding FFT code, I assume you want this for use in the C or Cython code?  If so, I think that would require more discussion of the appropriate approach and would probably best fit in a separate issue/PR.  Maybe just omit the Meyer wavelet in the present PR for simplicity.  Alternatively, if you can do the FFTs from python you can just use numpy.fft without needing to add any new dependencies.

It is not so easy to create to seperate structs for cwt and dwt.
The table Wavelet Families and Associated Properties  from http://de.mathworks.com/help/wavelet/ug/wavelet-families-additional-discussion.html?refresh=true
shows this.
Maybe we need three structs in wavelet.h?
/* Wavelet structure holding pointers to filter arrays and property attributes /
typedef struct {
/ Wavelet properties */
int vanishing_moments_psi;
int vanishing_moments_phi;
index_t support_width;
SYMMETRY symmetry;

unsigned int orthogonal:1;
unsigned int biorthogonal:1;
unsigned int compact_support:1;

unsigned int dwt_possible:1;
unsigned int cwt_possible:1;

float lower_bound;
float upper_bound;

char* family_name;
char* short_name;
DiscreteWavelet* discrete_wavelet;
ContinuousWavelet* continous_wavelet;

} Wavelet;
typedef struct {
double* dec_hi_double;  /* highpass decomposition /
double dec_lo_double;  /* lowpass decomposition /
double rec_hi_double;  /* highpass reconstruction /
double rec_lo_double;  /* lowpass reconstruction /
float dec_hi_float;
float* dec_lo_float;
float* rec_hi_float;
float* rec_lo_float;
size_t dec_len;   /* length of decomposition filter */
size_t rec_len;   /* length of reconstruction filter */

} DiscreteWavelet;
typedef struct {
/* Parameters for shan, fbsp, cmor*/
float center_frequency;
float bandwidth_frequency;
unsigned int fbsp_order;

} ContinuousWavelet;

Perhaps, though with the example you showed I would do it the other way around. Instead of:
struct Wavelet {
  unsigned int cwt_possible:1;
  unsigned int dwt_possible:1;
  DiscreteWavelet* discrete_wavelet;
  ContinuousWavelet* continuous_wavelet;
}

struct DiscreteWavelet {...};
struct ContinuousWavelet {...}:

Perhaps the following:
struct BaseWavelet {...};
struct DiscreteWavelet {
  BaseWavelet base;
  ...
};
struct ContinuousWavelet {
  BaseWavelet base;
  ...
};

This makes sure that the type hierarchy is defined by which struct a function takes, and not a boolean flag (xyz_possible) that is checked at runtime. Hopefully that will let the compiler catch more errors, and it'll mirror the inheritance used in the Python classes better too.
For functions that can take either a continuous or discrete wavelet, I suggest the dispatching is done in the Cython layer. Shared data that can use the same code should operate on the base wavelet struct, and if the code is significantly different separate functions are nicer anyway. We could use C11 generic functions, except we need to support MSVC.
PS apologies for the long absence, I'm writing some papers a thesis and applications all at the same time and it's a bit hectic.

PS apologies for the long absence, I'm writing some papers a thesis and applications all at the same time and it's a bit hectic.

No need to apologize (and I'm sure I've been missing in action for longer .....), and good luck with all of those!

What do you think? It is not finished, but a first start.

This looks like a good start.  I would remove dwt_possible and cwt_possible from the BaseWavelet struct in wavelets.c.  I don't think these properties are needed anymore.  It seems like wavelet.is_discrete_wavelet should be sufficient to determine if the wavelet is discrete or not.
Similarly, do you really need separate .dw and .cw properties in the Wavelet object of _pywt.pyx?  It seems much simpler to just keep a .w property and in the few places where the code path is actually different, use is_discrete_wavelet to choose the appropriate branch.

For image support, we need 2D cwt. The cwt convention function has to be enhanced to 2D.

If we want to be similar to matlab than 2d-cwt has to be done by a new function cwtft2
mathworks.com/help/wavelet/ref/cwtft2.html
There is also a new set of wavelet supported:
Wavelet name	Parameters
'morl'	{'Omega0',6;'Sigma',1;'Epsilon',1}
'mexh'	{'p',2;'sigmax',1;'sigmay',1}
'paul'	{'p',4}
'dog'	{'alpha',1.25}
'cauchy'	{'alpha','pi/6';'sigma',1;'L',4;'M',4}
'escauchy'	{'alpha','pi/6';'sigma',1;'L',4;'M',4}
'gaus'	{'p',1;'sigmax',1;'sigmay',1}
'wheel'	{'sigma',2}
'fan'	{'Omega0X',5.336;'Sigma',1;'Epsilon',1;'J',6.5}
'pethat'	None
'dogpow'	{'alpha',1.25;'p',2}
'esmorl'	{'Omega0',6;'Sigma',1;'Epsilon',1}
'esmexh'	{'Sigma',1;'Epsilon',0.5}
'gaus2'	{'p',1;'sigmax',1;'sigmay',1}
'gaus3'	{'A',1;'B',1;'p',1;'sigmax',1;'sigmay',1}
'isodog'	{'alpha',1.25,1.25}
'dog2'	{'alpha',1.25,1.25}
'isomorl'	{'Omega0',6;'Sigma',1}
'rmorl'	{'Omega0',6;'Sigma',1;'Epsilon',1}
'endstop1'	{'Omega0',6}
'endstop2'	{'Omega0',6;'Sigma',1}
'gabmexh'	{'Omega0',5.336;'Epsilon',1}
'sinc'	{'Ax',1;'Ay',1;'p',1;'Omega0X',0;'Omega0Y',0}

So, what is missing for this request? Should we add 2D?
We need no think about the convolution. We can use scipy.signal.convolve for 2D-Convolution as it is used by scikit-image. So if we want 2d-cwt we should use scipy.signal. Is this ok?

I think it will be simpler to make 2D a separate, future PR.

I almost finished now.
[coef, freq]=pywt.cwt(data),np.arange(1,11),'db3',0.001)
as well as
[coef, freq]=pywt.cwt(data),np.arange(1,11),'gaus1',0.001)
is now possible.
The output of cwt is exactly the same as the matlab output.

Why is appveyor for python 3.4 failing?
[00:02:57] alloc: invalid block: 03C1DE18: 40 3
[00:03:00] Command exited with code -2147483645

Anyone has a idea?

Why is appveyor for python 3.4 failing?

This is very likely related to the changes to the build system made in gh-161. Note that the tests don't seem to be run on Appveyor for Python 2.7 and 3.5, so it's not just 3.4 that fails.

This PR is massive. Nice going @holgern

Question about the API of the functions added. Do you like the Matlab style, and do we want/need to keep following it in all cases?
Something like cwt(data, scales, wavelet, sampling_period=1.) looks good, but signatures like cmorwavf(lb,ub,n,fb,fc) are about as unreadable as they come ....

[Current coverage][cc-pull] is 83.64%

Merging [#172][cc-pull] into [master][cc-base-branch] will decrease coverage by 0.99%

@@             master       #172   diff @@
==========================================
  Files            17         20     +3
  Lines          2154       2696   +542
  Methods           0          0
  Messages          0          0
  Branches        242        252    +10
==========================================
+ Hits           1823       2255   +432
- Misses          262        368   +106
- Partials         69         73     +4

Powered by Codecov. Last updated by [718a7b3...569278a][cc-compare]
[cc-base-branch]: https://codecov.io/gh/PyWavelets/pywt/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/PyWavelets/pywt/compare/718a7b3b19336e3807138d3b5160a57f583a7049...569278ae592d72495c0c21ebfe3d4d8ce98f91f0?src=pr
[cc-pull]: https://codecov.io/gh/PyWavelets/pywt/pull/172?src=pr

Would it be possible to again separate the (python) class into a continuous and discrete wavelet class (as a sub-class of a base wavelet class)? Then the various *wavf functions can live on the continuous wavelet class.

I've left a couple inline comments, but I think the big thing to decide is whether to split the discrete and continuous wavelet python classes.
Apart from that, I second what Ralf said. Massive effort, good work :)

OkI splitted the Wavelet into ContinuousWavelet and DiscreteWavelet. What do you think?

Ok, I renamed DiscreteWavelet to Wavelet, so that is isinstance(w, pywt.Wavelet) works again.
The function DiscreteContinuousWavelet gives now either a Wavelet class for discrete wavelet or a ContinuousWavelet class for Continuous wavelets.
How it is possible to mark isinstance(w, pywt.Wavelet) as obsolete? Such that isinstance(w, pywt.ContinuousWavelet) or isinstance(w, pywt.DiscreteWavelet) should be used in the future?

I think I will remove the various "morlet", "gauswavf", "mexihat","cmorwavf", "shanwavf", "fbspwavf", "cgauwavf" functions.
There are all using wavefun from ContinuousWavelet.
For example the gauswavf function just doing:
    wavelet.upper_bound = ub
    wavelet.lower_bound = lb
    psi, x = wavelet.wavefun(length=n)

Do you agree?

I took a look again at the several most recent commits.  I appreciate that a lot of work has gone into this PR and think it is a great asset to PyWavelets!  I think the bottleneck has just been the amount of time we have had available as reviewers.
From what I can see, the recent changes look good and are in line with the previous feedback.  I made only a couple of minor comments.

Also, I think it should be fine to declare all of the new functions imported from cwt.h in c_wt.pxd as nogil to be in line with the changes proposed for the discrete transforms in #188.

@holgern
This needs a rebase after #188 was merged.  Ideally, please also add nogil to the new functions added to c_wt.pxd in this PR.
@PyWavelets/core  Does anyone else have additional feedback for this PR?  If not I will plan to merge next week.

I will commit the change regarding passing the dtype to np.linspace when numpy is bumped to 1.9

sounds good.  If the numpy 1.9 PR is merged prior to this one i will ping you.
The nogil changes here look good to me.

that 1.9 PR is merged now

Overall looks quite good. The licensing of the dataset and whether DiscreteContinuousWavelet should be public are my only critical comments.

Wow this took off! Great work @holgern :smile:

I renamed DiscreteWavelet to Wavelet, so that isinstance(w, pywt.Wavelet) is not broken.
I will add documentation for DiscreteContinuousWavelet.
For future releases, DiscreteContinuousWavelet should be renamed to Wavelet and Wavelet should be renamed to DiscreteWavelet.
I will investigate the licensing of the dataset.

The nino3 data set is also available under:
http://hhstokes.people.uic.edu/ftp/b34sbin/b34sfiles/wavedata.mac
https://github.com/aaren/wavelets/blob/master/tests/nino3data.asc
The dataset is also used in this paper:
Torrence, C., and G. P. Compo, 1998: A practical guide to wavelet analysis. Bull. Amer. Meteor. Soc., 79, 6178 :
http://journals.ametsoc.org/doi/pdf/10.1175/1520-0477(1998)079<0061%3AAPGTWA>2.0.CO%3B2

Hi @holgern,  I think aside from the nino3 license issues this is good to go.  The original sea surface temperature data are avaialble from http://catalogue.ceda.ac.uk/uuid/fda780a7697f384010674ebb1d5e4f23 with full license terms here (here) [http://licences.ceda.ac.uk/image/data_access_condition/ukmo_agreement.pdf].  They specifically state

UKMO data / software so obtained may be used solely for the purpose for which they were supplied. They may not be used for any other projects unless specific prior permission has been obtained in writing from the UKMO by a NERC Data Centre. Note that this applies even for other bona fide academic work.
Data sets must not be passed on to third parties under any circumstances.

Have you tried requesting this permission?  Or could perhaps verify with one of the other sources linked to that they have obtained this permission previously?  As this is a tiny fraction (derived from?) the full dataset, I think it is fairly likely you could get permission, but I am not sure how long it might take.  I am also willing to write them to request this if you would like.

Another possibility I would be okay with is to merge this PR without the example for now and add the example through a separate PR once the licensing issues are resolved.

I found an alternative data source
http://www.cpc.ncep.noaa.gov/data/indices/ersst4.nino.mth.81-10.ascii
This dataset is public domain, it can be read here: http://www.weather.gov/disclaimer under Use of NOAA/NWS Data and Products.

@holgern
Great news on the data license!
merging #193 caused this to need a rebase again.  Let me know if you need any help with that.

Going to finally merge this one.  Thanks for sticking with it through all the revisions @holgern!

Excellent! Thanks @holgern, all.


window is not defined render server side

Hi I implementing you library with react and nextjs, and when I reload the page, I get the error:
window is not defined
ReferenceError: window is not defined
at Object. (/Users/albertocano/Documents/projects/project/project/node_modules/jodit/build/jodit.min.js:9:224)
at Module._compile (internal/modules/cjs/loader.js:689:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
at Module.load (internal/modules/cjs/loader.js:599:32)
at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
at Function.Module._load (internal/modules/cjs/loader.js:530:3)
at Module.require (internal/modules/cjs/loader.js:637:17)
at require (internal/modules/cjs/helpers.js:22:18)
at Object.jodit (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:581:18)
at webpack_require (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:23:31)
at Module../pages/test2.jsx (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:421:63)
at webpack_require (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:23:31)
at Object.4 (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:503:18)
at webpack_require (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:23:31)
at /Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:91:18
at Object. (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:94:10)
It's obviously that window obejct is only defined on browser side, but do you can support render server side? please

Same issue.
Is there any work around found to fix this?

For next js you can use this:
import React from 'react';
import dynamic from 'next/dynamic';

const importJodit = () => import('jodit-react');

const JoditEditor = dynamic(importJodit, {
    ssr: false,
});

Hi I implementing you library with react and nextjs, and when I reload the page, I get the error:
window is not defined
ReferenceError: window is not defined
at Object. (/Users/albertocano/Documents/projects/project/project/node_modules/jodit/build/jodit.min.js:9:224)
at Module._compile (internal/modules/cjs/loader.js:689:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
at Module.load (internal/modules/cjs/loader.js:599:32)
at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
at Function.Module._load (internal/modules/cjs/loader.js:530:3)
at Module.require (internal/modules/cjs/loader.js:637:17)
at require (internal/modules/cjs/helpers.js:22:18)
at Object.jodit (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:581:18)
at webpack_require (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:23:31)
at Module../pages/test2.jsx (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:421:63)
at webpack_require (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:23:31)
at Object.4 (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:503:18)
at webpack_require (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:23:31)
at /Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:91:18
at Object. (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:94:10)
It's obviously that window obejct is only defined on browser side, but do you can support render server side? please

For next js you can use this:
import React from 'react';
import dynamic from 'next/dynamic';

const importJodit = () => import('jodit-react');

const JoditEditor = dynamic(importJodit, {
    ssr: false,
});

For Gatsby:
import React from "react";

const Jodit = React.lazy(() => {
  return import("jodit-react");
});

export const RichTextEditor = (props) => {
  const { value, onChange, ...otherProps } = props;
  const isSSR = typeof window === "undefined";

  return (
    <>
      {!isSSR && (
        <React.Suspense fallback={
Loading RTE
}>
          <Jodit value={value} onChange={onChange} {...otherProps}/>
        </React.Suspense>
      )}
    </>
  );
});

For Gatsby:
import React, { useState } from "react";

const Jodit = React.lazy(() => {
  return import("jodit-react");
});

export const RichTextEditor = () => {
  const [value, setValue] = useState("");
  const isSSR = typeof window === "undefined";

  return (
    <>
      {!isSSR && (
        <React.Suspense fallback={
Loading RTE
}>
          <Jodit value={value} onChange={(val) => setValue(val)}/>
        </React.Suspense>
      )}
    </>
  );
});

That solution worked for me @Fedeorlandau, you definitely have a place in heaven!
"gatsby": "^2.18.12"

For Gatsby:
import React, { useState } from "react";

const Jodit = React.lazy(() => {
  return import("jodit-react");
});

export const RichTextEditor = () => {
  const [value, setValue] = useState("");
  const isSSR = typeof window === "undefined";

  return (
    <>
      {!isSSR && (
        <React.Suspense fallback={
Loading RTE
}>
          <Jodit value={value} onChange={(val) => setValue(val)}/>
        </React.Suspense>
      )}
    </>
  );
});

That solution worked for me @Fedeorlandau, you definitely have a place in heaven!
"gatsby": "^2.18.12"

Thank you @oqnunes. I strongly recommend you checking out https://github.com/ianstormtaylor/slate and https://docs.slatejs.org/ before focusing on jodit. I've found an enormous amount of bugs on jodit while working with react and SSR. Some stuff like dynamic configuration for the rte or customize some inputs on the controls aren't 100% possible with jodit. Regards!


nixos/kubernetes: remove dashboard

Motivation for this change
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

@ofborg test kubernetes

cc @johanot @saschagrunert @srhb

Should we also remove addons/dns or maybe the entire addons module?

@ofborg test kubernetes


Use DllImport for Libuv native binaries

/cc @moozzyk

Depends on https://github.com/aspnet/dnx/issues/402

@moozzyk where are we with this one?

@muratg - this worked on Friday. Need to retest after latest changes to native lib loading and possibly change a layout a little bit - "osx" is not a valid fallback anymore. One breaking change is that it will require to link the libuv as libuv.so and not libuv.so.1 on Linux (I think this is why Travis is failing on my branch). Will try to send a PR for this but ultimately in the worst case RC1 can ship with what we have right now.

Fixed in d4853f9b7c31a28cecac8ac7e965f8f2f783c06b. Needs an announcement for libuv name change on Linux

@moozzyk Please post the announcement issue when you're back. That would be still before RC1 release.


Difficult to determine number of documents fetched by StreamBuilder

I'm using StreamBuilder in a real-time chat app using Firestore. To pull old chat messages from Firestore when the user scrolls to the top, I'm just incrementing _queryLimit in the following code:
            StreamBuilder(
              stream: Firestore.instance
                  .collection('messages')
                  .document(groupChatId)
                  .collection(groupChatId)
                  .orderBy('timestamp', descending: true)
                  .limit(_queryLimit)
                  .snapshots(),
              builder: (context, snapshot) {...

If Flutter is doing what I think it's doing, only one document read should happen when _queryLimit is incremented by one.
While debugging, I could see that  snapshot.data.documentChanges contains only one document, viz., the new one visible on the screen.
But that doesn't tell me that only one document was actually fetched from Firestore. If a document doesn't change, is it never downloaded? What if the local snapshot had to be removed for memory reasons? Although I expect the answer is 'no' in most cases, I'd like to see it while running my code... so that my Firebase charges are predictable.
Put differently, the parameter documentChanges doesn't tell the whole story. Maybe a documentsFetched will remove (at least) my confusion...

Without additional information, we are unfortunately not sure how to resolve this issue. We are therefore reluctantly going to close this bug for now. Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away!
Thanks for your contribution.


Update the theme styles of the website

Is there an existing issue for this?

[X] I have searched existing issues, it hasn't been reported yet

Use case description
I would like to update the theme color of the website as well as the default styles of charts and table components.
Proposed solution (optional)
No response

@frank-zsy I am trying to understand the issue, since we already have a toggle-btn between light/dark mode, I'm curious to know what is meant by update of the theme color of the website 🤔
Is it something connected to this?

Yes, actually I found a UI designer to give us a new design of our color theme and especially the chart styles. I will do this and the preview looks like:

The charts background color will stay identical with the website background color and I will optimize the options for charts and inject default styles for line, bar charts, WDYT?

@frank-zsy I loved it! 😃
Just curious, is the color theme inspired by both OpenAtom & PolarDB dashboard (themes)? 🤔  Nevertheless, the UI components look much more consistent across the pages. For the charts, I'd recommend aiming for high-contrast colors (as shown in the preview) to maximize the page's aesthetics.
And yes, let's go for this! 🚀

@Neilblaze Not quite inspired by OpenAtom & PolarDB dashboard. Actually I am personally prefer deep blue color theme which has a stronger sense of technology.😄


Any similar iOS lib?

Not rally an issue.
This is indeed good lib for a tutorial\help wizard. Just needed to check any one found similar iOS lib?
because  our team develops android and iOS apps.
Thanks

I'm unaware of a similar library for iOS, however this is not the place to ask. Sorry.


Update sample-material.ttl

closes #211
Sandstone, Shale, Siltstone added to vocabulary. Details taken from the upcoming lithology vocabulary that had details and definitions derived from IUGS, BGS, mindat.org (i.e. definitions and alt terms are internationally recognised and not my fault)
So you don't have to read the actual ttl code the relevant changes are below. There should be a green review button (top right where you can approve or comment these additions)
sandstone
definition "A sand-grade clastic-sedimentary-rock."@en ;
broader rock ;
alias "Sand-grade-sedimentary-rock"@en ;
shale
definition "A fine-grained detrital sedimentary rock, formed by the consolidation (esp. by compression) of clay, silt, or mud. It is characterized by finely laminated structure, which imparts a fissility approx. parallel to the bedding, along which the rock breaks readily into thin layers, and by an appreciable content of clay minerals and detrital quartz; a thinly laminated or fissile claystone, siltstone, or mudstone."@en ;
broader rock ;
siltstone
definition "A mudstone with silt-grade clasts. An indurated silt having the texture and composition of shale but lacking its fine lamination or fissility; a massive mudstone in which the silt predominates over clay; a nonfissile silty shale. It tends to be flaggy, containing hard, durable, generally thin layers, and often showing various primary current structures."@en ;
broader rock ;
alias "Siltite"@en ;

Great thanks, definitions are better than anything I could come up with (without taking hours or days).
Cheers
Sal
From: KellyVance<EMAIL_ADDRESS>Sent: Wednesday, 13 May 2020 3:51 PM
To: geological-survey-of-queensland/vocabularies
Cc: EDWARDS Sally; Review requested
Subject: [geological-survey-of-queensland/vocabularies] Update sample-material.ttl (#214)
closes #211https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_issues_211&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=NHVL3wKzzuYiQncSM6rCR0zJh9bvIFhJT1yocCMpIgg&m=lzANRXuu4NANv3dRgHqf4z3rCVG6_ar-W-5cRskhZZo&s=3uaKh6BaNYwpMxh2f4ISsAAzgIbijEIV9lZ9DALt-NI&e=
Sandstone, Shale, Siltstone added to vocabulary. Details taken from the upcoming lithology vocabulary that had details and definitions derived from IUGS, BGS, mindat.org (i.e. definitions and alt terms are internationally recognised and not my fault)
So you don't have to read the actual ttl code the relevant changes are below. There should be a green review button (top right where you can approve or comment these additions)
sandstone
definition "A sand-grade clastic-sedimentary-rock."@enhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_en&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=NHVL3wKzzuYiQncSM6rCR0zJh9bvIFhJT1yocCMpIgg&m=lzANRXuu4NANv3dRgHqf4z3rCVG6_ar-W-5cRskhZZo&s=XkVKxMYKWwq3MKCaBKuhK6Ycz6UbTlHY0cSYz0Dlvkc&e= ;
broader rock ;
alias "Sand-grade-sedimentary-rock"@enhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_en&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=NHVL3wKzzuYiQncSM6rCR0zJh9bvIFhJT1yocCMpIgg&m=lzANRXuu4NANv3dRgHqf4z3rCVG6_ar-W-5cRskhZZo&s=XkVKxMYKWwq3MKCaBKuhK6Ycz6UbTlHY0cSYz0Dlvkc&e= ;
shale
definition "A fine-grained detrital sedimentary rock, formed by the consolidation (esp. by compression) of clay, silt, or mud. It is characterized by finely laminated structure, which imparts a fissility approx. parallel to the bedding, along which the rock breaks readily into thin layers, and by an appreciable content of clay minerals and detrital quartz; a thinly laminated or fissile claystone, siltstone, or mudstone."@enhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_en&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=NHVL3wKzzuYiQncSM6rCR0zJh9bvIFhJT1yocCMpIgg&m=lzANRXuu4NANv3dRgHqf4z3rCVG6_ar-W-5cRskhZZo&s=XkVKxMYKWwq3MKCaBKuhK6Ycz6UbTlHY0cSYz0Dlvkc&e= ;
broader rock ;
siltstone
definition "A mudstone with silt-grade clasts. An indurated silt having the texture and composition of shale but lacking its fine lamination or fissility; a massive mudstone in which the silt predominates over clay; a nonfissile silty shale. It tends to be flaggy, containing hard, durable, generally thin layers, and often showing various primary current structures."@enhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_en&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=NHVL3wKzzuYiQncSM6rCR0zJh9bvIFhJT1yocCMpIgg&m=lzANRXuu4NANv3dRgHqf4z3rCVG6_ar-W-5cRskhZZo&s=XkVKxMYKWwq3MKCaBKuhK6Ycz6UbTlHY0cSYz0Dlvkc&e= ;
broader rock ;
alias "Siltite"@enhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_en&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=NHVL3wKzzuYiQncSM6rCR0zJh9bvIFhJT1yocCMpIgg&m=lzANRXuu4NANv3dRgHqf4z3rCVG6_ar-W-5cRskhZZo&s=XkVKxMYKWwq3MKCaBKuhK6Ycz6UbTlHY0cSYz0Dlvkc&e= ;

You can view, comment on, or merge this pull request online at:
https://github.com/geological-survey-of-queensland/vocabularies/pull/214https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_214&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=NHVL3wKzzuYiQncSM6rCR0zJh9bvIFhJT1yocCMpIgg&m=lzANRXuu4NANv3dRgHqf4z3rCVG6_ar-W-5cRskhZZo&s=9Re8FxZAN4hOOQSx2HvZpHURijOJWoJPFs3dB-IpG8M&e=
Commit Summary

Update sample-material.ttl

File Changes

M vocabularies/sample-material.ttlhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_214_files-23diff-2Ddab90c4c7acedcd1f3434a421c61b50c&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=NHVL3wKzzuYiQncSM6rCR0zJh9bvIFhJT1yocCMpIgg&m=lzANRXuu4NANv3dRgHqf4z3rCVG6_ar-W-5cRskhZZo&s=bjYj0yUwH99pRcmI5ZQG7hr5g8JFm0KKUhtqt7bj7GQ&e= (29)

Patch Links:

https://github.com/geological-survey-of-queensland/vocabularies/pull/214.patchhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_214.patch&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=NHVL3wKzzuYiQncSM6rCR0zJh9bvIFhJT1yocCMpIgg&m=lzANRXuu4NANv3dRgHqf4z3rCVG6_ar-W-5cRskhZZo&s=UYIEPF1nyarlJM9aKPFe3JTqnSF8qJ3F5LQroBRvEDo&e=
https://github.com/geological-survey-of-queensland/vocabularies/pull/214.diffhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_214.diff&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=NHVL3wKzzuYiQncSM6rCR0zJh9bvIFhJT1yocCMpIgg&m=lzANRXuu4NANv3dRgHqf4z3rCVG6_ar-W-5cRskhZZo&s=UvqYixw0rTHsXc87-jFnSj0O7-07yK4OuKWkj2Z2d-4&e=

—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_214&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=NHVL3wKzzuYiQncSM6rCR0zJh9bvIFhJT1yocCMpIgg&m=lzANRXuu4NANv3dRgHqf4z3rCVG6_ar-W-5cRskhZZo&s=9Re8FxZAN4hOOQSx2HvZpHURijOJWoJPFs3dB-IpG8M&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_APDQ7MSAU3F47NQIRO4A6ULRRIYNPANCNFSM4M7N3QVQ&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=NHVL3wKzzuYiQncSM6rCR0zJh9bvIFhJT1yocCMpIgg&m=lzANRXuu4NANv3dRgHqf4z3rCVG6_ar-W-5cRskhZZo&s=OD37GXfoldCVGcL_-uV894Rn3Bwzdb1Hwjz9sVao12k&e=.

The information in this email together with any attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. There is no waiver of any confidentiality/privilege by your inadvertent receipt of this material.
Any form of review, disclosure, modification, distribution and/or publication of this email message is prohibited, unless as a necessary part of Departmental business.
If you have received this message in error, you are asked to inform the sender as quickly as possible and delete this message and any copies of this message from your computer and/or your computer system network.

The definitions are great thanks.

I think I hit the wrong button, and closed the issue.
Hope you are able to do whatever you need to do with it after my comment accepted your definitions.
From: KellyVance<EMAIL_ADDRESS>Sent: Wednesday, 13 May 2020 3:51 PM
To: geological-survey-of-queensland/vocabularies
Cc: EDWARDS Sally; Review requested
Subject: Re: [geological-survey-of-queensland/vocabularies] Update sample-material.ttl (#214)
@KellyVancehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_KellyVance&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=NHVL3wKzzuYiQncSM6rCR0zJh9bvIFhJT1yocCMpIgg&m=JA3uFSVEta3kWzRwCZgNwpYALmm-njFMsVrSl-681-E&s=p7pX3IGzEs7PlKsn15Ca-aZ1Z95fsaTrHcnwKBXg8LY&e= requested your review on: #214https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_214&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=NHVL3wKzzuYiQncSM6rCR0zJh9bvIFhJT1yocCMpIgg&m=JA3uFSVEta3kWzRwCZgNwpYALmm-njFMsVrSl-681-E&s=O7QuBZ5ZCIhcgJx2yw4NAq6AkQGt2IEPDImw9CQ8gAE&e= Update sample-material.ttl.
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geological-2Dsurvey-2Dof-2Dqueensland_vocabularies_pull_214-23event-2D3330419915&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=NHVL3wKzzuYiQncSM6rCR0zJh9bvIFhJT1yocCMpIgg&m=JA3uFSVEta3kWzRwCZgNwpYALmm-njFMsVrSl-681-E&s=cegf34DyMe5kInP_8S-yJ41BY-j75x4UK0-JShFxdwU&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_APDQ7MSAU3F47NQIRO4A6ULRRIYNPANCNFSM4M7N3QVQ&d=DwMCaQ&c=tpTxelpKGw9ZbZ5Dlo0lybSxHDHIiYjksG4icXfalgk&r=NHVL3wKzzuYiQncSM6rCR0zJh9bvIFhJT1yocCMpIgg&m=JA3uFSVEta3kWzRwCZgNwpYALmm-njFMsVrSl-681-E&s=IMRohehnaMdWZ3SU3h7ASR1lBnDit9AQys3LsG3I9as&e=.

The information in this email together with any attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. There is no waiver of any confidentiality/privilege by your inadvertent receipt of this material.
Any form of review, disclosure, modification, distribution and/or publication of this email message is prohibited, unless as a necessary part of Departmental business.
If you have received this message in error, you are asked to inform the sender as quickly as possible and delete this message and any copies of this message from your computer and/or your computer system network.


Code of Conduct

We are intending to adopt the soon-to-be-announced Open Code of Conduct for all @github projects. This adds it to the contributing guidelines. Let me know if you have any questions or concerns.

Really great, @bkeepers! :+1:

:+1: I saw the same code of conduct is applying to Go this morning at the keynote of GopherCon, wondering if there's any correlation of that :)

I saw the same code of conduct is applying to Go this morning at the keynote of GopherCon, wondering if there's any correlation of that :)

Nice, is there a link to that? This is an effort we're coordinating with several other companies, but I didn't realize Go was adopting it too.

Nice, is there a link to that?

The Go team published this blog post immediately after the keynote (scroll down to the section "Code of Conduct"). I'm sure the recordings and slides of @rsc's keynote will be available soon.

The Go team is still deciding details, but seeing this version is useful input to that process. Thanks. /cc @adg

:clap:


Enable oidc auth

To pass tests, you need a PR in the main repo with only adding dependency (not updating /base submodule).

@mbohlool thank you, I updated the PR for (https://github.com/kubernetes-incubator/client-python/pull/331) removing the submodule update.

Re-run test

look like there is only formatting issues with test: https://travis-ci.org/kubernetes-client/python-base/jobs/270071663
Sorry that I didn't spend time on this yet because of the rush of PRs before kubernetes code freeze. I will do a proper review very soon.

Codecov Report

Merging #31 into master will decrease coverage by 3.62%.
The diff coverage is 20.93%.

@@            Coverage Diff             @@
##           master      #31      +/-   ##
==========================================
- Coverage   93.93%   90.31%   -3.63%
==========================================
  Files          11       11
  Lines         824      867      +43
==========================================
+ Hits          774      783       +9
- Misses         50       84      +34

Impacted Files
Coverage Δ

config/kube_config.py
80.64% <20.93%> (-12.53%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 84d1284...c246d19. Read the comment docs.

Thanks @mbohlool Ill write some unit tests this week or next (time depending).

any update on this? also did you test this? e.g. is there a minikube way to setup oidc and test this? or what flavor of kubernetes cluster have oidc support?

@mbohlool sorry I have been down with the cold.. I have tested this with CoreOS Tectonic which uses OIDC for authentication.
I can probably just write unit tests to begin with before integration tests.

@dmyerscough no worries. Integration tests are not required for this PR to be merged, but if you are up to add it, go for it. But unit tests are more important.
Thanks for doing this.

@dmyerscough friendly ping. Any updates?

@dmyerscough you may want to look at this comment https://github.com/kubernetes-incubator/client-python/issues/368#issuecomment-337778192

@mbohlool I have been pretty busy and haven't had time to look at this.

Hi @dmyerscough,
I think I am ready for the PR, my only problem is that I cannot pass the SSL certificate to the refresh_token  call. According to this PR (https://github.com/requests/requests-oauthlib/pull/293) it is possible, but it didn't work for me. My workaround is not to verify the certificate using request.verify=false ( on the call when the client try to refresh the token). Is that OK  @mbohlool @dmyerscough?

@dmyerscough  @mbohlool  forgot my last message, I found my problem

hi @dmyerscough
I have problems with travis on the test test_oidc_with_refresh (on coverage), any idea?
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.yxT81y67iD/python/.tox/coverage/lib/python2.7/site-packages/mock/mock.py", line 1297, in patched
    arg = patching.__enter__()
  File "/tmp/tmp.yxT81y67iD/python/.tox/coverage/lib/python2.7/site-packages/mock/mock.py", line 1353, in __enter__
    self.target = self.getter()
  File "/tmp/tmp.yxT81y67iD/python/.tox/coverage/lib/python2.7/site-packages/mock/mock.py", line 1523, in <lambda>
    getter = lambda: _importer(target)
  File "/tmp/tmp.yxT81y67iD/python/.tox/coverage/lib/python2.7/site-packages/mock/mock.py", line 1206, in _importer
    thing = __import__(import_path)
ImportError: No module named config

I apologize in advance if this is already supported, but from looking at the PR, I think there needs to be a way to refresh the token not only at configuration initialization time but also during invocation of the K8 API calls. The token might expire right before a call thus causing the call to fail with an authentication failure. This will be needed for services that run a long time and rely on being able to reach the K8 Api Server (also short services that happen to be unlucky).
I think this can be done with a check to 'exp' field in the id-token right before a call_api or catch a 401 return status and perform a refresh token operation and update the configuration object with the new id_token and refresh_token values from the OAuth server.

Please ref #48


Can a Table implement sql.IndexAddressableTable without implementing sql.DriverIndexableTable?

I have an implementation of Table that also implements IndexAddressableTable but not DriverIndexableTable. When using this table in a the join node is correctly being replaced with an IndexedJoin node, and the table node is being replaced with an IndexedTableAccess as expected. However, the trackProcess rule is wrapping my table with a ProcessTable node which no longer implements IndexAddressableTable. This causes the IndexedTableAccess.RowIter to return ErrNoIndexableTable. Looking at the trackProcess rule, it seems like a similar concern was addressed because it takes special care of DriverIndexableTable nodes, but not IndexAddressableTable nodes. Should trackProcess be handling IndexAddressableTable nodes, or should I be implementing DriverIndexableTable (even though I don't think that is correct for my use-case).

I think this is a bug. Can you submit a patch?

Will do

Enough time has passed that I'm not certain that this bug still applies, and we definitely do not implement the Driver* interfaces in dolt, where indexed lookups work. Please reopen an issue if you can get a working repro demonstrating a defect here.


feat(typescript): more progress toward strict types

Build currently fails, there are some issues in waterfall plugin which I don't have enough brain juice currently to figure out.
Only 4 issues left towards being able to enable the strictNullChecks flags which is a nice milestone and good guard for future code.
@pyatyispyatil  would appreciate if you could checkout the branch and check the issue in waterfall plugin and in case you fix it to commit it in this pr

alright, I fixed issues in waterfall-plugin
now only 4 issues left in flame-chart.ts, but that seems like a ts limitation


Command Line Argument Support

Command line argument support would be useful for users trying to add EzUnlock to the context menu (for example in send to) to easily unlock files.

Yes. Also, are you interested to drop a PR for this?

I don't know C, C# or C++ but I can try to learn some to make a PR as I was going to learn some type of C anyways most likely. It will probably be not of good code quality though.


Set max ssh connection count

Coverage decreased (-0.1%) to 75.571% when pulling ad5cd07abf783704d787f45aaeea42ff25355c88 on lukasmacko:master into f6e8487ec631896fa44f6712eacacfc81b33285f on contiv:master.


Fix for $in and $nin

This now allows $in and $nin to support fields that are not arrays

awesome, 6f0196c3eee7046b858c46779fd35439fae456e2


test: make function provided to WithTimeout run asynchronously

The provided function was previously ran in the same goroutine as that which
would fail upon timeout. If the function itself took longer than the amount of
time in the timeout, or got stuck, the timeout would never get hit because it
was running in the same goroutine as the function itself. So, run the function
in a goroutine and return the result via a channel, which we can select upon. If
the timeout is hit before the function exits, WithTimeout will now return an
error as expected instead of waiting forever.
Signed-off by: Ian Vernon<EMAIL_ADDRESS>

This change is

Coverage increased (+0.03%) to 41.616% when pulling ab93bf31a3540615cd7e9f19be787d6690d0c3f4 on pr/ianvernon/fix-bodyfunc-async into 2dcd58c504b20ed3c96e34502b2a82d2bc2ae35c on master.

Build failed: https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Validated/11919/
ClusterIP connectivity test failed. This has been occurring on master, and is not related to the nature of this change IMO. I'll run the tests again for good measure, but I think this can be merged.

test-me-please


mTLS authn method

Adds support for client TLS certificate validation at application layer in Authorino for connections with the protected service and Authorino raw HTTP authorization interface relying on Mutual Transport Layer Security (mTLS) for authentication.
Closes #8
Verification steps
Create a cluster, build and deploy the apps:
make local-setup FF=1

Create a CA certificate and store it in a secret to be discovered by Authorino:
openssl req -x509 -sha256 -days 365 -nodes -newkey rsa:2048 -subj "/CN=talker-api-ca" -keyout /tmp/ca.key -out /tmp/ca.crt
kubectl create secret tls talker-api-ca --cert=/tmp/ca.crt --key=/tmp/ca.key
kubectl label secret talker-api-ca authorino.kuadrant.io/managed-by=authorino app=talker-api

Redeploy Envoy acknowledging the CA cert and enabling client certificate validation:
(This steps sucks, but otherwise Envoy will not pass the client peer certificate to Authorino in the payload to external authorization. Effectively, we're adding redundant client cert validation – i.e. Envoy doing it at L4 and Authorino at L7. For a hope on finding a way to avoid this, see https://github.com/envoyproxy/envoy/issues/20563#issuecomment-1152242226.)
kubectl delete deployment/envoy
kubectl apply -f -<<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: envoy
  name: envoy
data:
  envoy.yaml: |
    static_resources:
      listeners:
      - address:
          socket_address:
            address: <IP_ADDRESS>
            port_value: 8000
        filter_chains:
        - transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              common_tls_context:
                tls_certificates:
                - certificate_chain: {filename: "/etc/ssl/certs/talker-api/tls.crt"}
                  private_key: {filename: "/etc/ssl/certs/talker-api/tls.key"}
                validation_context:
                  trusted_ca:
                    filename: /etc/ssl/certs/talker-api/tls.crt
          filters:
          - name: envoy.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: local
              route_config:
                name: local_route
                virtual_hosts:
                - name: local_service
                  domains: ['*']
                  routes:
                  - match: { prefix: / }
                    route: { cluster: talker-api }
              http_filters:
              - name: envoy.filters.http.ext_authz
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
                  transport_api_version: V3
                  failure_mode_allow: false
                  include_peer_certificate: true
                  grpc_service:
                    envoy_grpc: { cluster_name: authorino }
                    timeout: 1s
              - name: envoy.filters.http.router
                typed_config: {}
              use_remote_address: true
      clusters:
      - name: authorino
        connect_timeout: 0.25s
        type: strict_dns
        lb_policy: round_robin
        http2_protocol_options: {}
        load_assignment:
          cluster_name: authorino
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: authorino-authorino-authorization
                    port_value: 50051
        transport_socket:
          name: envoy.transport_sockets.tls
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
            common_tls_context:
              validation_context:
                trusted_ca:
                  filename: /etc/ssl/certs/authorino-ca-cert.crt
      - name: talker-api
        connect_timeout: 0.25s
        type: strict_dns
        lb_policy: round_robin
        load_assignment:
          cluster_name: talker-api
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: talker-api
                    port_value: 3000
    admin:
      access_log_path: "/tmp/admin_access.log"
      address:
        socket_address:
          address: <IP_ADDRESS>
          port_value: 8001
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: envoy
  name: envoy
spec:
  selector:
    matchLabels:
      app: envoy
  template:
    metadata:
      labels:
        app: envoy
    spec:
      containers:
      - args:
        - --config-path /usr/local/etc/envoy/envoy.yaml
        - --service-cluster front-proxy
        - --log-level info
        - --component-log-level filter:trace,http:debug,router:debug
        command:
        - /usr/local/bin/envoy
        image: envoyproxy/envoy:v1.19-latest
        name: envoy
        ports:
        - containerPort: 8000
          name: web
        - containerPort: 8001
          name: admin
        volumeMounts:
        - mountPath: /usr/local/etc/envoy
          name: config
          readOnly: true
        - mountPath: /etc/ssl/certs/authorino-ca-cert.crt
          name: authorino-ca-cert
          readOnly: true
          subPath: ca.crt
        - mountPath: /etc/ssl/certs/talker-api
          name: talker-api-ca
          readOnly: true
      volumes:
      - configMap:
          items:
          - key: envoy.yaml
            path: envoy.yaml
          name: envoy
        name: config
      - name: authorino-ca-cert
        secret:
          defaultMode: 420
          secretName: authorino-ca-cert
      - name: talker-api-ca
        secret:
          defaultMode: 420
          secretName: talker-api-ca
---
apiVersion: v1
kind: Service
metadata:
  name: envoy
spec:
  selector:
    app: envoy
  ports:
  - name: web
    port: 8000
    protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-wildcard-host
spec:
  rules:
  - host: talker-api-authorino.<IP_ADDRESS>.nip.io
    http:
      paths:
      - backend:
          service:
            name: envoy
            port: { number: 8000 }
        path: /
        pathType: Prefix
EOF

Create the AuthConfig:
kubectl apply -f -<<EOF
apiVersion: authorino.kuadrant.io/v1beta1
kind: AuthConfig
metadata:
  name: talker-api-protection
spec:
  hosts:
  - talker-api-authorino.<IP_ADDRESS>.nip.io
  identity:
  - name: mtls
    mtls:
      labelSelectors:
        app: talker-api
  authorization:
  - name: acme
    json:
      rules:
      - selector: auth.identity.Organization
        operator: incl
        value: ACME Inc.
EOF

Try the API with a TLS certificate signed by the trusted CA:
openssl genrsa -out /tmp/aisha.key 2048
openssl req -new -key /tmp/aisha.key -out /tmp/aisha.csr -subj "/CN=aisha/C=PK/L=Islamabad/O=ACME Inc./OU=Engineering"
openssl x509 -req -in /tmp/aisha.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -CAcreateserial -out /tmp/aisha.crt -days 1 -sha256

curl -k --cert /tmp/aisha.crt --key /tmp/aisha.key https://talker-api-authorino.<IP_ADDRESS>.nip.io:8000 -i
# HTTP/1.1 200 OK

Try the API with a TLS certificate signed by the trusted CA, though missing an authorized Organization:
openssl genrsa -out /tmp/john.key 2048
openssl req -new -key /tmp/john.key -out /tmp/john.csr -subj "/CN=john/C=UK/L=London"
openssl x509 -req -in /tmp/john.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -CAcreateserial -out /tmp/john.crt -days 1 -sha256

curl -k --cert /tmp/john.crt --key /tmp/john.key https://talker-api-authorino.<IP_ADDRESS>.nip.io:8000 -i
# HTTP/1.1 403 Forbidden
# x-ext-auth-reason: Unauthorized

Try the AuthConfig via raw HTTP authorization interface:
Expose the interface:
kubectl port-forward service/authorino-authorino-authorization 5001:5001 &

Consume the raw HTTP authorization interface with a TLS certificate signed by the trusted CA:
curl -k --cert /tmp/aisha.crt --key /tmp/aisha.key -H 'Content-Type: application/json' -d '{}' https://talker-api-authorino.<IP_ADDRESS>.nip.io:5001/check -i
# HTTP/2 200

Consume the raw HTTP authorization interface with a TLS certificate signed by a unknown authority:
openssl req -x509 -sha256 -days 365 -nodes -newkey rsa:2048 -subj "/CN=untrusted" -keyout /tmp/untrusted-ca.key -out /tmp/untrusted-ca.crt
openssl genrsa -out /tmp/niko.key 2048
openssl req -new -key /tmp/niko.key -out /tmp/niko.csr -subj "/CN=niko/C=JP/L=Osaka"
openssl x509 -req -in /tmp/niko.csr -CA /tmp/untrusted-ca.crt -CAkey /tmp/untrusted-ca.key -CAcreateserial -out /tmp/niko.crt -days 1 -sha256

curl -k --cert /tmp/niko.crt --key /tmp/niko.key -H 'Content-Type: application/json' -d '{}' https://talker-api-authorino.<IP_ADDRESS>.nip.io:5001/check -i
# HTTP/2 401
# www-authenticate: Basic realm="mtls"
# x-ext-auth-reason: x509: certificate signed by unknown authority

Revoke an entire chain of certificates:
kubectl delete secret/talker-api-ca

Even if the deleted root certificate is still cached and accepted at the gateway, Authorino will revoke access at application level immediately.
Try the API again with a previously accepted certificate to verify:
curl -k --cert /tmp/aisha.crt --key /tmp/aisha.key https://talker-api-authorino.<IP_ADDRESS>.nip.io:8000 -i
# HTTP/1.1 401 Unauthorized
# www-authenticate: Basic realm="mtls"
# x-ext-auth-reason: x509: certificate signed by unknown authority

Cleanup:
make local-cleanup

Verification steps using intermediate CAs
The certificate tree in this example looks like the following:

Steps:
# build and deploy everything
make local-setup FF=1

# create the authconfig
kubectl apply -f -<<EOF
apiVersion: authorino.kuadrant.io/v1beta1
kind: AuthConfig
metadata:
  name: talker-api-protection
spec:
  hosts:
  - talker-api-authorino.<IP_ADDRESS>.nip.io
  identity:
  - name: mtls
    mtls:
      labelSelectors:
        app: talker-api
  authorization:
  - name: acme
    json:
      rules:
      - selector: auth.identity.Organization
        operator: incl
        value: ACME Inc.
EOF

# create the root ca and store it in a k8s secret
mkdir /tmp/mtls

cat <<EOF > /tmp/mtls/openssl.cnf
HOME                    = .
RANDFILE                = $ENV::HOME/.rnd
oid_section             = new_oids

[ new_oids ]

[ ca ]
default_ca              = CA_default

[ CA_default ]
dir                     = /tmp/mtls
new_certs_dir           = /tmp/mtls
database                = /tmp/mtls/index.txt
serial                  = /tmp/mtls/serial
certificate             = /tmp/mtls/ca.crt
private_key             = /tmp/mtls/ca.key
default_md              = sha256
policy                  = policy_match

[ policy_match ]
countryName             = optional
stateOrProvinceName     = optional
organizationName        = optional
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ policy_anything ]
countryName             = optional
stateOrProvinceName     = optional
localityName            = optional
organizationName        = optional
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ req ]
default_bits            = 4096
default_md              = sha256
default_keyfile         = privkey.pem
distinguished_name      = req_distinguished_name
x509_extensions         = v3_ca
string_mask             = nombstr

[ req_distinguished_name ]
countryName                     = Country Name (2 letter code)
countryName_default             = ES
countryName_min                 = 2
countryName_max                 = 2
stateOrProvinceName             = State or Province Name (full name)
stateOrProvinceName_default     = Barcelona
localityName                    = Locality Name (eg, city)
localityName_default            = Barcelona
0.organizationName              = Organization Name (eg, company)
0.organizationName_default      = ACME Inc.
organizationalUnitName          = Organizational Unit Name (eg, section)
commonName                      = Common Name (eg, your name or your server\'s hostname)
commonName_max                  = 64
emailAddress                    = Email Address
emailAddress_max                = 64

[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
basicConstraints = critical,CA:true

[ v3_intermediate_ca ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign
EOF

echo 01 > /tmp/mtls/serial
touch /tmp/mtls/index.txt

openssl req -x509 -sha256 -days 3650 -nodes -newkey rsa:2048 -config /tmp/mtls/openssl.cnf -extensions v3_ca -subj "/CN=talker-api-authorino.<IP_ADDRESS>.nip.io" -keyout /tmp/mtls/ca.key -out /tmp/mtls/ca.crt
kubectl create secret tls talker-api-ca --cert=/tmp/mtls/ca.crt --key=/tmp/mtls/ca.key

# redeploy envoy to start using the root ca
kubectl delete deployment/envoy
kubectl apply -f -<<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: envoy
  name: envoy
data:
  envoy.yaml: |
    static_resources:
      listeners:
      - address:
          socket_address:
            address: <IP_ADDRESS>
            port_value: 8000
        filter_chains:
        - transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              common_tls_context:
                tls_certificates:
                - certificate_chain: {filename: "/etc/ssl/certs/talker-api/tls.crt"}
                  private_key: {filename: "/etc/ssl/certs/talker-api/tls.key"}
                validation_context:
                  trusted_ca:
                    filename: /etc/ssl/certs/talker-api/tls.crt
          filters:
          - name: envoy.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: local
              route_config:
                name: local_route
                virtual_hosts:
                - name: local_service
                  domains: ['*']
                  routes:
                  - match: { prefix: / }
                    route: { cluster: talker-api }
              http_filters:
              - name: envoy.filters.http.ext_authz
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
                  transport_api_version: V3
                  failure_mode_allow: false
                  include_peer_certificate: true
                  grpc_service:
                    envoy_grpc: { cluster_name: authorino }
                    timeout: 1s
              - name: envoy.filters.http.router
                typed_config: {}
              use_remote_address: true
      clusters:
      - name: authorino
        connect_timeout: 0.25s
        type: strict_dns
        lb_policy: round_robin
        http2_protocol_options: {}
        load_assignment:
          cluster_name: authorino
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: authorino-authorino-authorization
                    port_value: 50051
        transport_socket:
          name: envoy.transport_sockets.tls
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
            common_tls_context:
              validation_context:
                trusted_ca:
                  filename: /etc/ssl/certs/authorino-ca-cert.crt
      - name: talker-api
        connect_timeout: 0.25s
        type: strict_dns
        lb_policy: round_robin
        load_assignment:
          cluster_name: talker-api
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: talker-api
                    port_value: 3000
    admin:
      access_log_path: "/tmp/admin_access.log"
      address:
        socket_address:
          address: <IP_ADDRESS>
          port_value: 8001
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: envoy
  name: envoy
spec:
  selector:
    matchLabels:
      app: envoy
  template:
    metadata:
      labels:
        app: envoy
    spec:
      containers:
      - args:
        - --config-path /usr/local/etc/envoy/envoy.yaml
        - --service-cluster front-proxy
        - --log-level info
        - --component-log-level filter:trace,http:debug,router:debug
        command:
        - /usr/local/bin/envoy
        image: envoyproxy/envoy:v1.19-latest
        name: envoy
        ports:
        - containerPort: 8000
          name: web
        - containerPort: 8001
          name: admin
        volumeMounts:
        - mountPath: /usr/local/etc/envoy
          name: config
          readOnly: true
        - mountPath: /etc/ssl/certs/authorino-ca-cert.crt
          name: authorino-ca-cert
          readOnly: true
          subPath: ca.crt
        - mountPath: /etc/ssl/certs/talker-api
          name: talker-api-ca
          readOnly: true
      volumes:
      - configMap:
          items:
          - key: envoy.yaml
            path: envoy.yaml
          name: envoy
        name: config
      - name: authorino-ca-cert
        secret:
          defaultMode: 420
          secretName: authorino-ca-cert
      - name: talker-api-ca
        secret:
          defaultMode: 420
          secretName: talker-api-ca
---
apiVersion: v1
kind: Service
metadata:
  name: envoy
spec:
  selector:
    app: envoy
  ports:
  - name: web
    port: 8000
    protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-wildcard-host
spec:
  rules:
  - host: talker-api-authorino.<IP_ADDRESS>.nip.io
    http:
      paths:
      - backend:
          service:
            name: envoy
            port: { number: 8000 }
        path: /
        pathType: Prefix
EOF
kubectl port-forward deployment/envoy 8000:8000 &

# create 3 intermediate cas and store 2 of them in k8s secrets watched by authorino
openssl genrsa -out /tmp/mtls/intermediate-ca-1.key 2048
openssl req -new -config /tmp/mtls/openssl.cnf -key /tmp/mtls/intermediate-ca-1.key -out /tmp/mtls/intermediate-ca-1.csr -subj "/CN=intermediate-1"
openssl ca -config /tmp/mtls/openssl.cnf -in /tmp/mtls/intermediate-ca-1.csr -out /tmp/mtls/intermediate-ca-1.crt -extensions v3_intermediate_ca -days 3650
openssl x509 -in /tmp/mtls/intermediate-ca-1.crt -out /tmp/mtls/intermediate-ca-1.crt -outform PEM
kubectl create secret tls intermediate-ca-1 --cert=/tmp/mtls/intermediate-ca-1.crt --key=/tmp/mtls/intermediate-ca-1.key
kubectl label secret intermediate-ca-1 authorino.kuadrant.io/managed-by=authorino app=talker-api

openssl genrsa -out /tmp/mtls/intermediate-ca-2.key 2048
openssl req -new -config /tmp/mtls/openssl.cnf -key /tmp/mtls/intermediate-ca-2.key -out /tmp/mtls/intermediate-ca-2.csr -subj "/CN=intermediate-2"
openssl ca -config /tmp/mtls/openssl.cnf -in /tmp/mtls/intermediate-ca-2.csr -out /tmp/mtls/intermediate-ca-2.crt -extensions v3_intermediate_ca -days 3650
openssl x509 -in /tmp/mtls/intermediate-ca-2.crt -out /tmp/mtls/intermediate-ca-2.crt -outform PEM
kubectl create secret tls intermediate-ca-2 --cert=/tmp/mtls/intermediate-ca-2.crt --key=/tmp/mtls/intermediate-ca-2.key
kubectl label secret intermediate-ca-2 authorino.kuadrant.io/managed-by=authorino app=talker-api

openssl genrsa -out /tmp/mtls/intermediate-ca-3.key 2048
openssl req -new -config /tmp/mtls/openssl.cnf -key /tmp/mtls/intermediate-ca-3.key -out /tmp/mtls/intermediate-ca-3.csr -subj "/CN=intermediate-3"
openssl ca -config /tmp/mtls/openssl.cnf -in /tmp/mtls/intermediate-ca-3.csr -out /tmp/mtls/intermediate-ca-3.crt -extensions v3_intermediate_ca -days 3650
openssl x509 -in /tmp/mtls/intermediate-ca-3.crt -out /tmp/mtls/intermediate-ca-3.crt -outform PEM

# generate leaf certificates signed with each intermediate ca and try to authenticate to the api with them
openssl genrsa -out /tmp/mtls/aisha.key 2048
openssl req -new -config /tmp/mtls/openssl.cnf -key /tmp/mtls/aisha.key -out /tmp/mtls/aisha.csr -subj "/CN=aisha/C=PK/L=Islamabad/O=ACME Inc./OU=Engineering"
openssl x509 -req -in /tmp/mtls/aisha.csr -out /tmp/mtls/aisha.crt -CA /tmp/mtls/intermediate-ca-1.crt -CAkey /tmp/mtls/intermediate-ca-1.key -CAcreateserial -days 1 -sha256
cat /tmp/mtls/intermediate-ca-1.crt >> /tmp/mtls/aisha.crt

curl -k --cert /tmp/mtls/aisha.crt --key /tmp/mtls/aisha.key https://talker-api-authorino.<IP_ADDRESS>.nip.io:8000 -i
# 200

openssl genrsa -out /tmp/mtls/john.key 2048
openssl req -new -config /tmp/mtls/openssl.cnf -key /tmp/mtls/john.key -out /tmp/mtls/john.csr -subj "/CN=john/C=UK/L=London"
openssl x509 -req -in /tmp/mtls/john.csr -out /tmp/mtls/john.crt -CA /tmp/mtls/intermediate-ca-2.crt -CAkey /tmp/mtls/intermediate-ca-2.key -CAcreateserial -days 1 -sha256
cat /tmp/mtls/intermediate-ca-2.crt >> /tmp/mtls/john.crt

curl -k --cert /tmp/mtls/john.crt --key /tmp/mtls/john.key https://talker-api-authorino.<IP_ADDRESS>.nip.io:8000 -i
# 403

openssl genrsa -out /tmp/mtls/niko.key 2048
openssl req -new -config /tmp/mtls/openssl.cnf -key /tmp/mtls/niko.key -out /tmp/mtls/niko.csr -subj "/CN=niko/C=JP/L=Osaka"
openssl x509 -req -in /tmp/mtls/niko.csr -out /tmp/mtls/niko.crt -CA /tmp/mtls/intermediate-ca-3.crt -CAkey /tmp/mtls/intermediate-ca-3.key -CAcreateserial -days 1 -sha256
cat /tmp/mtls/intermediate-ca-3.crt >> /tmp/mtls/niko.crt

curl -k --cert /tmp/mtls/niko.crt --key /tmp/mtls/niko.key https://talker-api-authorino.<IP_ADDRESS>.nip.io:8000 -i
# 401

Thanks for the review, @alexsnaps!

I think convention is to pass a ctx as the first argument of a method. But, even if do this differently (I honestly don't care), I think it'd be nice to have some consistency.

I think you're referring to AuthConfigEvaluator.Call(AuthPipeline, context.Context), right? Sorry. Bad decisions in the past.

I try to keep changes to a minimum, e.g. renaming receiver of methods or import identifiers.

Some people like doing refactoring as complete different PRs (as to rename receiver of method for instance), or just keep them as a separate commit in one PR, so the diff refactoring vs functional diffs can be reviewed independently...

We can still extract https://github.com/Kuadrant/authorino/pull/305/commits/f7660c58a6a4ee18a030c257414567dd3cc4998d and https://github.com/Kuadrant/authorino/pull/305/commits/5349997f9c0f43b94ad956f4961cf7f8a2d5afd3 to a separate PR if you think it's worth it. I guess I was just lazy not to do so in the first place, hoping people would review the history.
Same does not apply to renaming decodeCertificate -> certificateFromSecret in https://github.com/Kuadrant/authorino/pull/305/commits/2c59f8805e8cbf6cc560e883419b94a933f7ade9. That was rubbish. Sincere apologies.


Update docker-compose.yml

Looks like @xakraz and you, @fbstn use different docker network setups. How about to discard network config from docker-compose.yaml? It should be totally environment independent.

Yep 172.17 conflicts with our AWS VPC setup ...
I would say this kind of changes are endless since everybody have a different network setup...
@fbstn, have you looked at https://docs.docker.com/compose/extends/ ?

I've removed network specification from compose file


Build error in Mac OS

How can I solve this?
cargo build                                                                                2.3.5
   Compiling indy v1.4.0 (file:///Users/jima/jdd/indy-sdk/libindy)
error: non-reference pattern used to match a reference (see issue #42640)
   --> src/commands/did.rs:555:24
    |
555 |                 if let Some(data) = &res.data {
    |                        ^^^^^^^^^^ help: consider using: `&Some(data)`

error: aborting due to previous error

error: Could not compile `indy`.

From this, this might be related to https://github.com/rust-lang/rust/issues/42640
I followed this instruction https://github.com/hyperledger/indy-sdk/blob/master/doc/mac-build.md

@atyenoria You need to update your Rust compiler. 1.26 or 1.27 version will work.

@vimmerru
Thanks! It works.
rustup update


Update the Standard User friendly name.

##What

Rename the Standard User selection strings to "Update and Rollback" and "All installer operations".
Add a friendly name () for the Standard User Presentation: "Define Which Installer Commands Standard Users Can Execute".
Add a friendly name () for the Enable Automatic Updates setting: "Microsoft Update Channel"

##Why
These values are not showing up in the Intune UI.
##How
For the friendly names, add a  element.

您好，您的邮件我已经收到，我一定会尽快回复的！谢谢！【此为自动回复邮件】Hello, I've received your email. I will reply to you as soon as possible! Thank you! 【This is an automatic reply email】