Android build: avoid using variable to access variable.

It returns empty path on Ubuntu 18.04 and it seems that the only reason to use it was to make sure that it will work for paths with spaces.

Your assumption is incorrect.  $ndk_var is the name of the environment variable that had a value, so for example this string:
"-gcc-toolchain \$($ndk_var)/toolchains"

means that the resulting command line will contain -gcc-toolchain \$ANDROID_NDK_HOME/toolchains if you have ANDROID_NDK_HOME defined.  Which your change, the command line will instead contain -gcc-toolchain /what/android_ndk_home/contained/toolchains
If you had an empty value, it means that you didn't maintain the variable ANDROID_NDK_HOME (if that's what you defined in the first place) when running make, i.e. you had it only defined for the configuration command, am I right?

Indeed I defined ANDROID_NDK_HOME only for configure command and it was the reason that it didn't work.
Actually I don't see any benefit of using \$($ndk_var), because it should have the same content as $ndk. But now it works, so it doesn't matter much.


Switching between configured Responses

Hi, I found that switching between configured responses is possible by using the ui, which is great. I am trying to find out if this can also be done by rest api. E.g. by put request on a certain path. Is that possible or is there any other way to configure responses using the api endpoints?

Hi @tobiasneubert,
yes there are multiple ways.

You can use collections.
By using _expected query or header


[Feature-Request] Camera - IMU synced in Sync Node

Start with the why:
Camera-IMU messages synchronization are very important in VIO/SLAM development.
Move to the what:
A pull request #917 implemented a Sync Node that synchronized timestamp of camera left/right/center on device then send to host for display. It would be nice if the IMU data can be synchronized with camera data.
Move to the how:
I don't have any idea to implement that yet.
Hi @asahtik ,
Know that you are the author of pull request #917 . Do you have any plans to support the IMU data to be synchronized with camera data through the Sync Node and Message Groups? Thank you in advance!
Best regards,
Hiep Nguyen

Hi @michaelnguyen11, great question. Currently the Sync node synchronizes a group of messages based on their device timestamps. The issue with IMUData is that it groups multiple packets (with timestamps)  together and so it does not have its own timestamp - this requires some special handling that is not yet implemented. However, we are planning on supporting syncing IMU packets through the Sync node.


[scheduler] Valid cron expressions are not parsed correctly

For example, 0 3 L * FRI is a valid expression and works with actual Worker.

https://developers.cloudflare.com/workers/platform/cron-triggers/#supported-cron-expressions

But with miniflare, it throws [mf:err] SchedulerError [ERR_INVALID_CRON]: Unable to parse CRON "0 3 L * FRI": Failed to parse L: L is NaN. since cron-schedule does not support this patterns.

I tried to find alternatives, but I couldn't find any JS implementations. 😓
I wish I could use https://github.com/cloudflare/saffron like html-rewriter-wasm.

Try out https://github.com/hexagon/croner. Supports L and more 👍

Made a PR that might help you(/someone?) get started, #516 🚀


java.lang.ClassNotFoundException: rx.Single on upgrading from 1.1.0.M5 to 1.1.0.RC2

Hi,
I was using Brixton.RC1 as spring-cloud-dependencies  but with the following dependency
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-hystrix</artifactId>
    <version>1.1.0.M5</version>
</dependency>

When I upgrade to Brixton.RC2 and comment out the 1.1.0.M5 on the spring-cloud-starter-hystrix dependency. Am getting following error.
Caused by: java.lang.ClassNotFoundException: rx.Single

Are you using the BOM as it shows in the docs?
<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-cloud-starter-parent</artifactId>
    <version>1.3.3.RELEASE</version>
</parent>
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>Brixton.RC2</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>
<dependencies>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-config</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-eureka</artifactId>
    </dependency>
</dependencies><repositories>
    <repository>
        <id>spring-milestones</id>
        <name>Spring Milestones</name>
        <url>https://repo.spring.io/libs-milestone</url>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
</repositories>

Hi,
I earlier had the following in my pom.xml
spring-boot-starter-parent is in my local repo.

org.springframework.boot
spring-boot-starter-parent
1.3.3.RELEASE

spring-snapshots
Spring Snapshots
https://repo.spring.io/snapshot

true

spring-milestones
Spring Milestones
https://repo.spring.io/milestone

false

But when I simply removed the above snippets and replace with the ones you gave, it couldnt download the spring-cloud-starter-parent-1.3.3.RELEASE.pom from any repo.

Actually that should come from maven central. Maybe your company has a proxy?

Hi,
My company has proxy and able to pull from maven central but am getting confused here.
When I search for "spring-cloud-starter-parent" in search.maven.org, I see the latest/last version only at 1.0.0.RELEASE and group id is org.springframework.cloud or org.springframework.boot.
Seems like Brixton just got pushed.

org.springframework.cloud
spring-cloud-starter-parent
Brixton.RELEASE

group id is org.springframework.cloud or org.springframework.boot? If so where can I find 1.3.3.RELEASE? It is surely not in maven central nor in https://repo.spring.io/libs-milestone

Without showing your whole pom, it's very confusing.

pom-BrixtonRC2.zip
From my Maven settings.xml

maven2-repository.dev.java.net
Java.net Repository for Maven 2
http://repo.maven.apache.org/maven2/

please use triple backticks for code blocks (like xml). I've had to reformat all of your comments to read them.

That pom looks correct.

But still am not able to download this from central maven
Do you see the below in maven central repo.
<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-cloud-starter-parent</artifactId>
    <version>1.3.3.RELEASE</version>
</parent>

O/p of mvn clean install
Downloading: http://lvdma541.phx.aexp.com/maven-repo/org/springframework/boot/sp
ring-cloud-starter-parent/1.3.3.RELEASE/spring-cloud-starter-parent-1.3.3.RELEAS
E.pom
Downloading: http://repo.maven.apache.org/maven2/org/springframework/boot/spring
-cloud-starter-parent/1.3.3.RELEASE/spring-cloud-starter-parent-1.3.3.RELEASE.po
m
Downloading: http://repo.XXX.extra.XXX.com/archiva/repository/third-party/org/
springframework/boot/spring-cloud-starter-parent/1.3.3.RELEASE/spring-cloud-star
ter-parent-1.3.3.RELEASE.pom
Downloading: https://repo.spring.io/libs-milestone/org/springframework/boot/spri
ng-cloud-starter-parent/1.3.3.RELEASE/spring-cloud-starter-parent-1.3.3.RELEASE.
pom
Downloading: https://repo.maven.apache.org/maven2/org/springframework/boot/sprin
g-cloud-starter-parent/1.3.3.RELEASE/spring-cloud-starter-parent-1.3.3.RELEASE.p
om
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
................. Return code is: 502 , R
easonPhrase:notresolvable. and 'parent.relativePath' points at wrong local POM @
 line 14, column 10 -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit
ch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please rea
d the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildin
gException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableMo
delException

I see it should be org.springframework.boot:spring-boot-starter-parent

Hi,
Got this issue resolved. Please close it. But still seeing Invalid HTTP method: PATCH method issue with the new RELEASE jars.Opening new issue
Thanks for the support.

I also experienced this issue upon upgrading.
Simply bumping the io.reactive.rxjava dependency version fixed it.
From:
<dependency>
        <groupId>io.reactivex</groupId>
        <artifactId>rxjava</artifactId>
        <version>1.0.10</version>
</dependency>

To:
<dependency>
        <groupId>io.reactivex</groupId>
        <artifactId>rxjava</artifactId>
        <version>1.1.5</version>
</dependency>


Login/Signup only apply to Store - not needed elsewhere

Login/Signup only apply to Store - not needed elsewhere
Login/SignUp pages could be modals from the store instead of independent pages.
When user is signed in - offer Logout option.

The fields are in the right place - Logout is not done yet


Youtube Playlist not scraped correctly (Downloads half the playlist)

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like this: [x])
Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.12.17. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2018.12.17

Before submitting an issue make sure you have:

[x] At least skimmed through the README, most notably the FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones
[x] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):
youtube-dl -v https://www.youtube.com/playlist?list=PLEA9BB361C67272AC
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.youtube.com/playlist?list=PLEA9BB361C67272AC']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.12.17
[debug] Python version 3.5.2 (CPython) - Linux-4.4.0-141-generic-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: none
[debug] Proxy map: {}
[youtube:playlist] PLEA9BB361C67272AC: Downloading webpage
[download] Downloading playlist: Shmoocon 2006
[youtube:playlist] playlist Shmoocon 2006: Downloading 36 videos

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

Playlist: https://www.youtube.com/playlist?list=PLEA9BB361C67272AC

Description of your issue, suggested solution and other information
If I browse to https://www.youtube.com/playlist?list=PLEA9BB361C67272AC, I can see 70 videos, if I attempt to download via youtube-dl it only downloads 36.

Most of videos are duplicated in this playlist.


is_safe used in templatetags/highlighting.py

Django 1.6 removes the usage of is_safe.  This is still used in templatetags/highlighting.py.  It was handled nicely in templatetags/truncate_letters.py though.

Thanks for reporting this !
Could you please verify/test the patch ?


Add load reporter for server load reporting

This is a continuation of #14793 ~and currently based on that PR~.
~This is not yet ready for review. I hope to see some quick feedback from @isturdy on my usage of the opencensus APIs.~
This is ready for review except for the get_cpu_stats() implementation on the platforms other than Linux.
I skipped some low-risk but detailed tests for now.

This change is

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

[microbenchmarks] No significant performance differences

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[trickle] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

[microbenchmarks] No significant performance differences

[microbenchmarks] No significant performance differences

[microbenchmarks] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[trickle] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

[microbenchmarks] No significant performance differences

[microbenchmarks] No significant performance differences

Something is wrong with my merging from upstream. I will clean it tomorrow.

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[microbenchmarks] No significant performance differences

Thanks for reviewing!

Review status: 19 of 23 files reviewed at latest revision, 9 unresolved discussions, some commit checks failed.

build.yaml, line 1894 at r1 (raw file):
Previously, markdroth (Mark D. Roth) wrote…
I thought we decided that this won't build via make?

I keep this for now because it seems that we still need some discussion about this? See https://github.com/grpc/grpc/pull/15349#discussion_r188754176

src/cpp/server/load_reporter/get_cpu_stats.h, line 29 at r1 (raw file):
Previously, markdroth (Mark D. Roth) wrote…
This function name should be C++-style instead of C style: GetCpuStats().

Done.

src/cpp/server/load_reporter/load_data_store.h, line 78 at r1 (raw file):
Previously, markdroth (Mark D. Roth) wrote…
No need for explicit here, since there are 2 parameters.
Actually, it looks like we're unnecessarily using explicit even for some of the existing ctors.  Please remove throughout, except in cases where the ctor has a single parameter.

Done.

src/cpp/server/load_reporter/load_data_store.h, line 127 at r1 (raw file):
Previously, markdroth (Mark D. Roth) wrote…
Why are these parameters changing from int64 to double?
For that matter, why are some of the parameters doubles even before this PR?  Shouldn't these all be integers?  It's not clear to me why we would have fractional values for any of these.

Logically they should be integers. But there is some tricky problem with OpenCensus here: when making testing view data, OpenCensus will treat the input values as separate measurements and add them together according to the data type (i.e., if it's int, add 1; if it's double, add the original input value). For the testing to be effective and the code length to be reasonable, I changed the data type to double.
But after upon thought, I should actually make a feature request or maybe a pull request to OpenCensus. @isturdy Does it sound reasonable to add a method to set the final view data values directly?

src/cpp/server/load_reporter/load_data_store.cc, line 81 at r1 (raw file):
Previously, markdroth (Mark D. Roth) wrote…
I would avoid using std::stoi(), since it can generate exceptions, which should not be used in Google C++ code, as per the style guide.
Since this code imposes a transitive dependency on absl anyway, consider using absl::SimpleAtoi() instead.

Oh, good to know that.
Changed to sscanf() to be consistent with a change below.

src/cpp/server/load_reporter/load_data_store.cc, line 99 at r1 (raw file):
Previously, markdroth (Mark D. Roth) wrote…
Similarly, suggest using absl::SimpleAtoi() instead of std::stoul().  Same thing throughout this function.

Done.
Since absl::SimpleAtoi() only supports base of 10, I changed to sscanf()

src/cpp/server/load_reporter/load_reporter.h, line 67 at r1 (raw file):
Previously, markdroth (Mark D. Roth) wrote…
Data members should always be private, not protected.
https://google.github.io/styleguide/cppguide.html#Access_Control

I see.

src/cpp/server/load_reporter/load_reporter.cc, line 212 at r1 (raw file):
Previously, markdroth (Mark D. Roth) wrote…
Consider using absl::StrFormat().

Unfortunately, I can't find str_format.h in our submodule. I guess it's a version problem?
Anyways, I changed to sprintf(), which seems similar to absl::StrFormat().

src/cpp/server/load_reporter/util.h, line 25 at r1 (raw file):
Previously, markdroth (Mark D. Roth) wrote…
Since all that's in this file is constants, how about renaming it from util.h to constants.h?

Done.

Comments from Reviewable

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

Review status: 7 of 23 files reviewed at latest revision, 9 unresolved discussions, some commit checks failed.

src/cpp/server/load_reporter/load_data_store.h, line 127 at r1 (raw file):
Previously, AspirinSJL (Juanli Shen) wrote…
Logically they should be integers. But there is some tricky problem with OpenCensus here: when making testing view data, OpenCensus will treat the input values as separate measurements and add them together according to the data type (i.e., if it's int, add 1; if it's double, add the original input value). For the testing to be effective and the code length to be reasonable, I changed the data type to double.
But after upon thought, I should actually make a feature request or maybe a pull request to OpenCensus. @isturdy Does it sound reasonable to add a method to set the final view data values directly?

The only public interface for making ViewData for testing is in https://github.com/census-instrumentation/opencensus-cpp/blob/master/opencensus/stats/testing/test_utils.h. (Otherwise, the only way to populate a ViewData without using implementation headers should be the normal stats workflow: make a View, record data, and then get data for the view.)
We would rather not expose a means of setting final data directly (i.e. bypassing aggregation) to avoid exposing implementation details.

Comments from Reviewable

[microbenchmarks] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

Reviewed 3 of 11 files at r2, 12 of 13 files at r4, 1 of 1 files at r5, 2 of 2 files at r6.
Review status: all files reviewed at latest revision, 2 unresolved discussions, some commit checks failed.

src/cpp/server/load_reporter/load_data_store.h, line 127 at r1 (raw file):
Previously, isturdy (Ian Sturdy) wrote…
The only public interface for making ViewData for testing is in https://github.com/census-instrumentation/opencensus-cpp/blob/master/opencensus/stats/testing/test_utils.h. (Otherwise, the only way to populate a ViewData without using implementation headers should be the normal stats workflow: make a View, record data, and then get data for the view.)
We would rather not expose a means of setting final data directly (i.e. bypassing aggregation) to avoid exposing implementation details.

Even if the opencensus API requires doubles, I don't see why we need to use that type in our code.  We should use int64 here and convert to/from double at the points at which we interact with opencensus.

src/cpp/server/load_reporter/load_data_store.cc, line 83 at r6 (raw file):

  GPR_ASSERT(client_ip_and_token.size() >= 2);
  int ip_hex_size;
  sscanf(client_ip_and_token.substr(0, 2).c_str(), "%d", &ip_hex_size);

I think we need to check the return value here to ensure that sscanf() actually processed one item.
Same for all calls to sscanf() below.

src/cpp/server/load_reporter/load_reporter.cc, line 211 at r6 (raw file):

    // Convert to padded hex string for a 32-bit LB ID. E.g, "0000ca5b".
    char buf[kLbIdLength];
    sprintf(buf, "%08lx", lb_id);

Suggest using snprintf() for safety.

Comments from Reviewable

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

Review status: all files reviewed, 4 unresolved discussions (waiting on @markdroth, @isturdy, @summerxyt, and @nicolasnoble)

src/cpp/server/load_reporter/load_data_store.h, line 127 at r1 (raw file):
Previously, markdroth (Mark D. Roth) wrote…
Even if the opencensus API requires doubles, I don't see why we need to use that type in our code.  We should use int64 here and convert to/from double at the points at which we interact with opencensus.

I see. Done.

src/cpp/server/load_reporter/load_data_store.cc, line 83 at r6 (raw file):
Previously, markdroth (Mark D. Roth) wrote…
I think we need to check the return value here to ensure that sscanf() actually processed one item.
Same for all calls to sscanf() below.

Done.

src/cpp/server/load_reporter/load_reporter.cc, line 211 at r6 (raw file):
Previously, markdroth (Mark D. Roth) wrote…
Suggest using snprintf() for safety.

Done.
I didn't realize that sprintf() is null-terminated. Thanks!

Comments from Reviewable

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

[microbenchmarks] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

This looks pretty good -- just one issue left.

Reviewed 2 of 4 files at r7, 1 of 1 files at r8, 2 of 2 files at r9.
Review status: all files reviewed, 2 unresolved discussions (waiting on @markdroth, @isturdy, @summerxyt, and @nicolasnoble)

src/cpp/server/load_reporter/load_data_store.h, line 127 at r1 (raw file):
Previously, AspirinSJL (Juanli Shen) wrote…
I see. Done.

Can we also change the other args that use doubles to use ints?

Comments from Reviewable

Thanks for reviewing!

Review status: all files reviewed, 5 unresolved discussions (waiting on @markdroth, @isturdy, @yetianx, @summerxyt, and @nicolasnoble)

src/cpp/server/load_reporter/load_data_store.h, line 127 at r1 (raw file):
Previously, markdroth (Mark D. Roth) wrote…
Can we also change the other args that use doubles to use ints?

Done.
Theses are all ints from the channel filter.

src/cpp/server/load_reporter/load_data_store.h, line 34 at r9 (raw file):
Previously, yetianx wrote…
Also move constant.h?

Done.

src/cpp/server/load_reporter/load_data_store.cc, line 104 at r9 (raw file):
Previously, yetianx wrote…
Does this ASSERT terminate the server?
It might be better to log an warning or error instead of crashing the server.

Done.
Also, apparently I had some misunderstanding about the endianness here. I just added back the hton conversion.

src/cpp/server/load_reporter/load_reporter.cc, line 356 at r9 (raw file):
Previously, yetianx wrote…
Use emplace_back?

Done.
I was too lazy to add ctor to the struct.

src/cpp/server/load_reporter/load_reporter.cc, line 379 at r9 (raw file):
Previously, yetianx wrote…
nit: It might be better to split this function into 4 pieces as your comment suggests. And you have a helper function for each piece. Then each helper function is within 50 lines.

Done.

Comments from Reviewable

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

This now builds on open PR #15799.

****************************************************************

libgrpc.so

     VM SIZE                                                            FILE SIZE
 ++++++++++++++ GROWING                                              ++++++++++++++
  +0.0%    +128 [None]                                                  +432  +0.0%
  +0.9%     +32 src/core/lib/iomgr/socket_utils_common_posix.cc          +32  +0.9%
       +13%     +22 [Unmapped]                                               +22   +13%
      [NEW]      +7 grpc_ntohs(unsigned short) [clone .localalias.3]          +7  [NEW]
      [NEW]      +5 grpc_htonl                                                +5  [NEW]
      [NEW]      +5 grpc_ntohl                                                +5  [NEW]

  +0.0%    +160 TOTAL                                                   +464  +0.0%

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE                                                            FILE SIZE
 ++++++++++++++ GROWING                                              ++++++++++++++
  +0.0%    +128 [None]                                                  +432  +0.0%
  +0.9%     +32 src/core/lib/iomgr/socket_utils_common_posix.cc          +32  +0.9%
       +13%     +22 [Unmapped]                                               +22   +13%
      [NEW]      +7 grpc_ntohs(unsigned short) [clone .localalias.3]          +7  [NEW]
      [NEW]      +5 grpc_htonl                                                +5  [NEW]
      [NEW]      +5 grpc_ntohl                                                +5  [NEW]

  +0.0%    +160 TOTAL                                                   +464  +0.0%

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc.so

     VM SIZE                                                            FILE SIZE
 ++++++++++++++ GROWING                                              ++++++++++++++
  +0.0%    +128 [None]                                                  +432  +0.0%
  +0.9%     +32 src/core/lib/iomgr/socket_utils_common_posix.cc          +32  +0.9%
       +13%     +22 [Unmapped]                                               +22   +13%
      [NEW]      +7 grpc_ntohs(unsigned short) [clone .localalias.3]          +7  [NEW]
      [NEW]      +5 grpc_htonl                                                +5  [NEW]
      [NEW]      +5 grpc_ntohl                                                +5  [NEW]

  +0.0%    +160 TOTAL                                                   +464  +0.0%

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

[microbenchmarks] No significant performance differences

[microbenchmarks] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE                                                            FILE SIZE
 ++++++++++++++ GROWING                                              ++++++++++++++
  +0.0%    +128 [None]                                                  +432  +0.0%
  +0.9%     +32 src/core/lib/iomgr/socket_utils_common_posix.cc          +32  +0.9%
       +13%     +22 [Unmapped]                                               +22   +13%
      [NEW]      +7 grpc_ntohs(unsigned short) [clone .localalias.3]          +7  [NEW]
      [NEW]      +5 grpc_htonl                                                +5  [NEW]
      [NEW]      +5 grpc_ntohl                                                +5  [NEW]

  +0.0%    +160 TOTAL                                                   +464  +0.0%

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

All the failures seen have been fixed. This is ready for review again.

Other than the outstanding question for Nico about build/test integration, this looks good!

Reviewed 2 of 6 files at r10, 3 of 4 files at r11, 3 of 10 files at r12, 7 of 7 files at r13, 6 of 6 files at r14.
Review status: all files reviewed, 4 unresolved discussions (waiting on @markdroth, @yetianx, @summerxyt, and @nicolasnoble)

build.yaml, line 1894 at r1 (raw file):
Previously, AspirinSJL (Juanli Shen) wrote…
I keep this for now because it seems that we still need some discussion about this? See https://github.com/grpc/grpc/pull/15349#discussion_r188754176

Still need feedback from @nicolasnoble about this.

Comments from Reviewable

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

Cool! I've cleaned it up. This PR is ready for a final review now. @yetianx @markroth.

Review status: 17 of 18 files reviewed, 4 unresolved discussions (waiting on @markdroth, @yetianx, @summerxyt, and @nicolasnoble)

build.yaml, line 1894 at r1 (raw file):
Previously, markdroth (Mark D. Roth) wrote…
Still need feedback from @nicolasnoble about this.

Done. @nicolasnoble says the Bazel tests are running well. So I have removed these targets here.

Comments from Reviewable

[trickle] No significant performance differences

Squashing. (Heads-up to @yetianx)

[microbenchmarks] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

I have one small change between approval and squashing.
We already have OpenCensus dependency set up after #15070. So I removed my change to WORKSPACE and followed #15070 to add an external_deps.

[trickle] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

[microbenchmarks] No significant performance differences

Basic Tests MacOS [opt] (internal CI): #15027


Remove old "jmc-latest" job

The following Jenkins Job is no longer needed and can be removed:
https://ci.adoptopenjdk.net/view/JMC/job/jmc-latest/

Done


Should petr4 give an error for program with 2 actions, same name & signature, different bodies?

I do not see how the attached program should not be ambiguous or give some kind of error, unless the intent is that the later definition of action foo is supposed to override or shadow the earlier one.  Even if that is the case, the JSON output by petr4 includes both definitions of foo, which doesn't seem correct.
Related to these 2 issues on p4c, but the program attached to this issue is slightly different than any of the ones on those issues:

https://github.com/p4lang/p4c/issues/1932
https://github.com/p4lang/p4c/issues/1936

The latest p4test as of 2019-May-10 does give an error message when attempting to compile the attached program.
actions-same-name3.p4.txt

Similarly, two functions, controls, or parsers with the same signature but different bodies cause the latest p4test as of 2019-May-10 to give errors, but petr4 quietly accepts them.  Attaching the programs for reference.
parsers-same-name1.p4.txt
functions-same-name1.p4.txt
controls-same-name1.p4.txt

This public version of petr4 (we have a development version that is private) only includes the parser. So what these examples show, assuming it's not a bug in our parser, is that the programs that the open-source C++ code is rejecting are being rejected in semantic analysis, not in the parser (which I believe Mihai conjectured).
We do have a type-checker in preparation, which we plan to release soon. It should give a complete front-end equivalent to the open-source C++ code.

Got it.  I had been guessing that the current public petr4 had some semantic analysis as well.  I will hold off on creating more issues like this unless I find something that looks amiss in petr4's syntax parsing.

Actually, this comment suggests that at least one of the examples should be a parse error.
https://github.com/p4lang/p4c/issues/1932#issuecomment-490982424
It may well be a bug in our parser...

The way I understood that comment of Mihai's is that he wishes some of those error messages were better, but it is difficult to make them clearer the way they use Bison and the lexer, I think by installing defined type names as new recognized lexer symbols.

Closing as a duplicate of #135


Updating the oblivious driver example (BFV) to use actual lat/long data (signed data)

Hi,
I think I am right in saying that BFV is not capable of doing math on encrypted floating point numbers. For my own education really (and perhaps others) I am trying to update the example to take a lat/long coordinate for the rider's location, convert this to signed integers and then continue the process to work out which driver is closest to the rider.

The first thing I have done is set the lat/long of the rider
Then I convert the lat/long to an integer, by multiplying by 1000 and rounding.
Next I have set each driver's location to something similar to the rider +/- a bit of randomness
Due to the fact that my values can be negative (lat/long can be negative) I have updated the example to use int64 instead of uint64.
Finally I need to calculate the expected and actual distances. This is where I am getting an error.

My code example is below. I believe the issue is on this line:
result := encoder.DecodeIntNew(decryptor.DecryptNew(evaluator.MulNew(RiderCiphertext, RiderCiphertext)))
The issue being its tricky for me to see exactly what that line is doing - but apparently its working out the distance between the rider and every driver...(?) - One question - is there another way to write that so that its more 'mathsy' - i.e a bit more like the distance() function - but for encrypted values, or is the reason its in this format precisely the point of lattice approaches to HE?
I haven't done it yet, but the last step will be to convert the distance back to a float and divide by 1000 to get the actual distance.
One step I am not sure about is whether I can use this encryption method to then send the closest driver the rider's actual location so that they can decrypt it and go there. Can the public key of the driver be used then by the rider to encrypt their location and send it to the driver who can then decrypt it, and is this comparable to RSA or AES encryption in efficiency and security?
This is a really nice example of how you could use HE btw - thanks for providing it, it would be cool if I could get it to use actual lat/long for my own peace of mind! Thank you all!
full code snippet:
package main

import (
	"fmt"
	"math"
	"math/rand"

	"github.com/ldsec/lattigo/v2/bfv"
)

func obliviousRiding() {

	// Number of drivers in the area
	nbDrivers := uint64(5) //max is N //2048 max

	// BFV parameters (128 bit security) with plaintext modulus 65929217
	params := bfv.DefaultParams[bfv.PN13QP218].WithT(0x3ee0001)

	encoder := bfv.NewEncoder(params)

	// Rider's keygen
	kgen := bfv.NewKeyGenerator(params)

	riderSk, riderPk := kgen.GenKeyPair()

	decryptor := bfv.NewDecryptor(params, riderSk)

	encryptorRiderPk := bfv.NewEncryptorFromPk(params, riderPk)

	encryptorRiderSk := bfv.NewEncryptorFromSk(params, riderSk)

	evaluator := bfv.NewEvaluator(params)

	fmt.Println("============================================")
	fmt.Println("Homomorphic computations on batched integers")
	fmt.Println("============================================")
	fmt.Println()
	fmt.Printf("Parameters : N=%d, T=%d, Q = %d bits, sigma = %f \n",
		1<<params.LogN(), params.T(), params.LogQP(), params.Sigma())
	fmt.Println()
	// Rider coordinates [x, y, x, y, ....., x, y]
	riderPosX := int64(math.Round(51.5014 * 1000)) //buckingham palace
	riderPosY := int64(math.Round(-0.1419 * 1000))

	fmt.Printf("riderPosX %d, riderPoxY %d\r\n", riderPosX, riderPosY)
	Rider := make([]int64, 1<<params.LogN())
	for i := uint64(0); i < nbDrivers; i++ {
		Rider[(i << 1)] = riderPosX
		Rider[(i<<1)+1] = riderPosY
	}

	riderPlaintext := bfv.NewPlaintext(params)
	encoder.EncodeInt(Rider, riderPlaintext)

	// driversData coordinates [0, 0, ..., x, y, ..., 0, 0]
	driversData := make([][]int64, nbDrivers)

	driversPlaintexts := make([]*bfv.Plaintext, nbDrivers)

	min := -500 //add some randomness to drivers positions
	max := 500  //add some randomness to drivers positions
	fmt.Println(rand.Intn(max - min) + min)

	for i := uint64(0); i < nbDrivers; i++ {
		driversData[i] = make([]int64, 1<<params.LogN())
		driversData[i][(i << 1)] = riderPosY + int64(rand.Intn(max - min) + min)
		driversData[i][(i<<1)+1] = riderPosY + int64(rand.Intn(max - min) + min)
		driversPlaintexts[i] = bfv.NewPlaintext(params)
		encoder.EncodeInt(driversData[i], driversPlaintexts[i])
	}
	k := 3
	driversData[k] = make([]int64, 1<<params.LogN())
	driversData[k][(k << 1)] = riderPosX
	driversData[k][(k<<1)+1] = riderPosY
	driversPlaintexts[k] = bfv.NewPlaintext(params)
	encoder.EncodeInt(driversData[k], driversPlaintexts[k])

	fmt.Printf("Encrypting %d driversData (x, y) and 1 Rider (%d, %d) \n",
		nbDrivers, riderPosX, riderPosY)
	fmt.Println()

	RiderCiphertext := encryptorRiderSk.EncryptNew(riderPlaintext)

	DriversCiphertexts := make([]*bfv.Ciphertext, nbDrivers)
	for i := uint64(0); i < nbDrivers; i++ {
		DriversCiphertexts[i] = encryptorRiderPk.EncryptNew(driversPlaintexts[i])
	}

	fmt.Println("Computing encrypted distance = ((CtD1 + CtD2 + CtD3 + CtD4...) - CtR)^2 ...")
	fmt.Println()

	evaluator.Neg(RiderCiphertext, RiderCiphertext)
	for i := uint64(0); i < nbDrivers; i++ {
		evaluator.Add(RiderCiphertext, DriversCiphertexts[i], RiderCiphertext)
	}

	result := encoder.DecodeIntNew(decryptor.DecryptNew(evaluator.MulNew(RiderCiphertext, RiderCiphertext)))

	minIndex, minPosX, minPosY, minDist := uint64(0), int64(0), int64(0), float64(0)
	errors := 0

	for i := uint64(0); i < nbDrivers; i++ {

		driverPosX, driverPosY := driversData[i][i<<1], driversData[i][(i<<1)+1]

		computedDist := math.Abs(float64(result[i<<1] + result[(i<<1)+1]))
		expectedDist := math.Abs(float64(distance(driverPosX, driverPosY, riderPosX, riderPosY)))
		fmt.Printf("x %d y %d calculated %.2f expected %.2f\r\n", driverPosX, driverPosY, computedDist, expectedDist)
		if computedDist == expectedDist {
			if computedDist < minDist {
				minIndex = i
				minPosX, minPosY = driverPosX, driverPosY
				minDist = computedDist
			}
		} else {
			errors++
		}

		if i < 4 || i > nbDrivers-5 {
			fmt.Printf("Distance with Driver %d : %8f = (%4d - %4d)^2 + (%4d - %4d)^2 --> correct: %t\n",
				i, computedDist, driverPosX, riderPosX, driverPosY, riderPosY, computedDist == expectedDist)
		}

		if i == nbDrivers>>1 {
			fmt.Println("...")
		}
	}

	fmt.Printf("\nFinished with %.2f%% errors\n\n", 100*float64(errors)/float64(nbDrivers))
	fmt.Printf("Closest Driver to Rider is n°%d (%d, %d) with a distance of %d units\n",
		minIndex, minPosX, minPosY, uint64(math.Sqrt(float64(minDist))))
}

func distance(a, b, c, d int64) int64 {
	if a > c {
		a, c = c, a
	}
	if b > d {
		b, d = d, b
	}
	x, y := a-c, b-d
	return x*x + y*y
}

func main() {
	obliviousRiding()
}

Hello amlwwalker,
The circuit is computing d = (x_i - x)^2 + (y_i - y)^2, hence d should be smaller than your plaintext modulus 65929217 (divided by two if you want to keep the sign). However you encrypt values as large as 51501, so the computation (x_i - 51501)^2 has a high probability of being larger than 65929217. You must either reduce the upper-bound on the inputs or increase the plaintext modulus accordingly (it must be a prime and congruent to 1 mod 2N).

Hello amlwwalker,
The circuit is computing d = (x_i - x)^2 + (y_i - y)^2, hence d should be smaller than your plaintext modulus 65929217 (divided by two if you want to keep the sign). However you encrypt values as large as 51501, so the computation (x_i - 51501)^2 has a high probability of being larger than 65929217. You must either reduce the upper-bound on the inputs or increase the plaintext modulus accordingly (it must be a prime and congruent to 1 mod 2N).

Thanks for your response @Pro7ech - really helpful.
First test was to lower the size of the values and that worked perfectly. I need to keep the accuracy to 3dp (*1000) however so I am attempting to increase the plaintext modulus. The next prime number that is bigger than the possible values I will be feeding it with is<PHONE_NUMBER>. However (from testing) I also can see this is apparently not fulfilling the second requirement
panic: invalid modulus (Modulus[0] != 1 mod 2N)

But my understanding of "congruent to 1 mod 2N" is lacking. What is N in this case and my understanding of congruence is that my prime is equal to the area of a right angle triangle thats sides are made up of rational numbers, however I suspect that isn't the definition here. Is this documented within the repository for me to do some further reading?
Thanks again

Thanks for your response @Pro7ech - really helpful.
First test was to lower the size of the values and that worked perfectly. I need to keep the accuracy to 3dp (*1000) however so I am attempting to increase the plaintext modulus. The next prime number that is bigger than the possible values I will be feeding it with is<PHONE_NUMBER>. However (from testing) I also can see this is apparently not fulfilling the second requirement
panic: invalid modulus (Modulus[0] != 1 mod 2N)

But my understanding of "congruent to 1 mod 2N" is lacking. What is N in this case and my understanding of congruence is that my prime is equal to the area of a right angle triangle thats sides are made up of rational numbers, however I suspect that isn't the definition here. Is this documented within the repository for me to do some further reading?
Thanks again

For T, a plaintext modulus, congruent to 1 mod 2N means that T-1 is divisible by 2N, where N is the degree of your ring (params.N()). This prime should work for N=2^13 (the parameters that you use) : 0x9e188001.
However, the BFV scheme is very sensitive to the size of the plaintext modulus, the larger it is, the more noise is generated and the less operations you can do. 32 bits is quite large for a plaintext modulus, especially for such small parameters, and I'm quite sure multiplication, especially squaring, will fail.
To be fair, the BFV scheme is a good starting point to get introduced to FHE, but as if you are looking to do any serious arithmetic, be it on integers or floating point, the CKKS scheme is the go-to scheme. It is faster and much more versatile than the BFV scheme, but also much harder to use properly, especially for its full RNS variant.  From my experience, the only real life cases where BFV is as good as CKKS is when the circuit is limited to very simple operations (scalar products and additions).
The main downside of CKKS is that it can only encrypt N/2 complex values (although this is not quite true because a real variant allows to encrypt N real values). However, for some circuits, such as computing x^2 + y^2, the imaginary part can be cleverly used double the throughput. For example (x + iy)^2 = x^2 + y^2 + i2xy uses only one complex slot.

For T, a plaintext modulus, congruent to 1 mod 2N means that T-1 is divisible by 2N, where N is the degree of your ring (params.N()). This prime should work for N=2^13 (the parameters that you use) : 0x9e188001.
However, the BFV scheme is very sensitive to the size of the plaintext modulus, the larger it is, the more noise is generated and the less operations you can do. 32 bits is quite large for a plaintext modulus, especially for such small parameters, and I'm quite sure multiplication, especially squaring, will fail.
To be fair, the BFV scheme is a good starting point to get introduced to FHE, but as if you are looking to do any serious arithmetic, be it on integers or floating point, the CKKS scheme is the go-to scheme. It is faster and much more versatile than the BFV scheme, but also much harder to use properly, especially for its full RNS variant.  From my experience, the only real life cases where BFV is as good as CKKS is when the circuit is limited to very simple operations (scalar products and additions).
The main downside of CKKS is that it can only encrypt N/2 complex values (although this is not quite true because a real variant allows to encrypt N real values). However, for some circuits, such as computing x^2 + y^2, the imaginary part can be cleverly used double the throughput. For example (x + iy)^2 = x^2 + y^2 + i2xy uses only one complex slot.

Hi @Pro7ech
Thanks so much for this, yeah I originally looked at CKKS as I knew that it could handle floating points but was a bit much to jump straight in on.
After your advice towards the end (yeah that large prime takes too long to process with the ..... parameters) I ended up with this:
I got a site to generate 50 random lat/longs around Buckingham Palace and it comes with the actual distances between Buck House and the lat/long pair. The following runs smoothly and picks out "Driver 38" as the closest driver. We are using pythagoras which on the earth is inaccurate but I think close enough although makes bug hunting harder :)
It responds with
Closest Driver to Rider is n°38 (51.49, -0.09) with a distance of **330.42 KM** (using pythag). Actual distance is **4.7829 km**

Thats obviously not the same as the actual distance (or close). Im probably making a simple error somewhere but any ideas why that might not be playing ball?
Thanks
package main

import (
	"fmt"
	"github.com/ldsec/lattigo/v2/bfv"
	"math"
)
var latitudes = [...]float64 {51.55505845, 51.35049502, 51.49342404, 51.35576144, 51.46367303, 51.59827867, 51.44975987, 51.41530716, 51.51790999, 51.52325115, 51.37090813, 51.48248203, 51.48224685, 51.57928026, 51.41250126, 51.59095946, 51.55059409, 51.60192801, 51.57055526, 51.43968032, 51.42249985, 51.60153806, 51.44619761, 51.41079237, 51.52254237, 51.32881429, 51.52059371, 51.54308946, 51.42119378, 51.62363479, 51.46497323, 51.49570821, 51.6119193, 51.53123382, 51.60930381, 51.61608966, 51.5484012, 51.55255689, 51.4864284, 51.63758953, 51.43994719, 51.53815565, 51.46523161, 51.5708527, 51.59856109, 51.45217381, 51.34832873, 51.38392884, 51.63273957, 51.59322371}
var longs = [...]float64 {-0.03196547,-0.19744777, 0.03596891,-0.00561684,-0.06120532,-0.23269885,-0.38122187,-0.32031603, 0.10251308, 0.02074109,-0.22620037,-0.32248626,-0.27654621,-0.32562411,-0.2569608,-0.3761417,-0.32063787,-0.30631856,-0.22557072,-0.22545223,-0.33839268,-0.10436118, 0.07544437,-0.23393792,-0.2254365,-0.13135078, 0.0740383,-0.39272936, 0.09767021,-0.07333396, 0.0615448, 0.05258168,-0.29546331,-0.31173471,-0.00397991,-0.05041521,-0.37625559,-0.08845615,-0.09096628,-0.19127254}
var distances = [...]string {"9.9542 km","17.587 km","13.1179 km","19.5754 km","7.8838 km","11.6607 km","17.0457 km","15.4079 km","17.6936 km","12.1346 km","15.8837 km","12.0684 km","8.9977 km","14.4866 km","12.6845 km","18.1274 km","12.6506 km","15.0644 km","8.7768 km","8.9553 km","15.8929 km","11.1229 km","17.132 km","12.013 km","5.3769 km","19.7726 km","15.7559 km","17.1335 km","19.7342 km","14.1698 km","15.5074 km","14.2388 km","15.3759 km","11.3774 km","15.3979 km","14.1205 km","16.1901 km","6.8024 km","4.7829 km","14.8703 km","13.2825 km","13.7532 km","15.5078 km","17.665 km","18.3112 km","7.142 km","17.7673 km","14.172 km","15.1218 km","12.0783 km"}

func obliviousRiding() {

	// Number of drivers in the area
	nbDrivers := uint64(40) //max is N //2048 max

	// BFV parameters (128 bit security) with plaintext modulus 65929217 --> 0x3ee0001
	// need to be able to find values up to 51000 so need a prime that is<PHONE_NUMBER> or greater
	// from https://github.com/ldsec/lattigo/issues/91 using prime number 0x20008001
	params := bfv.DefaultParams[bfv.PN14QP438].WithT(0x20008001)

	encoder := bfv.NewEncoder(params)

	// Rider's keygen
	kgen := bfv.NewKeyGenerator(params)

	riderSk, riderPk := kgen.GenKeyPair()

	decryptor := bfv.NewDecryptor(params, riderSk)

	encryptorRiderPk := bfv.NewEncryptorFromPk(params, riderPk)

	encryptorRiderSk := bfv.NewEncryptorFromSk(params, riderSk)

	evaluator := bfv.NewEvaluator(params)

	fmt.Println("============================================")
	fmt.Println("Homomorphic computations on batched integers")
	fmt.Println("============================================")
	fmt.Println()
	fmt.Printf("Parameters : N=%d, T=%d, Q = %d bits, sigma = %f \n",
		1<<params.LogN(), params.T(), params.LogQP(), params.Sigma())
	fmt.Println()
	const scaler = 318
	// Rider coordinates [x, y, x, y, ....., x, y]
	riderPosX := int64(math.Round(51.5014*scaler)) //buckingham palace
	riderPosY := int64(math.Round(-0.1419*scaler))

	fmt.Printf("riderPosX %d, riderPoxY %d\r\n", riderPosX, riderPosY)
	Rider := make([]int64, 1<<params.LogN())
	for i := uint64(0); i < nbDrivers; i++ {
		Rider[(i << 1)] = riderPosX
		Rider[(i<<1)+1] = riderPosY
	}

	riderPlaintext := bfv.NewPlaintext(params)
	encoder.EncodeInt(Rider, riderPlaintext)

	// driversData coordinates [0, 0, ..., x, y, ..., 0, 0]
	driversData := make([][]int64, nbDrivers)

	driversPlaintexts := make([]*bfv.Plaintext, nbDrivers)
	for i := uint64(0); i < nbDrivers; i++ {

		driversData[i] = make([]int64, 1<<params.LogN())
		driversData[i][(i << 1)] = int64(math.Round(latitudes[i]*scaler))//driversData[i][i<<1] + xRand
		driversData[i][(i<<1)+1] =int64(math.Round(longs[i]*scaler))
		driversPlaintexts[i] = bfv.NewPlaintext(params)
		encoder.EncodeInt(driversData[i], driversPlaintexts[i])
	}

	fmt.Printf("Encrypting %d driversData (x, y) and 1 Rider (%d, %d) \n",
		nbDrivers, riderPosX, riderPosY)
	fmt.Println()

	RiderCiphertext := encryptorRiderSk.EncryptNew(riderPlaintext)

	DriversCiphertexts := make([]*bfv.Ciphertext, nbDrivers)
	for i := uint64(0); i < nbDrivers; i++ {
		DriversCiphertexts[i] = encryptorRiderPk.EncryptNew(driversPlaintexts[i])
	}

	fmt.Println("Computing encrypted distance = ((CtD1 + CtD2 + CtD3 + CtD4...) - CtR)^2 ...")
	fmt.Println()

	evaluator.Neg(RiderCiphertext, RiderCiphertext)
	for i := uint64(0); i < nbDrivers; i++ {
		evaluator.Add(RiderCiphertext, DriversCiphertexts[i], RiderCiphertext)
	}

	result := encoder.DecodeIntNew(decryptor.DecryptNew(evaluator.MulNew(RiderCiphertext, RiderCiphertext)))

	minIndex, minPosX, minPosY, minDist := uint64(0), int64(0), int64(0), float64(0)
	errors := 0

	for i := uint64(0); i < nbDrivers; i++ {

		driverPosX, driverPosY := driversData[i][i<<1], driversData[i][(i<<1)+1]

		computedDist := math.Abs(float64(result[i<<1] + result[(i<<1)+1]))
		expectedDist := math.Abs(float64(distance(driverPosX, driverPosY, riderPosX, riderPosY)))
		//fmt.Printf("x %d y %d calculated %.2f expected %.2f\r\n", driverPosX, driverPosY, computedDist, expectedDist)
		if computedDist == expectedDist {
			if i == 0 {
				minIndex = i
				minDist = computedDist
			} else if computedDist < minDist {
				minIndex = i
				minPosX, minPosY = driverPosX, driverPosY
				minDist = computedDist
				fmt.Println(i, "minDist ", minDist)
			}
		} else {
			errors++
		}

		if i < 4 || i > nbDrivers-5 {
			fmt.Printf("Distance with Driver %d : %8f = (%4d - %4d)^2 + (%4d - %4d)^2 --> correct: %t\n",
				i, computedDist, driverPosX, riderPosX, driverPosY, riderPosY, computedDist == expectedDist)
		}

		if i == nbDrivers>>1 {
			fmt.Println("...")
		}
	}

	fmt.Printf("\nFinished with %.2f%% errors\n\n", 100*float64(errors)/float64(nbDrivers))
	fmt.Printf("Closest Driver to Rider is n°%d (%.2f, %.2f) with a distance of %.2f KM (using pythag). Actual distance is %s\n",
		minIndex, float64(minPosX)/scaler, float64(minPosY)/scaler, 6371 * math.Sqrt(float64(minDist))/scaler, distances[minIndex])
}

func distance(a, b, c, d int64) int64 {
	if a > c {
		a, c = c, a
	}
	if b > d {
		b, d = d, b
	}
	x, y := a-c, b-d
	return x*x + y*y
}

func main() {
	obliviousRiding()
}

Hi @Pro7ech
Thanks so much for this, yeah I originally looked at CKKS as I knew that it could handle floating points but was a bit much to jump straight in on.
After your advice towards the end (yeah that large prime takes too long to process with the ..... parameters) I ended up with this:
I got a site to generate 50 random lat/longs around Buckingham Palace and it comes with the actual distances between Buck House and the lat/long pair. The following runs smoothly and picks out "Driver 38" as the closest driver. We are using pythagoras which on the earth is inaccurate but I think close enough although makes bug hunting harder :)
It responds with
Closest Driver to Rider is n°38 (51.49, -0.09) with a distance of **330.42 KM** (using pythag). Actual distance is **4.7829 km**

Thats obviously not the same as the actual distance (or close). Im probably making a simple error somewhere but any ideas why that might not be playing ball?
Thanks
package main

import (
	"fmt"
	"github.com/ldsec/lattigo/v2/bfv"
	"math"
)
var latitudes = [...]float64 {51.55505845, 51.35049502, 51.49342404, 51.35576144, 51.46367303, 51.59827867, 51.44975987, 51.41530716, 51.51790999, 51.52325115, 51.37090813, 51.48248203, 51.48224685, 51.57928026, 51.41250126, 51.59095946, 51.55059409, 51.60192801, 51.57055526, 51.43968032, 51.42249985, 51.60153806, 51.44619761, 51.41079237, 51.52254237, 51.32881429, 51.52059371, 51.54308946, 51.42119378, 51.62363479, 51.46497323, 51.49570821, 51.6119193, 51.53123382, 51.60930381, 51.61608966, 51.5484012, 51.55255689, 51.4864284, 51.63758953, 51.43994719, 51.53815565, 51.46523161, 51.5708527, 51.59856109, 51.45217381, 51.34832873, 51.38392884, 51.63273957, 51.59322371}
var longs = [...]float64 {-0.03196547,-0.19744777, 0.03596891,-0.00561684,-0.06120532,-0.23269885,-0.38122187,-0.32031603, 0.10251308, 0.02074109,-0.22620037,-0.32248626,-0.27654621,-0.32562411,-0.2569608,-0.3761417,-0.32063787,-0.30631856,-0.22557072,-0.22545223,-0.33839268,-0.10436118, 0.07544437,-0.23393792,-0.2254365,-0.13135078, 0.0740383,-0.39272936, 0.09767021,-0.07333396, 0.0615448, 0.05258168,-0.29546331,-0.31173471,-0.00397991,-0.05041521,-0.37625559,-0.08845615,-0.09096628,-0.19127254}
var distances = [...]string {"9.9542 km","17.587 km","13.1179 km","19.5754 km","7.8838 km","11.6607 km","17.0457 km","15.4079 km","17.6936 km","12.1346 km","15.8837 km","12.0684 km","8.9977 km","14.4866 km","12.6845 km","18.1274 km","12.6506 km","15.0644 km","8.7768 km","8.9553 km","15.8929 km","11.1229 km","17.132 km","12.013 km","5.3769 km","19.7726 km","15.7559 km","17.1335 km","19.7342 km","14.1698 km","15.5074 km","14.2388 km","15.3759 km","11.3774 km","15.3979 km","14.1205 km","16.1901 km","6.8024 km","4.7829 km","14.8703 km","13.2825 km","13.7532 km","15.5078 km","17.665 km","18.3112 km","7.142 km","17.7673 km","14.172 km","15.1218 km","12.0783 km"}

func obliviousRiding() {

	// Number of drivers in the area
	nbDrivers := uint64(40) //max is N //2048 max

	// BFV parameters (128 bit security) with plaintext modulus 65929217 --> 0x3ee0001
	// need to be able to find values up to 51000 so need a prime that is<PHONE_NUMBER> or greater
	// from https://github.com/ldsec/lattigo/issues/91 using prime number 0x20008001
	params := bfv.DefaultParams[bfv.PN14QP438].WithT(0x20008001)

	encoder := bfv.NewEncoder(params)

	// Rider's keygen
	kgen := bfv.NewKeyGenerator(params)

	riderSk, riderPk := kgen.GenKeyPair()

	decryptor := bfv.NewDecryptor(params, riderSk)

	encryptorRiderPk := bfv.NewEncryptorFromPk(params, riderPk)

	encryptorRiderSk := bfv.NewEncryptorFromSk(params, riderSk)

	evaluator := bfv.NewEvaluator(params)

	fmt.Println("============================================")
	fmt.Println("Homomorphic computations on batched integers")
	fmt.Println("============================================")
	fmt.Println()
	fmt.Printf("Parameters : N=%d, T=%d, Q = %d bits, sigma = %f \n",
		1<<params.LogN(), params.T(), params.LogQP(), params.Sigma())
	fmt.Println()
	const scaler = 318
	// Rider coordinates [x, y, x, y, ....., x, y]
	riderPosX := int64(math.Round(51.5014*scaler)) //buckingham palace
	riderPosY := int64(math.Round(-0.1419*scaler))

	fmt.Printf("riderPosX %d, riderPoxY %d\r\n", riderPosX, riderPosY)
	Rider := make([]int64, 1<<params.LogN())
	for i := uint64(0); i < nbDrivers; i++ {
		Rider[(i << 1)] = riderPosX
		Rider[(i<<1)+1] = riderPosY
	}

	riderPlaintext := bfv.NewPlaintext(params)
	encoder.EncodeInt(Rider, riderPlaintext)

	// driversData coordinates [0, 0, ..., x, y, ..., 0, 0]
	driversData := make([][]int64, nbDrivers)

	driversPlaintexts := make([]*bfv.Plaintext, nbDrivers)
	for i := uint64(0); i < nbDrivers; i++ {

		driversData[i] = make([]int64, 1<<params.LogN())
		driversData[i][(i << 1)] = int64(math.Round(latitudes[i]*scaler))//driversData[i][i<<1] + xRand
		driversData[i][(i<<1)+1] =int64(math.Round(longs[i]*scaler))
		driversPlaintexts[i] = bfv.NewPlaintext(params)
		encoder.EncodeInt(driversData[i], driversPlaintexts[i])
	}

	fmt.Printf("Encrypting %d driversData (x, y) and 1 Rider (%d, %d) \n",
		nbDrivers, riderPosX, riderPosY)
	fmt.Println()

	RiderCiphertext := encryptorRiderSk.EncryptNew(riderPlaintext)

	DriversCiphertexts := make([]*bfv.Ciphertext, nbDrivers)
	for i := uint64(0); i < nbDrivers; i++ {
		DriversCiphertexts[i] = encryptorRiderPk.EncryptNew(driversPlaintexts[i])
	}

	fmt.Println("Computing encrypted distance = ((CtD1 + CtD2 + CtD3 + CtD4...) - CtR)^2 ...")
	fmt.Println()

	evaluator.Neg(RiderCiphertext, RiderCiphertext)
	for i := uint64(0); i < nbDrivers; i++ {
		evaluator.Add(RiderCiphertext, DriversCiphertexts[i], RiderCiphertext)
	}

	result := encoder.DecodeIntNew(decryptor.DecryptNew(evaluator.MulNew(RiderCiphertext, RiderCiphertext)))

	minIndex, minPosX, minPosY, minDist := uint64(0), int64(0), int64(0), float64(0)
	errors := 0

	for i := uint64(0); i < nbDrivers; i++ {

		driverPosX, driverPosY := driversData[i][i<<1], driversData[i][(i<<1)+1]

		computedDist := math.Abs(float64(result[i<<1] + result[(i<<1)+1]))
		expectedDist := math.Abs(float64(distance(driverPosX, driverPosY, riderPosX, riderPosY)))
		//fmt.Printf("x %d y %d calculated %.2f expected %.2f\r\n", driverPosX, driverPosY, computedDist, expectedDist)
		if computedDist == expectedDist {
			if i == 0 {
				minIndex = i
				minDist = computedDist
			} else if computedDist < minDist {
				minIndex = i
				minPosX, minPosY = driverPosX, driverPosY
				minDist = computedDist
				fmt.Println(i, "minDist ", minDist)
			}
		} else {
			errors++
		}

		if i < 4 || i > nbDrivers-5 {
			fmt.Printf("Distance with Driver %d : %8f = (%4d - %4d)^2 + (%4d - %4d)^2 --> correct: %t\n",
				i, computedDist, driverPosX, riderPosX, driverPosY, riderPosY, computedDist == expectedDist)
		}

		if i == nbDrivers>>1 {
			fmt.Println("...")
		}
	}

	fmt.Printf("\nFinished with %.2f%% errors\n\n", 100*float64(errors)/float64(nbDrivers))
	fmt.Printf("Closest Driver to Rider is n°%d (%.2f, %.2f) with a distance of %.2f KM (using pythag). Actual distance is %s\n",
		minIndex, float64(minPosX)/scaler, float64(minPosY)/scaler, 6371 * math.Sqrt(float64(minDist))/scaler, distances[minIndex])
}

func distance(a, b, c, d int64) int64 {
	if a > c {
		a, c = c, a
	}
	if b > d {
		b, d = d, b
	}
	x, y := a-c, b-d
	return x*x + y*y
}

func main() {
	obliviousRiding()
}

The BFV circuit correctly evaluates what it is asked to do, however computing the distance between two latitudes and longitudes isn't the same as computing the distance between two points on a grid.

The BFV circuit correctly evaluates what it is asked to do, however computing the distance between two latitudes and longitudes isn't the same as computing the distance between two points on a grid.

Thanks yeah I knew about that, just didn't expect it to be quite so different!
I can happily update that distance function but the calculatedDistance equation: ((CtD1 + CtD2 + CtD3 + CtD4...) - CtR)^2 ...")  for the circuit, which I think is happening here 	result := encoder.DecodeIntNew(decryptor.DecryptNew(evaluator.MulNew(RiderCiphertext, RiderCiphertext))) is a little more involved - is the Haversine formula possible to turn into a BFV circuit? Can BFV handle the operations?

Thanks yeah I knew about that, just didn't expect it to be quite so different!
I can happily update that distance function but the calculatedDistance equation: ((CtD1 + CtD2 + CtD3 + CtD4...) - CtR)^2 ...")  for the circuit, which I think is happening here 	result := encoder.DecodeIntNew(decryptor.DecryptNew(evaluator.MulNew(RiderCiphertext, RiderCiphertext))) is a little more involved - is the Haversine formula possible to turn into a BFV circuit? Can BFV handle the operations?

No it can't, BFV provides vectorized integer modular arithmetic. You would need to use the CKKS scheme. But even then, it would be much more efficient to first convert from a spherical coordinate system to some other more HE friendly coordinate system.

No it can't, BFV provides vectorized integer modular arithmetic. You would need to use the CKKS scheme. But even then, it would be much more efficient to first convert from a spherical coordinate system to some other more HE friendly coordinate system.


fix: Check for initialized Client in AddBreadcrumbs

Otherwise, we get a segfault at https://github.com/getsentry/sentry-go/blob/master/hub.go#L184 when Client is not initialized (e.g., empty DSN)

We can most likely just store it as is, without checcking for BeforeBreadcrumb and MaxBreadcrumbs as they live on the scope anyway. Will think about it next week.


Bootstrap rework PoC

What this PR does / why we need it:
This pr shows a poc of how it might be possible to rework the appplication installation by removing namespaces and crd installation from our "bootstrap" step.
Which issue this PR fixes
fixes #362
Special notes for reviewer:
CRDs:

While helm allows us to install CRDs, it does nothing in terms of lifecycle management - aka upgrading and removing them.
So, we'll need to devise some other way of handling that.
We can no loger run template --validate in bin/ck8s apply for obvious reasons - the CRDs dosen't exist.
Use apply {wc,sc} --skip-template-validate to skip templating.
We need to disable validation disableValidationOnInstall: true on all releases that'll install a cr.

Namespaces:

I had to remove 00-rbac installation as that one installs roles in namespace that have yet to be created.
I tried to enable psps in all charts that allowed it.

I had to upgrade harbor in order for it to start.
Influxdb does not work without access to priviledged psp atm, I just removed the cronjob sidecar to deploy influxdb.

I did not bother to create a chart or any other means of labling namespaces like our namespaces bootstrap chart did - so no namespace is labled with owner: operator and the namespaces kube-system and gatekeeper-system are not labled withadmission.gatekeeper.sh/ignore: "true".
Fluentd secrets needed to be moved to after the helmfile apply step in the deploy-sc/wc script.
This of course means that we might have some problem in the pipeline where fluentd is crashlooping because it could not mount the secrets immediately.
Gatekeeper is another tricky topic as it installs an validation admission webhook that essentially blocks creation of namespaces until the gatekeeper pods are ready to serve the hook.
I had to make sure that gatekeeper is installed last, thus mitigating the problem that it does not allow for any concurrecnt chart installations that creates a new namespace.
I have essentially done a complete rework of the DAG in helmfile.
It was pretty strange that a lot of releases depended on cert-manager even though they did not create any cert-manager cr.
It would probably to be good to have the DAG on paper for easy visualization.
That's essntially what i did on paper in order for this to work and visualize the dependencies.

Cleanup:

The cleanup scripts will need to be changed.
It does not uninstall crds.

The afformentioned issue mentioned something about timing the bin/ck8s apply step. I have not done that. However, I very much doubt that difference would be anything else than in the order of seconds.
Alertmanager (user):

I did somehting for it so that it does not install psps for it if it is disabled.

Checklist:

[ ] Added relevant notes to WIP-CHANGELOG.md
[ ] Proper commit message prefix on all commits
[ ] Updated the public facing documentation
Is this changeset backwards compatible for existing clusters? Applying:

[ ] is completely transparent, will not impact the workload in any way.
[ ] requires running a migration script.
[ ] will create noticeable cluster degradation.
E.g. logs or metrics are not being collected or Kubernetes API server
will not be responding while upgrading.
[ ] requires draining and/or replacing nodes.
[ ] will change any APIs.
E.g. removes or changes any CK8S config options or Kubernetes APIs.
[ ] will break the cluster.
I.e. full cluster migration is required.

Pipeline config (if applicable)
If you change some config options (e.g. add/rename variable or change the default value) you may need to update the config used by the pipeline in pipeline/config.

Very interesting!
Does helmfile diff still work with this? You only mentioned helmfile template --validate
Labeling namespaces might be something that we want to keep, but it should be easy to do with a chart as you imply.
A DAG of the helmfile would be nice if you could get it in a good format.

Does helmfile diff still work with this? You only mentioned helmfile template --validate

It should still work, see https://github.com/roboll/helmfile/pull/1618 for some explanation of --disableValidationOnInstall.
I mean after CRDs have been installed it should be fine to run template.

It should still work, see roboll/helmfile#1618 for some explanation of --disableValidationOnInstall.
I mean after CRDs have been installed it should be fine to run template.

Right, so helmfile diff does not work before the first installation, but afterwards it does work. That seems fair.

While helm allows us to install CRDs, it does nothing in terms of lifecycle management - aka upgrading and removing them.
So, we'll need to devise some other way of handling that.
We need to disable validation disableValidationOnInstall: true on all releases that'll install a cr.

I think this is enough reason to keep CRDs separate from the rest. We will anyway need a way to upgrade/remove CRDs so why not install them the same way? I.e. just like we do today.
The same is also true for namespaces I'd say. We want to label them, maybe even add annotations in the future, so why make it more complicated by mixing two ways to manage them? With this we would first need to install the relevant helm release before we even have a namespace to label.

Do we still have time as part of this issue?
We just discussed #391 and it seems to me that we are:

Installing too many releases in parallel. Those that "change Kubernetes" by installing webhooks should not run in parallel to anything else.
The bootstrap steps feels like an out-of-helmfile solution to serialization. The Helmfile documentation suggests (and @viktor-f confirms) that separate Helmfile-s are serialized.

Could we instead do the following instead?
00-namespaces.yaml => installs and labels namespaces
01-gatekeeper.yaml => installs Gatekeeper
02-prometheus-operator.yaml
10-cert-manager.yaml
11-nginx-ingress.yaml
50-everything-else.yaml

Do we still have time as part of this issue?
No.

Thanks @OlleLarsson. I created a follow-up issue.

Could we instead do the following instead?

I think this is a good idea. It makes it easier to understand what is going on with less massive files.
However, I'm not sure it solves the issue with webhooks. AFAIK helm is quite limited in what it can wait for so even when it thinks it is done with one helmfile it is no guarantee that the webhooks are ready. If the Pods become "ready" but it still needs time to set up the webhook then this issue can still happen.

@OlleLarsson Can we close this, thanks to #446 ?


Move manual in README to the Wiki (create a Wiki)

The README is getting gigantic. Info should be moved to Wiki as this gives to options of having multiple pages and a menu.

I created the first wiki page. We can clone the wiki locally.


Vite dev doesn't get the correct style when refreshing

Describe the bug
Vite dev doesn't get the correct style when refreshing.
A source code like this
// header.css.ts
export const header = style({
  display: 'flex'
})

generate code like this
import { createHotContext as __vite__createHotContext } from "/@vite/client";import.meta.hot = __vite__createHotContext("/src/components/Header/header.css.ts.vanilla.js");import { injectStyles } from '/node_modules/.vite/deps/@vanilla-extract_css_injectStyles.js?v=bf7f4fdc';

const inject = (css) => injectStyles({
  fileScope: {"filePath":"src/components/Header/header.css.ts.vanilla.js"},
  css
});

inject(".header__16c76pf0 {\n  display: flex;");

import.meta.hot.on('vanilla-extract-style-update:src/components/Header/header.css.ts.vanilla.js', (css) => {
  inject(css);
});

It seems that when changing the code, vite-plugin only runs inject(css) in meta.hot.on but does not change the file content. When refreshing the page, the original style will show up which was built in vite first startup.
<EMAIL_ADDRESS>

same as https://github.com/seek-oss/vanilla-extract/issues/578#issuecomment-1133020076

It turns out there is a bug in @vanillia-extract/vite-plugin
https://github.com/seek-oss/vanilla-extract/blob/aebf9349371d631a38145df70c7de40643843554/packages/vite-plugin/src/index.ts#L166
The correct id should be path.resolve('.', id) (which is absolute path) not id (relative path like src/components/...). Not sure it is vite's bug or plugin's bug.


Implements proposal by @snahor in #32 to make xsml track more consistent

Implements proposal by @snahor in #32 to make xsml track more consistent
Summary of changes:

All test files have been refactored in a consistent manner and provide output about which specific tests failed to the user without needing them to enter a REPL
All stub files now have type signatures to give the user hints about the expect input and output of each function
The SETUP.md file has been updated to reflect the new testing methodology

@mcmillhj @snahor I am so sorry for missing this! I had surgery in late February, and a lot of things have fallen through the cracks in the months since then.
Would the two of you be interested in having commit on this track? It looks like you've both contributed a fair amount. I would now be around to help get the conflicts sorted out.

By the way—maintaining the track wouldn't necessarily mean doing more than you're already doing, it would mostly mean that between the two of you you can accept the changes into master!
I have some docs about maintaining tracks here: https://github.com/exercism/docs/blob/master/maintaining-a-track/README.md

No worries.  I would be happy to manage this track though.

Thanks @mcmillhj! You're far more of an expert than I am, though admittedly the bar there is low :) 🌻

Could you split it in smaller PRs? it would be easier to review.

Yeah, I can do that. Probably wont be able to do so today, but in the next day or so I should have it up.

Closing this PR, going submit separate PRs for the accumulate, allergies, anagram, binary, flatten-array, hamming, nth-prime, and raindrop exercises.


Release version 0.2.1

PR for tracking changes for the 0.2.1 hotfix release. Target release date: Sunday, November 17.

[ ] Add changelog for 0.2.1 - PR #235
[ ] Version number updated in the docblock for the fix added in the PrefixAllGlobals sniff - PR #234
[ ] Merge this PR
[ ] Add release tag against master (careful, GH defaults to develop!) & copy & paste the changelog to it
[ ] Tweet about the release.
[ ] Post about it in Slack.
[ ] Submit for "Month in WordPress": https://make.wordpress.org/community/month-in-wordpress-submissions/

@jrfnl Should I bump the versions in the composer for WPCS now that 2.2.0 is out?

@jrfnl Should I bump the versions in the composer for WPCS now that 2.2.0 is out?

LOL, just left a comment elsewhere suggesting the same, so :+1: from me.

FYI: I've moved the PRs which were merged before and will be included in this release to the 0.2.1 milestone...


Testing VideoVista

We have tested your model on our video Benchmark (VideoVista), which achieved a superior performance.
Yet, we did not see the paper address corresponding to your model Video-CCAM.
When are you going to release it?  Could we discuss it further?
Best,
Yunxin Li
E-mail<EMAIL_ADDRESS>

Thank you for your evaluation on our work! Currently, the project is still undergoing revisions and there is no clear timeline for the release of the paper. Once the model's performance has been improved to a satisfactory level, we will release the paper and open-source all the training code and training data.


Prevent adding the same line and correct the starting position of offset

Here's a checklist for things that will be checked during review or continuous integration.

[x] Added passing unit tests
[x] cargo test passes locally. It takes much time.
[x] Run cargo fmt

changelog:
close #47

I made an issue for linux-arm build, and It merge...
https://github.com/kyoto7250/zhobo/issues/49


Use an explicit empty object array to prevent the varargs method allocating one

In some of the profiles we are seeing a lot of Object[] allocations from this method. I think the varargs method is allocating a new empty Object[] array for every call.
In master this has been changed to use Field rather than Method so this PR is not applicable.

Applies as HHH-10338, but on 5.0 only: apparently this code was removed on master (!?)

https://hibernate.atlassian.net/browse/HHH-10338


MNT: If an EventDescriptor has a 'name' it should be a string.

The 'name' field in EventDescriptors was introduced late and has always been
optional. This change leaves it optional, but requires that if the 'name' field
is present, that it is a string. We make this assumption in a couple places
downstream so it would be good to enforce it here.

looks good to me, would merge it but don't yet have write access to event-model !-)

@awalter-bnl Now you do.
Nevertheless I would like @tacaswell eyes on one this before merging, as he was also there for the discussions that introduced 'name' and might spot a pitfall with this change that I have not anticipated.


cannot open .raw file in mzmine3 nor project made from .raw files

Basic information

My operating system: windows10
My MZmine version: mzmine 3.1.0 beta

What happened
I was creating the project with .raw data import, it was working just fine, than I was trying to export data directly to gnps, I was trying different things since I was not successfu. In one moment mzmine freezed, I end task in task manager, when I wasnted to reopen the project or import .raw I get the error:

Hi Ana,
could you share one .raw file that is failing for you?
Also, could you try this on another PC?

Hi Tomaš,
This error appeared while I was working on my working stationary PC, I've
tried it on my laptop, and the same error occurred.
Project and .raw data with which I am working are stored on my google drive.
Then I tried on my home stationary PC where I have data on the computer and
it worked but when I copied the folder containing everything on the
computer at work, it again did not work.
Actually, none of the projects created from .raw files is working.
Please, find attached one .raw file.
Thank you for your help
Blank_1.raw
https://drive.google.com/file/d/1p2pkDSa4woCR0AvXw4eqZJZj_KtDHdnC/view?usp=drive_web
.
Best
Ana
On Thu, Jul 14, 2022 at 8:48 AM Tomáš Pluskal @.***>
wrote:

Hi Ana,
could you share one .raw file that is failing for you?
Also, could you try this on another PC?
—
Reply to this email directly, view it on GitHub
https://github.com/mzmine/mzmine3/issues/789#issuecomment-1184061875,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARHCSNMA2LBPSADZ5UKOTHTVT6Z3LANCNFSM53RECBQQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

I tested your file and it works fine for me, so there should be nothing wrong with the .raw files.
MZmine is internally using the ThermoRawFileParser library.
Could you please try to download and unzip the library from here: https://github.com/mzmine/mzmine3/raw/master/src/main/resources/vendorlib/thermo/ThermoRawFileParser.zip
and run this command in the command line:
ThermoRawFileParser.exe -i  Blank_1.raw

I did as suggested, it seems like it reads the file:
C:\Users\Ana>ThermoRawFileParser\ThermoRawFileParser.exe -i Blank_1.raw
2022-07-14 11:08:03 INFO Started parsing Blank_1.raw
2022-07-14 11:08:03 INFO Processing 7577 MS scans
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
2022-07-14 11:08:08 INFO Finished parsing Blank_1.raw
but the mzmine.exe app is still not working, giving the same error.
Also, it was working with no problem for 1 month, I haven't done sth that
would change the .raw files and I have tried with the ones from the local
disk as well.
The reinstallation of mzmine but still not working.
With mzml, everything there is no problem, I just don't want to lose the
project but luckily it is working on my third computer :).
Thank you.
Best
Ana
On Thu, Jul 14, 2022 at 9:16 AM Tomáš Pluskal @.***>
wrote:

I tested your file and it works fine for me, so there should be nothing
wrong with the .raw files.
MZmine is internally using the ThermoRawFileParser library.
Could you please try to download and unzip the library from here:
https://github.com/mzmine/mzmine3/raw/master/src/main/resources/vendorlib/thermo/ThermoRawFileParser.zip
and run this command in the command line:
ThermoRawFileParser.exe -i Blank_1.raw
—
Reply to this email directly, view it on GitHub
https://github.com/mzmine/mzmine3/issues/789#issuecomment-1184084197,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARHCSNMQDNJPYURFE35WRT3VT65GPANCNFSM53RECBQQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

Hi Ana, thanks for the update.
Could you please go to the MZmine menu Help -> Show log file and upload the complete log file here? (after you get the error message, of course).

Sure,
Please find attached the log file.
Thank you.
Ana
On Thu, Jul 14, 2022 at 11:35 AM Tomáš Pluskal @.***>
wrote:

Hi Ana, thanks for the update.
Could you please go to the MZmine menu Help -> Show log file and upload
the complete log file here? (after you get the error message, of course).
—
Reply to this email directly, view it on GitHub
https://github.com/mzmine/mzmine3/issues/789#issuecomment-1184221378,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARHCSNLWQG6RX2QPLFHJVF3VT7NNJANCNFSM53RECBQQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

mh thats weird.
please upload an example project that reproduces the issue.

Agree.
I thought it was sth with google drive, but then I copied the folder
containing raw files and project on the computer C disk, and the same error
appeared.
Please, find the link to the project:
https://drive.google.com/file/d/1xdPJAhyWQHjypXsNtY-uKpRETcUMbhf3/view?usp=sharing
On Thu, Jul 14, 2022 at 3:25 PM Steffen Heuckeroth @.***>
wrote:

mh thats weird.
please upload an example project that reproduces the issue.
—
Reply to this email directly, view it on GitHub
https://github.com/mzmine/mzmine3/issues/789#issuecomment-1184446070,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARHCSNPNOAYPXGQ7RMITETTVUAIKZANCNFSM53RECBQQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

Hi Ana, could you please try to delete this folder:
C:\Users\Ana\AppData\Local\Temp\mzmine_thermo_raw_parser
and after you delete it, try to run MZmine and import the files again?

The project imports fine for me.

could it be sth with the parser, tomas?

I believe the import issue should be fixed in my last pull request: https://github.com/mzmine/mzmine3/pull/790
That will probably also fix the project import.

Tomas,
that is correct, after deleting the suggested folder, the project is
working normally.
Thank you.
Ana
On Fri, Jul 15, 2022 at 10:29 AM Tomáš Pluskal @.***>
wrote:

I believe the import issue should be fixed in my last pull request: #790
https://github.com/mzmine/mzmine3/pull/790
That will probably also fix the project import.
—
Reply to this email directly, view it on GitHub
https://github.com/mzmine/mzmine3/issues/789#issuecomment-1185307922,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARHCSNKRZUYNGKTLMXB3EBTVUEONHANCNFSM53RECBQQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

Great work. Seems to be fixed.


Stack using LinkedList implementation added

Checklist

[x] I've looked at the contribution guidelines.
[x] This pull request is complete and ready for review.

Description

Hello.
I've added LinkedList implementation for Stack data structure. And added tests for linked list stack.

@n0an same reason as #666. It would be nice to separate the linkedlist and the stack. Have the stack call various methods of a linked list.

@vincentngo , I've made the same changes as in #666 . Should I change something? Maybe it's better to adopy StackProtocol by LinkedLIst class in LinkedList target? But I don't know will it be readable this way.


Implement select share (secureNN) (#154)

Implements select share in the secureNN protocol and adds a test

Looks great, maybe add an issue about updating the zeros call once we have the Abstract tensor etc. in place.

Created #159.


Error detected when opening dir in Vim

Whenever I open NT up with a dir I am hit with this error, but then everything seems to load fine after that.
"bin" is a directory
Error detected while processing function <SNR>190_startup:
line    3:
E85: There is no listed buffer

I'm not even sure where this error is originating from as I do not know Vimscript well at all, but maybe this is a candidate? Any tips would be greatly appreciated!
" /vim-nerdtree-tabs/nerdtree_plugin/vim-nerdtree-tabs.vim

" open NERDTree on gvim/macvim startup
if !exists('g:nerdtree_tabs_open_on_gui_startup')
  let g:nerdtree_tabs_open_on_gui_startup = 1
endif

Same here, and without OP's snippet.

The startup function is not defined in nerdtree. We can find out which file it is defined by:

reproduce the error and note the function name (it is <SNR>190_startup in your example)
note the number in the function name - 190 in the example
run :scriptnames (shows the files that vim has sourced so far). Scroll down until you find the script with the number from the function. This is where the startup func is defined.

Can anyone tell me what script it is?

Closed due to no response from issue's creator.


Create Interdomain version of NSE Composition example

Overview
Create Interdomain version of NSE Composition example. The first half of NSEs should be on the first cluster and the second half should be on the second cluster.
Steps

Deploy basic intedomain NSM setup
Deploy nse-composition service on the third cluster
Deploy nse-firewall and nse-passthrough-1 on the first cluster
Deploy nse-passthrough-2, nse-passthrough-3 and nse-kernel on the second cluster
Deploy nsc-kernel which wants<EMAIL_ADDRESS>service
Ping from nsc-kernel to nse-kernel

Diagram

@NikitaSkrynnik Could you re do diagram with https://github.com/networkservicemesh/deployments-k8s/edit/main/examples/multicluster/usecases/floating_vl3-basic/README.md?


Update policy-csp-authentication.md

Removing additional warning for web sign-in to reflect that it is now supported/out of private preview.
If a warning for EnableFastFirstSignIn is needed, language should be re-added with appropriate policy scope.

Why

Closes #[Issue Number]

Changes

Learn Build status updates of commit 4cd1fc3:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

windows/client-management/mdm/policy-csp-authentication.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

Thanks @tilarso for the contribution! We're actually working to automate this content based on what's in the DDF XML, so I cherry picked your change over to our work-in-progress in the private repository (see https://github.com/MicrosoftDocs/windows-docs-pr/pull/7585), and will close this PR. You won't see the change live for ... maybe a couple of weeks. But rest assured, it's captured! 😄

@tilarso you can also see the draft version of the new article here with your changes: https://review.learn.microsoft.com/en-us/windows/client-management/mdm/policy-csp-authentication?branch=pr-en-us-7585#enablefastfirstsignin


Adding more references to current maximum number of data networks supported

A support engineer could not find information in the public docs about the maximum number of supported DNs. It is currently only mentioned in "Collect information for a site", so I've added references on other pages where support for multiple data networks is discussed.

#sign-off