[SPARK-30202][ML][PYSPARK] impl QuantileTransform

What changes were proposed in this pull request?
Impl QuantileTransform, a non-parametric transformation to map the data to another
distribution
The impl followed scikit-learn' s impl, however there still are sereral differences:
1, use QuantileSummaries for approximation, no matter the size of dataset;
2, use linear interpolate, the logic is similar to existing IsotonicRegression, while scikit-learn use a bi-directional interpolate (the two methods only differ on a special case that  values are repeated in the features);
3, treat sparse vectors just like dense ones, while scikit-learn have two different logics for sparse and dense datasets.
Why are the changes needed?
1, it is common to map the data to another desired distribution, and was already impled in scikit-learn
2, it is easy for parallelism
Does this PR introduce any user-facing change?
Yes, a new model
How was this patch tested?
added testsuites

Test build #115102 has started for PR 26832 at commit 7572919.

Test build #115102 has finished for PR 26832 at commit 7572919.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaQuantileTransformExample
class QuantileTransform(override val uid: String)
 class QuantileTransformModelWriter(instance: QuantileTransformModel) extends MLWriter
class _QuantileTransformParams(HasInputCol, HasOutputCol, HasRelativeError):
class QuantileTransform(JavaEstimator, _QuantileTransformParams, JavaMLReadable, JavaMLWritable):
class QuantileTransformModel(JavaModel, _QuantileTransformParams, JavaMLReadable, JavaMLWritable):

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115102/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19916/
Test PASSed.

Test build #115146 has started for PR 26832 at commit a2d2c51.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19959/
Test PASSed.

Test build #115146 has finished for PR 26832 at commit a2d2c51.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115146/
Test PASSed.

Test build #115183 has started for PR 26832 at commit b998ef3.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19994/
Test PASSed.

Test build #115183 has finished for PR 26832 at commit b998ef3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115183/
Test PASSed.

I dont' really object to it, but how widely used would this be?
I like some of the functionality we're adding to MLlib even though it's kind of in maintenance mode, but don't want it to get too sprawling.

@srowen Sorry to not discuess this alg before this PR.
Distribution transformation is widely used in traditional statistical analysis, and it is impled in many tools like sklearn/r/sas/spss/matlab/etc.
Recently my team encounts some cases in which gaussian distribution is a strict prerequisite. (Actually many algorithms implicitly suppose a gaussian distribution dataset)
There is a famous method named [BOX-COX](https://en.wikipedia.org/wiki/Power_transform#Box%E2%80%93Cox_transformation) for this,  however it seems hard to implement in a distributed system (I am still working on it), so I switch to this non-parametric methods. I think it will be helpful since no such method is provided now.

I like some of the functionality we're adding to MLlib even though it's kind of in maintenance mode, but don't want it to get too sprawling.

Yes, I think we may discuss some roadmap in JIRA. However, I notice that the bandwidth for MLLIB is quite limited.

the code on Quantile Computation are similar to RobustScaler, I will try to make it common.

Test build #115542 has started for PR 26832 at commit 4a49a8e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20345/
Test PASSed.

Test build #115542 has finished for PR 26832 at commit 4a49a8e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115542/
Test PASSed.

Test build #115760 has started for PR 26832 at commit da4313e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20554/
Test PASSed.

Test build #115760 has finished for PR 26832 at commit da4313e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115760/
Test PASSed.

Test build #121412 has started for PR 26832 at commit 25c54b2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26095/
Test PASSed.

Test build #121412 has finished for PR 26832 at commit 25c54b2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121412/
Test PASSed.


migrate docs from Docz to Docusaurus

This PR migrates the documentation from Docz to Docusaurus.
A couple of comments:

I've added a favicon.
The examples don't require copies of the source code files with a file suffix .txt anymore for showing the code in the documentation.
I don't use a <Playground />-like component in the examples anymore because I think it doesn't add any value to show the source code of only the component call.
I use Bootstrap Icons as an icon library. It's a large collection of beautiful icons, has a permissive license, and the package just contains SVGs that are loaded via svgr, so there's no need for any special React icon (wrapper) library, e.g. react-fontawesome or @mui/icons-material.
In patches.mdx, snapshots.mdx, and treeLikeStructure.mdx, I've replaced headings with (sometimes) long function signatures by just function names and block code right below containing the function signatures because the headings were difficult to read in the on-page sidebar on the right. What do you think about this change?
I've kept the URL paths identical to the Docz-based documentation. However, I think it would be a bit nicer to use lower case characters and dashes to separate words. This would break existing links to the docs though unless we find a way to create redirects from the current to the new URLs.
I've not added a search box yet and would postpone it to a follow-up PR. Would you prefer using the Algolia search service or a local one? For local search, I think https://github.com/cmfcmf/docusaurus-search-local looks like the best option.

Let me know what you think. :slightly_smiling_face:
Closes #341.

that looks awesome! 👍I'll take a deeper look as soon as my laptop is repaired 😅

Just had the chance to look at it. it is awesome!! ❤

I've kept the URL paths identical to the Docz-based documentation. However, I think it would be a bit nicer to use lower case characters and dashes to separate words. This would break existing links to the docs though unless we find a way to create redirects from the current to the new URLs.

Feel free to change them

I've not added a search box yet and would postpone it to a follow-up PR. Would you prefer using the Algolia search service or a local one? For local search, I think https://github.com/cmfcmf/docusaurus-search-local looks like the best option.

I prefer local, but if it is too annoying to setup the algolia would be ok too.

By the way, I'm gonna merge it already, but feel free to open another PR to add search / fix urls!

Awesome! :slightly_smiling_face: Thanks also for fixing the Netlify config, I had forgotten about it.

By the way, what happened to the preview deployments of the docs on Netlify?

Thanks for this, it's beautiful!
And looking forward to the full text search!


Improve error message for when a port already exists on a module

Motivation
If exceptions are not being debugged, it can be difficult to identify where the UnavailableReservedNameException is coming from (e.g. which module).
Desired solution
In throwing the UnavailableReservedNameException in _checkForSafePortName, also communicate the module name.
Alternatives considered
No response
Additional details
No response

I would like to work on this!


The effect of Linux -O3 on io_uring throughput and latency

Hi @axboe :)
Recently there was talks on LKML about making it easier to build Linux with GCC -O3 optimization flag (which is, contrary to popular belief perfectly safe, unlike say -ffastmath)
The great Michael Larabel recently benchmarked it, and while the benchmarks do not show performance improvements for an average program, one could argue the average program isn't bottlenecked by Kernel CPU time.
However the ones that do might see very significant performance increases for free, especially programs that are heavy on IO syscalls.
PostgreSQL see 11% improvement! And
Stress-NG context switching benchmark see a crazy 269% performance improvement.
Unfortunately Michael has not benchmarked O3 on an io_uring benchmark. It might be that the improvement is mild but it is also a possibility that it would help unlock even more performance than current io_uring.
https://www.phoronix.com/vr.php?view=31250
As such I thought you might be interested in benchmarking it.

For my peak IOPS runs that I've done in the past, I do use -O3 for the kernel. The branch is updated by now, but my perf-wip branch includes a patch that enables that and has so for a long time:
https://git.kernel.dk/cgit/linux-block/commit/?h=perf-wip&id=a2c35cfa21d60051881820fbcf4f5c9089eefe48
I saw nice wins from that too.
I'd be fine adding -O3 for liburing, we already do so for fio and hence the t/io_uring tool I use for peak IOPS testing too. That's been the case for many years in that repo.

https://git.kernel.dk/cgit/liburing/

Was there any hints from that work @axboe on what optimizations actually made a difference, generating better code? I wonder if there were hints that could then be applied to the actual source code to be smarter?

I didn't investigate, I did post on some gcc 11.2 vs 12 vs clang, but more from a curiosity POV in what worked the best for my test cases.
I'm sure there are hints and code changes that could be made that would make O2 as good, but that was a much more lengthy project that I didn't have the time to undertake.


CoreLabel sample code, causes segfault

Software Versions

Python: 3.9.9
OS: window 10
Kivy: 2.0
Kivy installation method: conda

Sample code in kivy.core.text causes a segmentation fault.
https://kivy.org/doc/stable/api-kivy.core.text.html#module-kivy.core.text
Expected behavior
I expected to get the texture size in pixels.
To Reproduce
I ran the code below in vscode and ipython terminal.
from kivy.core.text import Label as CoreLabel
my_label = CoreLabel()
my_label.text = 'hello'
my_label.refresh()
hello_texture = my_label.texture
print(hello_texture.text_size())

Additional context
My intention from the code above is to get the string size in pixels for a given font and size.

Hi @abedhammoud !
CoreLabel uses Texture, and Texture needs the following:
Using Texture before OpenGL has been initialized will lead to a crash. If
you need to create textures before the application has started, import
Window first: `from kivy.core.window import Window`

The CoreLabel examples are meant to be used into an application, when Window is started.

Thanks, @misl6. I assumed the code in the docs was a fully contained example. It is clear now - thanks.


Update Go code to new interfaces

See:

https://github.com/senzing-garage/g2-sdk-go-base/issues/151

Issue moved to https://github.com/senzing-garage/go-rest-api-service/issues/153


Transparent OSM not working in Ol 6

In OL 5 Open seamap was an transparent Overlay .
With OL 6 the Layer the layer is mostly opaque white
https://codesandbox.io/s/semi-transparent-layer-038nq

Quite weird, also putting an opacity of 0.99 makes the layer see-through again, but anything above that makes it opaque.

The OSM layer is opaque by default:
https://openlayers.org/en/latest/apidoc/module-ol_source_OSM-OSM.html
This property is then used to optimize the canvas rendering; that's why the canvas layer has no transparency.
Passing opaque: true in the options fixes the issue.

The OSM layer in 5.3.3 had the same API comment and it worked.

@property {boolean} [opaque=true] Whether the layer is opaque.
https://github.com/openlayers/openlayers/blob/v5.3.3/src/ol/source/OSM.js

Are there any plans to change it back to the old behavior?

I found an other interesteing effect. If im adding an transparent layer bevor the OSM Layer it gets transparent. I updated the example

i stumbled upon my bug again and found more inconsistencies:
For the property opaque in ol/scource/
Tile
jsdoc no default
sets opaque to false if undefined.
UrlTile extends Tile
jsdoc no default
constructor passes on opaque from options to super without modification
TileImage extends UrlTile
jsdoc: default opaque=true
constructor passes on opaque from options to super without modification
XYZ extends TileImage
jsdoc: default opaque=true
constructor passes on opaque from options to super without modification
OSM extends XYZ
jsdoc: default opaque=true
defaults to true if options.opqaue is undefined
So OSM is the only Layer scource that behaves according to the jsdoc (default is true) but is also the only one that behaves not like the others. They default to opaque=false despite the js doc says otherwise for XYZ and TileImage.


Update concept-authentication-passwordless.md

Updating links for pin protocols to use final published CTAP2.1 spec docs. Previous links referred to CTAP 2.1 draft.

@maweeras-msft : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@maweeras-msft I apologize for missing this! Yesterday, I updated the topic to reflect other changes in addition to the updated spec URLs you shared. That update caused a merge conflict here. Can you please pull from main again and resubmit any remaining changes you want to see?

Update has correct link now.


Explain Execution Plan Node Type doesn't include "Parallel" keyword

System information:

Operating system (distribution) and version: Windows 10 64 bit
DBeaver version: <IP_ADDRESS>111011413
Java version: 1.8

Connection specification:

Database name and version: PostgreSQL v13.4
Driver name: PostgreSQL JDBC Driver
Do you use tunnels or proxies (SSH, SOCKS, etc)? No

Describe the problem you're observing:
When generating Explain with "EXPLAIN select ..." the plan appears with "Parallel Seq Scan" as expected,
but when generating Explain with CTRL+SHIFT+E, the Node Type value appears as "Seq Scan" (while parallel-aware=true on the right side)
It would be more clear and by the standard if DBeaver keeps the "parallel" keyword before "Seq Scan".

Hello @yonisade
Sorry, but I'm not sure that I understand you correctly.
DBeaver just returns an answer from the server for the "EXPLAIN ...." query. DBeaver does not add anything to the result.

You can check your Explain query and DBeaver's query in the Query Manager.
Can you tell us more about your idea?

Please change your query to "EXPLAIN select i,count(*) from test group by i order by 2 desc"
You're supposed to see "Parallel Seq Scan" under the plan's "Node Type" as opposed to the plan generated by "CTRL+Shift+E"

Sorry, didn't help. Obviously, the result depends on server settings, which are different on my server.

try:
set max_parallel_workers=4;
set max_parallel_workers_per_gather=2;
set force_parallel_mode=on;
set parallel_setup_cost = 10;
set parallel_tuple_cost = 0.001;

@yonisade Thank you for the report, I reproduced this bug.

This happens because the SELECT (FORMAT XML) is executed on server side and Postgres returns parallel qualification as separate attribute.
        <Plan>
         <Node-Type>Seq Scan</Node-Type>
         <Parent-Relationship>Outer</Parent-Relationship>
         <Parallel-Aware>true</Parallel-Aware>
         <Relation-Name>import_time2</Relation-Name>
         <Alias>import_time2</Alias>
         <Startup-Cost>0.00</Startup-Cost>
         <Total-Cost>127019.38</Total-Cost>
         <Plan-Rows>2083038</Plan-Rows>
         <Plan-Width>22</Plan-Width>
        </Plan>

Maybe there is a way to parse this

Verified


initial implementation

I'll debug this tomorrow, but quickly: the command exists silently: https://github.com/EmaApps/ema-template/pull/39#issuecomment-1343712883


gevent sleep in (possible) long loop in bluecoat formatter

BlueCode formatter includes a potentially long loop. Adding gevent.sleep(0) to avoid starvation.

:shipit:


pod vm image creation failing with latest ubuntu image

Ref image: https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img
gh-action job - https://github.com/confidential-containers/cloud-api-adaptor/actions/runs/4534666068/jobs/7998000405
cc @jtumber-ibm if you have any idea.

Without PACKER_INFO set, my only guess is it was running too slowly and didn't get to the point where it sets up the PeerP0D ssh username and password. As the s390x version, which has a longer boottime set was able to ssh in.
With failed to push quay.io/confidential-containers/podvm-generic-ubuntu-s390x:latest: unexpected status: 401 UNAUTHORIZED for s390x is that a expired API key?

Right. The s390x succeeded and it's available now. I had to fix the permissions.

Stale issue.


Create a linux-amd64 distribution of bimage

Create a linux platform distribution of bimage using grayskull and conda-build on cznworker02 and upload it to your channel on anaconda.org

https://anaconda.org/jnsproul/bimage


It creates new canvas once state is changed

I'm having a problem with my code where I am using P5-wrapper/react to show the 2D model I created. However, whenever React states changes it re-creates the canvas again and again it gives warning that WARNING: Too many active WebGL contexts. Oldest context will be lost..
My code is a bit complex, thus I have created a simple example in this codesandbox, but I will also give a bit information.
I got Measurement class where I am using to set up specific measurement objects.
class Measurement {
  constructor(p5, initVariables, onChangeDraw, _ref) {
    this.p5 = p5;
    this.initVariables = initVariables;
    this.onChangeDraw = onChangeDraw;
    this.ref = _ref;
    this.pointActive = false;
    this.activePoint = this.p5.createVector(0, 0);
  }

  checkPoints() {
    if (
      this.p5.dist(
        this.p5.mouseX,
        this.p5.mouseY,
        this.ref.x + 300,
        this.ref.y + 250
      ) < 26
    ) {
      this.activePoint = this.ref.x;
      this.pointActive = true;

      if (this.initVariables.mP) {
        this.ref.x = this.p5.mouseX - 300;
        this.ref.y = this.p5.mouseY - 250;
      }
    } else {
      this.pointActive = false;
    }

    if (this.initVariables.mP && this.onChangeDraw && this.pointActive) {
      this.onChangeDraw();
    }
  }

  drawMeasurement() {
    if (this.pointActive) {
      this.p5.noStroke();
      this.p5.fill(0, 20);
      this.p5.circle(this.ref.x, this.ref.y, 13);
    }
    this.p5.stroke(255, 0, 0);
    this.p5.fill(255, 0, 0);
    this.p5.strokeWeight(2);
    this.p5.circle(this.ref.x, this.ref.y, 5);
    this.p5.noFill();
  }
}

and some variables and function:
const mM = [];
let dataDimensions = [];
let setDataDimensions;

let initVariables = {
  mP: false,
};

const onChangeDraw = () => {
  setDataDimensions([...dataDimensions]);
};

and I have this sketch function I am using:
const sketch = (p5, measurements, setMeasurements) => {
  p5.setup = () => {
        measurements.map((measurement) => {
      const name = new Measurement(
        p5,
        initVariables,
        onChangeDraw,
        measurement.ref
      );
      mM.push(name);
    });
    dataDimensions = measurements;
    setDataDimensions = setMeasurements;
    p5.createCanvas(600, 500, p5.WEBGL);
  };

  p5.draw = () => {
    p5.background(237, 239, 237);
    p5.stroke(0);
    p5.strokeWeight(2);
    p5.circle();
    p5.beginShape();

    for (let m of mM) {
      m.drawMeasurement();
      m.checkPoints();
    }

    mM.map((measurement) => p5.vertex(measurement.ref.x, measurement.ref.y));
    p5.endShape(p5.CLOSE);
  };

  p5.mousePressed = () => {
    initVariables.mP = true;
  };
  p5.mouseReleased = async () => {
    initVariables.mP = false;
  };
};

problem seems to be once I click and drag any point where state changes because of onChangeDraw where I set the measurements into new measurements I get from p5 side.
So, also including measurements for reference:
const [measurements, setMeasurements] = useState([
    {
      id: 1,
      ref: {
        x: 100,
        y: 100,
      },
    },
    {
      id: 2,
      ref: {
        x: 200,
        y: 100,
      },
    },
    {
      id: 3,
      ref: {
        x: 200,
        y: 200,
      },
    },
    {
      id: 4,
      ref: {
        x: 100,
        y: 200,
      },
    },
  ]);

and wrapper:

      <ReactP5Wrapper
        sketch={(p5) => sketch(p5, measurements, setMeasurements)}
      />

This was my problem too. I ended up not using states cause it also clears the mouse event handlers on my matterjs side as the canvas gets re-rendered.  I tried to use React.memo on the wrapper but I got more bad side effects. I suggest using just refs to store values and put your sketch function inside the component where the wrapper is so that it has access to the refs:
export default function App() {
  const measurements = useRef([
    {
      id: 1,
      ref: {
        x: 100,
        y: 100,
      },
    },
    {
      id: 2,
      ref: {
        x: 200,
        y: 100,
      },
    },
  ]);

  const sketch = (p5) => {
    p5.setup = () => {
      // loop thru measurements.current
      // and create your Measurement objects
    }
  };

  return (

      <ReactP5Wrapper sketch={sketch} />

  )
}

Closing due to inactivity.

Sorry for the late reply, I found the solution myself to this problem, so hoping it would help other people in future.
I have tried to use React.memo for the dimensions:
import React from 'react';
import { ReactP5Wrapper } from "@p5-wrapper/react";
import { shoeSketch } from '~/sketch/shoe/sketch';

const Canvas = React.memo(({ sketchDimensions, setSketchDimensions, selectedSegment, setSelectedSegment }) => {

    if (sketchDimensions.length > 0)
        return (
            <ReactP5Wrapper
                sketch={shoeSketch}
                sketchDimensions={sketchDimensions}
                setSketchDimensions={setSketchDimensions}
                selectedSegment={selectedSegment}
                setSelectedSegment={setSelectedSegment}
            />
        );
});

export default Canvas;

and inside my sketch file, I am using updateWithProps function that comes with it to get and set the dimensions:
    p5.updateWithProps = props => {
        if (props.sketchDimensions !== dataDimensions) {
            dataDimensions = props.sketchDimensions;
        }
        // and other states

This solved my problem with creating new canvas elements after changing measurements.


Add UTF-8 Content-Type to the Django formatter

Django's ugettext() only returns unicode strings correctly when the Content-Type is set

Thanks!


CODEOWNERS: Update @tsteenbe responsiblities

Assign @tsteenbe as reviever for any docs, license aliases/mapping
or WebApp related changes as he has contributed in these areas.
Signed-off-by: Thomas Steenbergen<EMAIL_ADDRESS>

Fixed spelling mistake in commit message reviever => reviewer.


Use specified dimension separator for region index

I noticed that I didn't set the dimension separator when doing #290.
I also noticed that contig_id, contig_length, filter_id, and sample_id don't have a dimension separator set - which OK as they are one-dimensional - but slightly inconsistent as there are other fields that are one-dimensional that do have it set, like variant_id. Should we just set it everywhere?

Should we just set it everywhere?

I don't really mind. I set the dimension separator to / as this seemed the more scalable option for the large 2D fields and followed the defaults in v3 (making upgrades to format=3 stores later more straightforward, potentially).


Next.js best practices

We have done a good job at writing quality code, but it is important to always review the state of the codebase to improve it and update itL

Are we following Next.js' best practices?
Is our use of Next.js up-to-date?

This is an on-going "issue" that will remain open: do you see anything that could be improved? Let us know and feel free to open a pull request.

Hey @Bitnagar! Thanks for the heads up. Feel free to open a PR with those changes!


Added basic doc section for repo administration

Added basic doc section for repo administration, as mentioned in this issue.
Included doc in metadata file.

This issue has been automatically marked as stale because it has not had recent activity.

I mean, by that reasoning, then so has the entire repo...


Create Pantry App

What does this PR do

This PR creates a pantry app

Description of Task to be completed

Create a new pantry app and move the viewsets from waitress app to pantry

How should this be manually tested

N/A

Any background context you want to add

The Pantry app was previously coupled with the waitress app

Screenshots

N/A

Apart from the comments by @BolajiOlajide. LGTM


Support for timestamp downcasting when loading data to iceberg tables

Apache Iceberg version
0.7.0 (latest release)
Please describe the bug 🐞
As of release 0.7.0, pyiceberg tables support new data-loading method pyiceberg.table.Table.add_files(). However, this method currently does not respect well documented setting downcast-ns-timestamp-to-us-on-write.
Setting downcast-ns-timestamp-to-us-on-write is always defaulted to False if Table.add_files() is used. No matter if config file explicitly specifies downcast-ns-timestamp-to-us-on-write: "True".
Env variable PYICEBERG_DOWNCAST_NS_TIMESTAMP_TO_US_ON_WRITE is not respected as well.

I also have patch for this ready, however it seems like I have no permissions to push a new branch and to create PR
diff --git a/pyiceberg/io/pyarrow.py b/pyiceberg/io/pyarrow.py
index b99c3b1..153b8a5 100644
--- a/pyiceberg/io/pyarrow.py
+++ b/pyiceberg/io/pyarrow.py
@@ -2303,6 +2303,8 @@ def _check_pyarrow_schema_compatible(

 def parquet_files_to_data_files(io: FileIO, table_metadata: TableMetadata, file_paths: Iterator[str]) -> Iterator[DataFile]:
+    from pyiceberg.table import DOWNCAST_NS_TIMESTAMP_TO_US_ON_WRITE
+
     for file_path in file_paths:
         input_file = io.new_input(file_path)
         with input_file.open() as input_stream:
@@ -2313,7 +2315,12 @@ def parquet_files_to_data_files(io: FileIO, table_metadata: TableMetadata, file_
                 f"Cannot add file {file_path} because it has field IDs. `add_files` only supports addition of files without field_ids"
             )
         schema = table_metadata.schema()
-        _check_pyarrow_schema_compatible(schema, parquet_metadata.schema.to_arrow_schema())
+        downcast_ns_timestamp_to_us = Config().get_bool(DOWNCAST_NS_TIMESTAMP_TO_US_ON_WRITE) or False
+        _check_pyarrow_schema_compatible(
+            schema,
+            parquet_metadata.schema.to_arrow_schema(),
+            downcast_ns_timestamp_to_us
+        )

         statistics = data_file_statistics_from_parquet_metadata(
             parquet_metadata=parquet_metadata,

Hi @fusion2222 thank you for raising this issue. We'd love to see your contribution on the project!
Could you try forking the project and then creating a new branch there to create the PR?

Here's the setup I currently use

Git clone apache/iceberg-python repo (this repo)
Fork the repo (kevinjqliu/iceberg-python)
Add the forked repo as a remote named kevinjqliu (git remote add kevinjqliu git@github.com:kevinjqliu/iceberg-python.git).
Create a new branch (git checkout -b kevinjqliu/issue-1045)
Add changes and commit
Push the branch to the forked repo (git push kevinjqliu)
Come back to this repo's Github and open a PR

It would be good to add this to the Contributing docs :)

@kevinjqliu @sungwy can I help here? I faced the same issue while trying to write a csv into iceberg using pyiceberg/catalog/sql.py

import os os.environ['PYICEBERG_DOWNCAST_NS_TIMESTAMP_TO_US_ON_WRITE'] = 'true'
I think we can add this to the documentation

The documentation is at https://py.iceberg.apache.org/configuration/#nanoseconds-support
Do you think there's a better place to signal this to the users?


AD as auth provider, allow users to log in with their userPrincipalName (email address)

When using Active Directory, allow users to log in with their email addresses (aka userPrincipalName).

With the release of Rancher 2.0, development on v1.6 is only limited to critical bug fixes and security patches.


Add simplecov

Add SimpleCov gem and update config in .circleci/config.yml and spec/rails_helper.rb.
Note that SimpleCov is being added to the app generated by dls_template, not dls_template itself.
See https://github.com/pulibrary/rails-template/pull/23 for reference.

I am having some trouble testing the gem locally at the moment, so I haven't completely verified that this PR works yet. Will also need a rebase after https://github.com/pulibrary/dls_template/pull/6.

In setting up SimpleCov, a couple of lines are added to the CircleCI config as in: https://github.com/pulibrary/rails-template/pull/23/files#diff-78a8a19706dbd2a4425dd72bdab0502ed7a2cef16365ab7030a5a0588927bf47R80-R81
However, our current CircleCI .config file that is copied over doesn't have a test: block under jobs: at the moment. If we end up adding testing config for CircleCI (which should be set up for the new app) then we can add the SimpleCov lines, but that should probably be in a separate PR (or it might be already).


added create-react-app v2

Recreated the template from create-react-app cli v2 with all the new version
to solve issue #6

Woooo! Thank you 👍

you r god
@orYoffe

Thank you for your work.
The README.md is outdated though as the test:web or test:native commands are no longer available.


Build failed when link with libsodium on macOS

make all
==== Building yojimbo (debug_x64) ====
==== Building test (debug_x64) ====
Linking test
ld: library not found for -lsodium
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [bin/test] Error 1
make: *** [test] Error 2

macOS 10.12.3
Apple LLVM version 8.0.0 (clang-8<IP_ADDRESS>)
Target: x86_64-apple-darwin16.4.0
Thread model: posix
homebrew config
HOMEBREW_VERSION: 1.1.11
ORIGIN: https://github.com/Homebrew/brew.git
HEAD: 394f9fa0aaa5854aa52bc589708a079665dcf462
Last commit: 3 weeks ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 146a4bf1580b0cc2ea22b6df6320fab43f3f3e76
Core tap last commit: 23 hours ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_REPOSITORY: /usr/local/Homebrew
HOMEBREW_CELLAR: /usr/local/Cellar
HOMEBREW_BOTTLE_DOMAIN: https://homebrew.bintray.com
CPU: quad-core 64-bit sandybridge
Homebrew Ruby: 2.0.0-p648
Clang: 8.0 build 800
Git: 2.10.1 => /Applications/Xcode.app/Contents/Developer/usr/bin/git
Perl: /usr/bin/perl
Python: /usr/bin/python
Ruby: /Users/apple/.rvm/rubies/ruby-1.9.3-p194/bin/ruby
Java: N/A
macOS: 10.12.3-x86_64
Xcode: 8.2.1
CLT: N/A
X11: 2.7.7 => /opt/X11
brew ls libsodium
/usr/local/Cellar/libsodium/1.0.12/include/sodium/ (57 files)
/usr/local/Cellar/libsodium/1.0.12/include/sodium.h
/usr/local/Cellar/libsodium/1.0.12/lib/libsodium.18.dylib
/usr/local/Cellar/libsodium/1.0.12/lib/pkgconfig/libsodium.pc
/usr/local/Cellar/libsodium/1.0.12/lib/ (2 other files)

Is anybody else getting this issue? Working fine over here.

I haven't had any issues on Mac.

No issues. El Capitan 10.11.6.
I've seen linker errors like that after switching Xcode versions. Try xcode-select --install

I'm on Sierra over here and trying to upgrade brew to latest to see if that breaks anything.

Compiles and links fine over here. Seconded, try: xcode-select --install

gafferongames:libyojimbo gaffer$ brew --version
Homebrew 1.1.11
Homebrew/homebrew-core (git revision bfe7e; last commit 2017-03-27)

gafferongames:libyojimbo gaffer$ brew ls libsodium
/usr/local/Cellar/libsodium/1.0.12/include/sodium/ (57 files)
/usr/local/Cellar/libsodium/1.0.12/include/sodium.h
/usr/local/Cellar/libsodium/1.0.12/lib/libsodium.18.dylib
/usr/local/Cellar/libsodium/1.0.12/lib/pkgconfig/libsodium.pc
/usr/local/Cellar/libsodium/1.0.12/lib/ (2 other files)

gafferongames:libyojimbo gaffer$ brew --config
HOMEBREW_VERSION: 1.1.11
ORIGIN: https://github.com/Homebrew/brew.git
HEAD: 394f9fa0aaa5854aa52bc589708a079665dcf462
Last commit: 3 weeks ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: bfe7eefdf2c591c56742b6d07aa09fe2bf53746f
Core tap last commit: 60 minutes ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_REPOSITORY: /usr/local/Homebrew
HOMEBREW_CELLAR: /usr/local/Cellar
HOMEBREW_BOTTLE_DOMAIN: https://homebrew.bintray.com
CPU: octa-core 64-bit haswell
Homebrew Ruby: 2.0.0-p648
Clang: 8.0 build 800
Git: 2.12.2 => /usr/local/bin/git
Perl: /usr/bin/perl
Python: /usr/local/bin/python => /usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/bin/python2.7
Ruby: /usr/local/bin/ruby => /usr/local/Cellar/ruby/2.4.1_1/bin/ruby
Java: 1.8.0_111
macOS: 10.12.3-x86_64
Xcode: 8.2.1
CLT: <IP_ADDRESS>.1.1480973914
X11: 2.7.6 => /opt/X11

gafferongames:libyojimbo gaffer$ g++ --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.0.0 (clang-8<IP_ADDRESS>)
Target: x86_64-apple-darwin16.4.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

i'm not on Mac currently, will try xcode-select --install after going home.
Any other suggestion?

Did `xcode-select --install' sort it out for you?

Yes, thanks!
Sorry for the late reply.
➜  libyojimbo git:(968fa62) make all
==== Building yojimbo (debug_x64) ====
==== Building test (debug_x64) ====
Linking test
==== Building info (debug_x64) ====
Creating obj/x64/Debug/info
info.cpp
Linking info
==== Building client (debug_x64) ====
Creating obj/x64/Debug/client
client.cpp
Linking client
==== Building server (debug_x64) ====
Creating obj/x64/Debug/server
server.cpp
Linking server
==== Building secure_client (debug_x64) ====
Creating obj/x64/Debug/secure_client
secure_client.cpp
Linking secure_client
==== Building secure_server (debug_x64) ====
Creating obj/x64/Debug/secure_server
secure_server.cpp
Linking secure_server
==== Building client_server (debug_x64) ====
Creating obj/x64/Debug/client_server
client_server.cpp
Linking client_server
==== Building soak (debug_x64) ====
Creating obj/x64/Debug/soak
soak.cpp
Linking soak
==== Building profile (debug_x64) ====
Creating obj/x64/Debug/profile
profile.cpp
Linking profile

Excellent! Closing out


Upgrade react native iap

#195

Migration guide
https://react-native-iap.dooboolab.com/docs/migrate_to_10.0.0
https://react-native-iap.dooboolab.com/docs/migrate_to_11.0.0
https://react-native-iap.dooboolab.com/docs/migrate_to_12.0.0


Support new 0.5 based http health checks

Since 0.5 consul supports http based health checks as per https://consul.io/docs/agent/checks.html. Would be great if this could be supported especially with the /health actuator endpoint from Boot being available out of the box.

:+1: 0.5 also has other useful stuff.

I may actually get around of submitting a PR myself. I guess I have to sign the CLA first :-)

I have a very basic version working in https://github.com/mattreyuk/spring-cloud-consul if you're interested

Fixed via 0d3f028a5fd8241ee5a0d78d8f4e3a5de930f6e7


Feature Request: Relocate user folders (Desktop, Documents, Downloads, Music Pictures, Videos) to another drive (e.g., D:)

To simplify backups and general navigation, I like to relocate most of the user folders to the root of D: (and delete the others such as Favorites, Searches, 3D Objects, Contacts, etc.).
I understand there's a way using the answer file to do this, but I haven't been able to find much information about it. I did create a dskpart script to partition the drive at the outset, and this works in cschneegans's answer file.
I have in the past used the Location tab in each folder's Properties to relocate them one-at-a-time post install, but I like the idea of Setup putting them in the right place from the start.

I have never moved my user profiles from C:\Users to another location, and I don't see much benefit in doing so.
However, if you want to store your user profiles on another drive, setting this up via an autounattend.xml file is probably a good approach as this allows you to apply the required modifications early, i.e. before user accounts are even created.
Therefore, you could try the following:

Use a custom diskpart script (in the Partitioning and formatting section) to create the Windows partition and a data partition.
Use one of the following methods:

Generate and download your autounattend.xml file, locate the Microsoft-Windows-Shell-Setup component and add a ProfilesDirectory element.
Use a .reg file in the Scripts to run in the system context, before user accounts are created section to modify the ProfilesDirectory value in both keys [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList] and [HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion\ProfileList].

I have not tested this. Also note Microsoft warning from the link above: “We don’t recommend using this setting, except perhaps in a test environment.”

I have now tried to make this work using a .cmd script that runs “before user accounts are created”:
mkdir D:\Users
reg.exe add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList" /v ProfilesDirectory /t REG_SZ /d "D:\Users" /f
reg.exe add "HKLM\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion\ProfileList" /v ProfilesDirectory /t REG_SZ /d "D:\Users" /f

This link will configure the form accordingly. Works for me – the profiles for all users are created under D:\Users. I still advise against this.


Fix for FilePath mapping between Unix and Win

Hi Jonathan,
I tried to debug an app running on Win32 with VIM/vdebug on Linux and didn't see any code in the source window. Despite my path mapping FilePath didn't get the remaining path separators right.
There is one patch for FilePath which should fix path separators in Unix<->Windows mixed scenarios that comes with some additional unit tests. The other one fixes the pathes in the Stack Window where s.get('filename') doesn't give the local but the RemoteFilePath.
With these small fixes I'm now happily debugging in cross OS scenarios thanks to your great VIM plugin.
Kind regards,
Bernd

This has been merged manually into the dev branch, and is slated for release soon (next couple of weeks) in version 1.4.3.
Sorry for the utterly ridiculous wait time...


Username with "ö" in name is not importable

I can not import character from my account "Barsöckn", because the "ö" is not in the character set, as it will instantly become a "?" if you type it in.
Using my POESESSID does not work either, i guess because the username is still not usable
THanks in advance for your help

As per #258, non-ASCII characters in the account name have to be replaced with their URL encoding, so your account name becomes:
Bars%C3%B6ckn
The root cause of this issue is the program's lack of Unicode support, which unfortunately isn't easy to fix. I probably aught to put a note about this in the import section, though...


Consider splitting parser and generator into separate projects / concepts

I know you can pass in the -v flag to only do validation, but the default is to validate and generate.
I could see using basketry as a part of an integration test platform. A quick validation if the API conforms to all the rules before allowing a merge to go through.
You could do this now, using the -v flag, but I feel like these are two separate concepts and should have two separate pipelines and thus two separate projects. Have a core project for the IR that both the Parser and the Generator base projects can reference -- and possibly even Validation... I haven't jumped into the code enough to know if that can be split out, too, but probably!
Use case: I already use a generator to either generate my swagger from source, or vice versa. But having a rules engine to sit on top of that could be really interesting.
Use case 2: I generate the IR myself (becoming my first-class representation of my API), and then generate the swagger and the DTOs from the IR, i.e. no parsing necessary (although maybe I still want to run validators against my IR).
Just a thought...

First of all, I'm thrilled that you at least found Basketry interesting! 😃
I agree with breaking up the project into smaller parts. As the project matures and I have a better understanding of various use cases, I would love to find natural seams to split on. My thoughts so far are to break things apart into "core" and "cli" projects. From a CLI perspective, I think I want to move to a basketry {command} [options] syntax. The code as-is would mostly support that sort of decomposition without too much refactoring. I have a hunch that sort of major change would happen as part of an official "v1.x.x" version, but unfortunately I can commit to when exactly that might happen.
Regarding your use cases:

having a rules engine to sit on top of that could be really interesting

You can technically run the pipeline without any generators. This would have the effect of just running the rules engine, even if you don't pass the -v option. (If/when the generation and validation parts become separate commands, the generate command should require at least one generator, whereas the validate command wouldn't need to.)

generate the swagger and the DTOs from the IR

I am working (slowly) on adding a generator function to the swagger-2 parser package. Doing so will allow that package to create a Swagger doc from IR (instead of just generating IR from a Swagger doc). This will be useful for translating from one SDL to another, or directly from IR back to Swagger as in your second use case.

I generate the IR myself

It would be trivial to build a "parser" that just reads raw IR from a file. Having that low-level sort of generator would allow other toolchains to write IR as a JSON file which can then seamlessly be read or piped into the pipeline. I'll get that "raw IR" parser published, thanks for the suggestion! If you decide to go that route, let me know—I'd love to publish an article about that approach to pluggability 😃

The new @basketry/ir package supports reading and writing directly from IR: https://www.npmjs.com/package/@basketry/ir

The sounds ambitious!
FWIW, I'm finalizing a change to break the CLI into multiple commands: #33


[BUG] "Circular Borders" doesn't scale with "Scale Borders"

Module Version: v0.2.18
Describe the bug:
Title
Screenshots:

Browser:
Verified in Foundry VTT dedicated, Firefox, and Chrome
Foundry Version: v9.269
Game System: DnD5e 1.5.7
Additional context:
Is bugged with or without other modules.

Should be fixed with 0.2.19


Template service should respect and default a duration

Right now a new service built from a template goes to 100% as soon as it is ready. Instead it should get a rollout duration of 1m, or something similar. Similarly, if I pass --rollout-duration=1h at the time I create the template service, every rollout after that should use the 1h duration time specified, this will require an extra API field.

Also with this, the weight goes to 100 instead of 10000 when using template services. There's no weight value in template spec, but looking at the first version that is created from it:
...
spec:
  app: temptag
  build:
    branch: master
    repo: https://github.com/rancher-max/rio-demo
    revision: vme-alpha-rc-things
    tag: true
    webhookSecretName: githubtoken
  image: localhost:5442/default/temptag-v0l9scf-ccd1159-59d8a:vme-a
  ports:
  - port: 8080
    targetPort: 8080
  weight: 100
status:
  appEndpoints:
  - https://temptag-default.xxxxxx.on-rio.io
  - http://temptag-default.xxxxxx.on-rio.io
  buildLogToken: c8rhbgsls7xnl5ql68vr9ngnh7g57hcgtj6npr7wqjnzpv7wb8xv9w
  computedApp: temptag
  computedVersion: 9ea7977d
  computedWeight: 100
...

Working in master

All working in master now.


Refactor code with py3 type annotation

Label prediction was below confidence level 0.6 for Model:ServiceLabels: 'Cosmos:0.33119145,Service Bus:0.102717325,Data Factory:0.09431788'


Add 5 new accessibility quiz questions

Checklist:

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)

Related to #374

whoops, I just realized someone else recently added a question that is similar to one that I added. I'm busy at the moment but will clean up shortly.


[BUGFIX] CI: Remove workaround for phpspec/prophecy and PHP 8.2

phpspec/prophecy was released as version 1.16 on 2022-11-29, bringing support for PHP 8.2. Therefore, the existing workaround in the GitHub Actions workflow is now removed.

Seems like this has been introduced with PR #85 (briefly comparing changes)


Use WebP responsive images

This PR replaces the jekyll-responsive-images with jekyll-imagemagick for responsive WebP images.
WebP images are much smaller compared to PNG and JPEG, faster to load and most of the modern browsers recommend it.
More information about WebP images: https://developers.google.com/speed/webp

Tested. Work perfect. Great job!
The assets/resized/ folder is no longer needed, delete.


Fixed re-render flicker on mute and screenshare icons.

Summary
Created SVG components in /components/svg/ for the mute and screensharing icons. Using SVG's instead of <MaterialIcon /> components, seems to not flicker on re-render.
Ticket Link
https://trello.com/c/9cyI0Yv9/228-flicker-on-ss-and-mute-icons-in-riff-edu
Checklist

[x] Ran make check-style to check for style errors (required for all pull requests)
[x] Has UI changes

Mike, I addressed those issues, but I accidentally force-pushed onto the branch! Do you still have your changes locally?

 Flicker on ss and mute icons in Riff Edu

ok, I've gotten your updates, and made some additional style fixes and put it back on the original branch for this PR, and rebased it to the head of develop.


CI: Runtime (privileged): TestAgentCell: cells_test.go:37: found unexpected goroutines

CI failure
Instances:

https://github.com/cilium/cilium/actions/runs/8082558741/job/22083901676

--- FAIL: TestAgentCell (3.86s)
    cells_test.go:37: found unexpected goroutines:
        [Goroutine 202 in state chan receive, with github.com/cilium/cilium/pkg/identity/cache.(*IdentityWatcher).Watch.func1 on top of the stack:
        github.com/cilium/cilium/pkg/identity/cache.(*IdentityWatcher).Watch.func1()
        	/host/pkg/identity/cache/cache.go:130 +0x7d
        created by github.com/cilium/cilium/pkg/identity/cache.(*IdentityWatcher).Watch in goroutine 81
        	/host/pkg/identity/cache/cache.go:124 +0x67
        ]

I believe this was a PR commit, mistaken for main due to confusion regarding the "Summary" section (as discussed in the community meeting).


[3.x] Fix physics BVH pairing for teleported or fast moving objects

Updating the broadphase to find new collision pairs was done after checking for collision islands, so it was working in most cases due to the pairing margin used in the BVH, but in case of teleported objects the narrowphase collision could be skipped.
Now it's done before checking for collision islands, so we can ensure that broadphase pairing has been done at the same time as objects are marked as moved so their collision can be checked properly.
This issue didn't happen in the Octree/HashGrid because they do nothing on update and trigger pairs directly when objects move instead.
Fixes #54898 on the 3.x branch
Probably helps with #54855 (to be confirmed)
I'll make a 4.0 PR as well once this fix is validated.

This doesn't help with ΔV: Rings of Saturn - running 309696c3f with BVH bullets often phase through the asteroids. They don't do that with the old collision detection. Bullets tavel at ~300px/frame with raycast collision detection.

@lekoder Would you be able to provide me with a minimal project to reproduce this issue? It would make it much easier to investigate.

@pouleyKetchoupp of course, here you go: https://github.com/KoderaSoftwareUnlimited/godot-issue-bvp-raycast - tested on 309696c3f

I've made a change to update the broadphase after applying rigid body velocities (and still before generating collision islands) to make sure it registers pairs for fast moving bodies early enough (it was one frame late so the CCD raycast test could fail).
Now the case from https://github.com/godotengine/godot/pull/54925#issuecomment-968255885 is fixed too.

I can confirm that this patch works for both the MRP I supplied, but most importantly, for my game (which is a relief, as it means I made that MRP right :) ). Looking forward to seeing this in 3.4.1!

Thanks!

Cherry-picked for 3.4.1.


Remove (Begin/End)GetRequest async mechanism

Remove (Begin/End)GetRequest async mechanism
This caused too much problems (Requests don't getting closed properly, exceptions of all kinds and more..).

Using async await is cool but I want to preserve backward compatibility with old versions of .net - I didn't check it, so if async await approach is working also for the older .net projects I can consider this.

Alternatively, I thought about opening configurable number of worker threads that will take a bulk request object and handle the whole communication with the Elasticsearch server synchronously.
They all will read the requests from a queue with a configurable limit (when full all new requests will be lost into the wild universe).
This way we can handle more easily a connection loss (I think).

May the odds be ever in the requests' favor.

Closed - thanks to @erangil2


Fix length of names of iptables

Chain names in iptables are restricted to 28 characters. Our current format would result in updates failing after 1000 updates. With this PR we remove the descriptive characters in the iptables to allow for millions of unique updates for long running processing units.

Codecov Report

Merging #259 into master will increase coverage by 0.04%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #259      +/-   ##
==========================================
+ Coverage   55.07%   55.12%   +0.04%
==========================================
  Files          38       38
  Lines        4147     4147
==========================================
+ Hits         2284     2286       +2
+ Misses       1666     1665       -1
+ Partials      197      196       -1

Impacted Files
Coverage Δ

supervisor/iptablesctrl/iptables.go
52.08% <ø> (ø)
:arrow_up:

supervisor/ipsetctrl/acls.go
85.51% <ø> (ø)
:arrow_up:

processmon/processmon.go
71.81% <0%> (+1.34%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cbe149d...6aee743. Read the comment docs.


Support GDS links for files and folders

GDS Links should be available to CWL locations as we can may be able to use #206 to mount files appropriately.

User would need to support an ICA_ACCESS_TOKEN as an environment variable

Needs https://github.com/umccr-illumina/ica_v2/issues/151 to complete first


[experimental] gatsby/playroom for docs

an example/poc of using gatsby+playroom for docs
there are some codeblock and preview components from the apollo docs plugin but we might want to roll our own to make them better. i also setup a playroom instance to show that we can open a new playroom page directly from a codeblock
setup
you'll need to create an .env.development file and we don't have an algolia account but you can use apollo's to get search results
ALGOLIA_API_KEY=768e823959d35bbd51e4b2439be13fb7
ALGOLIA_INDEX_NAME=apollodata

Dane's findings:

Gatsby, playroom could be used for docs
Docsearch integratable with storybook or gatsby or anything

Closed as a spike.  Use findings in future PR.


serve-machine-learning-models-with-clipper notebook issue

I am also having a similar issue with the quick start after installing Clipper for the first time.
Just installed Clipper on Mac OSX 10.13.2, running conda version 4.4.8 and have created this env:
(clipper27) mauricemanning:clipper$conda list
packages in environment at /Users/mauricemanning/anaconda3/envs/clipper27:
Name                    Version                   Build  Channel
appnope                   0.1.0            py27hb466136_0
backports                 1.0              py27hb4f9756_1
backports.shutil_get_terminal_size 1.0.0            py27hc9115de_2
backports_abc             0.5              py27h6972548_0
ca-certificates           2017.08.26           ha1e5d58_0
cachetools                2.0.1
certifi                   2018.1.18                py27_0
chardet                   3.0.4
clipper-admin             develop
cloudpickle               0.5.2
conda-api                 1.1.0                    py27_0    anaconda
decorator                 4.2.1                    py27_0
docker                    2.7.0
docker-pycreds            0.2.1
enum34                    1.1.6            py27hf475452_1
google-auth               1.3.0
idna                      2.6
intel-openmp              2018.0.0             h8158457_8
ipaddress                 1.0.19
ipykernel                 4.8.0                    py27_0
ipython                   5.4.1                    py27_2
ipython_genutils          0.2.0            py27h8b9a179_0
jupyter_client            5.2.2                    py27_0
jupyter_core              4.4.0            py27h5ea6ba4_0
kubernetes                4.0.0
libcxx                    4.0.1                h579ed51_0
libcxxabi                 4.0.1                hebd6815_0
libedit                   3.1                  hb4e282d_0
libffi                    3.2.1                h475c297_4
libgfortran               3.0.1                h93005f0_2
libsodium                 1.0.15               hd9e47c5_0
mkl                       2018.0.1             hfbd8650_4
ncurses                   6.0                  hd04f020_2
numpy                     1.14.0           py27h8a80b8c_1
oauthlib                  2.0.6
openssl                   1.0.2n               hdbc3d79_0
pathlib2                  2.3.0            py27he09da1e_0
pexpect                   4.3.1                    py27_0
pickleshare               0.7.4            py27h37e3d41_0
pip                       9.0.1            py27h1567d89_4
prometheus-client         0.1.1
prompt_toolkit            1.0.15           py27h4a7b9c2_0
ptyprocess                0.5.2            py27h70f6364_0
pyasn1                    0.4.2
pyasn1-modules            0.2.1
pygments                  2.2.0            py27h1a556bb_0
python                    2.7.14              hde5916a_29
python-dateutil           2.6.1            py27hd56c96b_1
PyYAML                    3.12
pyzmq                     16.0.3           py27h91ccc67_0
readline                  7.0                  hc1231fa_4
requests                  2.18.4
requests-oauthlib         0.8.0
rsa                       3.4.2
scandir                   1.6              py27h97aa1ee_0
scikit-learn              0.19.1           py27h9788993_0
scipy                     1.0.0            py27h793f721_0
setuptools                38.4.0                   py27_0
simplegeneric             0.8.1            py27h6db5e31_0
singledispatch            <IP_ADDRESS>          py27he22c18d_0
six                       1.11.0           py27h7252ba3_1
sqlite                    3.21.0               h3efe00b_2
ssl_match_hostname        <IP_ADDRESS>          py27h8780752_2
subprocess32              3.2.7
tk                        8.6.7                h35a86e2_3
tornado                   4.5.3                    py27_0
traitlets                 4.3.2            py27hcf08151_0
urllib3                   1.22
wcwidth                   0.1.7            py27h817c265_0
websocket-client          0.46.0
wheel                     0.30.0           py27h677a027_1
zeromq                    4.2.2                ha360ad0_2
zlib                      1.2.11               hf3cbc9b_2
(clipper27) mauricemanning:clipper
But running into a strange import issue with conda.api
18-01-30:15:14:30 INFO     [deployer_utils.py:49] Saving function to /tmp/clipper/tmp9aO7DQ
18-01-30:15:14:33 INFO     [deployer_utils.py:58] Anaconda environment found. Verifying packages.
18-01-30:15:14:34 INFO     [deployer_utils.py:158]
18-01-30:15:14:34 INFO     [deployer_utils.py:159] Traceback (most recent call last):
File "/Users/mauricemanning/anaconda3/envs/clipper27/lib/python2.7/site-packages/clipper_admin/deployers/check_and_write_deps.py", line 2, in
from conda.api import get_index
ImportError: cannot import name 'get_index'

AttributeError                            Traceback (most recent call last)
 in ()
1 # Deploy the 'predict' function as a model
2 deploy_python_closure(clipper_conn, name="tree-model",
----> 3                       version=1, input_type="doubles", func=predict)
/Users/mauricemanning/anaconda3/envs/clipper27/lib/python2.7/site-packages/clipper_admin/deployers/python.pyc in deploy_python_closure(clipper_conn, name, version, input_type, func, base_image, labels, registry, num_replicas, batch_size)
181     serialization_dir = save_python_function(name, func)
182     # Special handling for Windows, which uses backslash for path delimiting
--> 183     serialization_dir = posixpath.join(*os.path.split(serialization_dir))
184     logger.info("Python closure saved")
185     # Deploy function
/Users/mauricemanning/anaconda3/envs/clipper27/lib/python2.7/posixpath.pyc in split(p)
83     """Split a pathname.  Returns tuple "(head, tail)" where "tail" is
84     everything after the final slash.  Either part may be empty."""
---> 85     i = p.rfind('/') + 1
86     head, tail = p[:i], p[i:]
87     if head and head != '/'*len(head):
AttributeError: 'bool' object has no attribute 'rfind'
Any help would be great.

Hello
Thanks for the help.
I installed anaconda 4.3.1 on an Ubuntu instance inside of vagrant on my
Mac. Then I installed Docker-CE, then I installed Clipper from Github.
Attempted to run the test script but I receive:
In [1]: from clipper_admin import ClipperConnection,
DockerContainerManager
In [2]: clipper_conn = ClipperConnection(DockerContainerManager())
In [3]: clipper_conn.start_clipper()
18-02-01:00:29:00 WARNING  [clipper_admin.py:114] Error starting Clipper:
Unable to Connect to Docker. Please Check if Docker is running.

ClipperException                          Traceback (most recent call last)
 in ()
----> 1 clipper_conn.start_clipper()
/home/vagrant/anaconda3/envs/clipper27/lib/python2.7/site-packages/clipper_admin/clipper_admin.pyc
in start_clipper(self, query_frontend_image, mgmt_frontend_image,
cache_size)

113*         except ClipperException as e:

114*             logger.warning("Error starting Clipper: {}".format(e.
msg))

--> 115             raise e

116*

117*     def connect(self):

ClipperException: Unable to Connect to Docker. Please Check if Docker is
running.
However, Docker is running. The Hello World test runs as expected. Is it
possible to run Clipper inside a Vagrant container? I don't have a
different dev environment currently and I prefer not to uninstall Anaconda
on my Mac to roll back to a prior version.
Again, thanks for your help
Maurice
On Wed, Jan 31, 2018 at 12:42 AM, Sungjun, Kim<EMAIL_ADDRESS>wrote:

I think that the latest anaconda(4.4.8) is not compatible with Clipper. I
recommend to use anaconda 4.3.X instead of 4.4.8.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ucbrise/clipper/issues/382#issuecomment-361654576,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAhildgGaWIdxpK_9uiRPk1WrZc4OON0ks5tP0ZtgaJpZM4Rx4tH
.

ok, was able to get Clipper running on that version of anaconda in an
Ubuntu instance on Azure.
On Wed, Jan 31, 2018 at 12:42 AM, Sungjun, Kim<EMAIL_ADDRESS>wrote:

I think that the latest anaconda(4.4.8) is not compatible with Clipper. I
recommend to use anaconda 4.3.X instead of 4.4.8.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ucbrise/clipper/issues/382#issuecomment-361654576,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAhildgGaWIdxpK_9uiRPk1WrZc4OON0ks5tP0ZtgaJpZM4Rx4tH
.


Compile all bindings

Fixes #547

Build failure on macOS is due to lack of a provisioning profile.


zero-decimal currencies are beeing divided by 100

According to the documentation, CLP (Chilean Peso) is classified as a zero-decimal currency. However, in our application, CLP values are being divided by 100, which is inconsistent with how zero-decimal currencies should be handled.
Description
We are experiencing an issue where prices in CLP are incorrectly processed as if they were decimal-based currencies. This results in all our CLP prices being divided by 100, leading to incorrect price calculations and displays in our application.
Expected Behavior
For zero-decimal currencies like CLP, the price should not be divided by 100. The amount should be treated as the smallest unit of the currency without any conversion.
Current Behavior
All amounts in CLP are divided by 100, which is the standard handling for decimal-based currencies. This leads to incorrect price representations, significantly impacting the user experience and pricing accuracy in our application.
Steps to Reproduce
Set the application currency to CLP.
Input or retrieve a product price (e.g., 1000 CLP).
Observe the price being processed and displayed as 10 CLP.

Thanks for pointing this out. It's actually an issue in our medusa-react package, this helper specifically. Please create an issue on our main repo so the core team will see it.
I'm currently working on an update for the starter that doesn't rely as much on medusa-react, and this specific list will be directly editable in the Starter project. I'll add CLP to it right away!


Add CDN distributable

This PR adds the CDN distributable for the Gleam highlight.js grammar.
I followed the steps here to generate this.
Resolves #5.

Sweet. Could you document how to do this? Or add a shell script or something.

Yep, documentation may be easier, as it involves cloning down the highlight.js repo, cloning this repo inside of that one, and then running a script from the parent repo.

Just made a quick script that does the same thing as the one from HLJS.
import * as esbuild from "esbuild";
import fs from "node:fs";

function stripWhitespace(text) {
    return text.replace(/(\n|  )*/g, "");
}

const rawBuildResult = await esbuild.build({
    entryPoints: ["index.js"],
    bundle: true,
    minify: true,
    target: "es6",
    format: "iife",
    globalName: "hljsGleamGrammar",
    write: false,
});
const bundledText = rawBuildResult.outputFiles?.[0].text;

if (fs.existsSync("./dist/") === false) {
    fs.mkdirSync("./dist/");
}

// ESM
fs.writeFileSync(
    "./dist/gleam.es.min.js",
    stripWhitespace(
        `${bundledText}
        export default hljsGleamGrammar;`
    ),
);

// IIFE
fs.writeFileSync(
    "./dist/gleam.min.js",
    stripWhitespace(
        `(()=>{
            ${bundledText}
            hljs.registerLanguage("gleam",hljsGleamGrammar)
        })();`
    ),
);


Request admin api port failed

Instruction
Local env before: cli v0.32.3 + fcl v0.0.77 + dev-wallet + emulator and everything is  fine.
When I try to upgrade @onflow/fcl to 1.0.0-alpha.1 , before each script ,sdk-send-http.module.js send a request to http://localhost:8080/v1/scripts, this request failed.
upgrade cli to v0.33.1 ,the same problem;
access to testnet, the same problem;
Problem
request
http://localhost:8080/v1/scripts?block_height=sealed
response: status code 500
gRPC requires HTTP/2

Thank you for reporting this, I will try to reproduce this issue and get back to you shortly. It looks like some serious regression happened.

I think this is still an issue just want to confirm?

You should configure to use the port 8888 as that is the port used for HTTP API. Moving to FCL alpha uses HTTP API instead of gRPC API so the access node API needs to be changed accordingly.


Strawberry Shake's Optional can be problematic

As far as I understand, the properties of classes that Strawberry Shake generates that represent for instance input objects, are wrapped by a generic struct called Optional<T> (the name can be very misleading, by the way) and what this does is it allows Strawberry Shake to distinguish between properties that are explicitly set to their default value, and fields that are not set at all.
However, this appears to cause problems in many situations, the most significant of which that I've encountered is that those generated classes then can't be used with Blazor's <EditForm> and <InputText>, etc. component, which is a common scenario, consider the following example:
Suppose we have an input object type called SignUpInput, defined in our SDL, like this:
input SignUpInput {
    email: String!
}

Strawberry Shake generates the following class from this input object type:
[System.CodeDom.Compiler.GeneratedCode("StrawberryShake", "11.0.0")]
public partial class SignUpInput
{
    public Optional<string> Email { get; set; }
}

We also have a Blazor component, called SignUpForm.razor:
<EditForm Model="formModel">
    <InputText @bind-Value="formModel.Email" />
</EditForm>

@code {
    SignUpInput formModel = new SignUpInput();
}

But the part where it's binding the Email property to the InputText component (@bind-Value="formModel.Email") doesn't work, and causes the following errors:
Argument 2: cannot convert from 'Microsoft.AspNetCore.Components.EventCallback<StrawberryShake.Optional<string>>' to 'Microsoft.AspNetCore.Components.EventCallback'.

Cannot convert lambda expression to intended delegate type because some of the return types in the block are not implicitly convertible to the delegate return type.

This problem wouldn't occur if SignUpInput.Email's type was string, as opposed to Optional<string>.
Is there anything I can do about this?
Tell me if I'm missing something.
I think it's worth noting that this is not the only case where Optional<T> causes headaches, another case would be if you want to use FluentValidation for these input object types, which I can expand on more if you want to know.
Can't there be a different mechanism for detecting set/unset properties?!
Thanks.

We are working on this... with the next preview of strawberry shake this is fixed and you also will have an option to completely opt-put of Optional.

We are working on this... with the next preview of strawberry shake this is fixed and you also will have an option to completely opt-put of Optional.

@michaelstaib Great to hear. Thanks! Looking forward to seeing the first preview.

@michaelstaib Great to hear. Thanks! Looking forward to seeing the first preview.

The new preview now no longer requires optional.


I cannot use CPU 100%

Expected behavior
[Describe expected behavior here]
Actual behavior
[Describe observed behavior here]
Steps to reproduce the behavior
hi, @amir20
as you said, it takes a lot of CPU and memory,I had two problems as blow:
1、I tested in diff machine with limit phantom num blow 200, but in some can start 60 phantom instances and some be 80 and some be 140 and some be 200, CPU and memory of all are not overload,what's the problem can lead to this, how can i use the cpu and memory more effective?
2、base on step 1, all of them I can consume 10 requests per second,no matter 80 or 200 phantom instances, is there some settings I did not cover?
how it works?
Thanks in advance.

OS: mac osx
Node version (node -v): 0.12.x
package version: 2.1.15

You can never have that many instances running on one machine. For starters, upgrade to latest node. You need to have multiple machines to be able to scale horizontally.
If you set up 10 machines where each can process 10 then you can process a total of 100.
Sorry I can't help more. Maybe try stackoverflow.
Sent via my iPhone

On Aug 30, 2016, at 2:58 AM, skysilence<EMAIL_ADDRESS>wrote:
Expected behavior
[Describe expected behavior here]
Actual behavior
[Describe observed behavior here]
Steps to reproduce the behavior
hi, @amir20
as you said, it takes a lot of CPU and memory,I had two problems as blow:
1、I tested in diff machine with limit phantom num blow 200, but in some can start 60 phantom instances and some be 80 and some be 140 and some be 200, CPU and memory of all are not overload,what's the problem can lead to this, how can i use the cpu and memory more effective?
2、base on step 1, all of them I can consume 10 requests per second,no matter 80 or 200 phantom instances, is there some settings I did not cover?
how it works?
Thanks in advance.
OS: mac osx
Node version (node -v): 0.12.x
package version: 2.1.15
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.


I'm unable to run this app locally

I have added url and anon key from supabase but i'm getting the below error.
Uncaught TypeError: auth.user is not a function
at Header (Header.jsx:22:13)
at renderWithHooks (react-dom.development.js:14985:18)
at mountIndeterminateComponent (react-dom.development.js:17811:13)
at beginWork (react-dom.development.js:19049:16)
at HTMLUnknownElement.callCallback2 (react-dom.development.js:3945:14)
at Object.invokeGuardedCallbackDev (react-dom.development.js:3994:16)
at invokeGuardedCallback (react-dom.development.js:4056:31)
at beginWork$1 (react-dom.development.js:23964:7)
at performUnitOfWork (react-dom.development.js:22776:12)
at workLoopSync (react-dom.development.js:22707:5)
this is the error message for /auth/v1 in postman: no Route matched with those values

The syntax in supabase-js v2 is a little different.
- supabase.auth.user()
+ supabase.auth.user

Maybe have a look at my recent changes in PR-18


ROX-20291: Move FE sorting of Policies table to avoid re-render

Description
This PR refactors the client-side sorting on the Policies table, by moving it up from the PoliciesTable.tsx component file, up to its parent PoliciesTablePage.tsx component. This is done to prevent the layout shift that occurs when the table was being rendered with an empty array, and then fed an unsorted API response, and then immediately re-rendered with the sorted list, which was sorted in a useEffect block. The initial render did not have wide empty header for the turndown triangle column--PatternFly did not render that header if there were no rows--and the subsequent layout shift was causing frequent "element is animating" failures in CI.
I decided move the sorting up into the promise-resolved block for the /policies API call in the parent, because that API call gets refreshed on Import, as well as Deletion and Reassess All functions. I could have attempted a more radical refactoring to move the API call down, but this seemed like a safer, easier-to-review option.
After the Policies API is paginated in the API--which should come soon--we can use start passing the URL sort parameters that I have added here directly to the API call. That's why I decided to take the performance hit of making the API call for every change in sort, because it will make moving to backend sorting easier.
Checklist

[ ] Investigated and inspected CI test results

Testing Performed
Here I tell how I validated my change

Inspect existing CI
Manual testing of each column sorted in each direction

Default with no sort specified in URL, sorts on Policy (name) column ascending (note the API response is not sort--sorting still happens on FE)

Clicking on Policy column header adds sort parameters to the URL and reverses the sort on Policy column

Clicking on Status column header sorts that column ascending

Clicking on Status column header again reverses its sort

Clicking on Origin column header sorts that column ascending

Clicking on Origin column header again reverses its sort

Clicking on Notifiers column header sorts that column ascending

Clicking on Notifiers column header again reverses its sort

Clicking on Severity column header sorts that column ascending

(note: I don't show clicking on Severity column header again to reverse it, but it works--it's just not an interesting use case, and I'm testing if the reviewer is paying attention :) )
Clicking on Lifecycle column header sorts that column ascending

Clicking on Lifecycle column header again reverses its sort

Reminder for reviewers
In addition to reviewing code here, reviewers must also review testing and request further testing in case the
performed one does not seem sufficient. As a reviewer, you must not approve the change until you understand the
performed testing and you are satisfied with it.

...
I might lose sleep though wondering if the severity column really does sort ascending...

Alright, here is proof


llm-providers need proxy agent supports

https://github.com/sourcegraph/cody/blob/1f926b54570591658310a6735f51f66db666f2e1/lib/shared/src/llm-providers/google/chat-client.ts#L50

Hey @Luoyayu
Thank you for reaching us out.
For feedback or feature suggestions, please use our https://community.sourcegraph.com/ community forum instead.
Thank you


chore(release): 2.46.0

See CHANGELOG

AWS CodeBuild CI Report

CodeBuild project: githubautobuild-for-cdk-v2
Commit ID: 822c7b2fae22b17b82e23d50813b3448412d7d65
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: githubautobuild-for-cdk-v2
Commit ID: f24161a5c77e08a544df6373c50aacadf9661ea3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: githubautobuild-for-cdk-v2
Commit ID: eb622e0da60f609e0dc1aca6f631a43a2e422e21
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: githubautobuild-for-cdk-v2
Commit ID: 50e9396777f768bd32cab4a796e33cad9adc09f9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: githubautobuild-for-cdk-v2
Commit ID: 6c447391ffc49a810adae57112b7e771df5c3ff4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: githubautobuild-for-cdk-v2
Commit ID: 36bab135ae58878f650b07ded1e1d6171432d14f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: githubautobuild-for-cdk-v2
Commit ID: 55df93cd4abd5e27379a75b69b8dd3bdccb1727a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: githubautobuild-for-cdk-v2
Commit ID: e2428a3d7e2cbad6ce630ba8d7163f0ac7dbe89e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository


CLI has no way to delete a snapshot

title

I might completely get this issue wrong, but lxc delete <container>/<snapshot> will delete only the snapshot

What @jsimonetti said.
To remove a snapshot:
lxc delete container/snapshot

To rename a snapshot:
lxc move container/snapshot container/new-name


Use go generate to update and validate GitHub v3 API URLs

Fixes #1476.

Thank you, @wesleimp!
Since this is a non-trivial addition and affects future maintenance, I would like to also get a review from @willnorris to make sure he is on-board with this change before going ahead with it.
No rush, @willnorris - nothing is waiting on this. Thank you!

I haven't done a thorough review of the generation code, but it certainly looks to be pretty clever (and probably a lot of fun to write!).  But it's also rather complex and hefty (and potentially fragile), for maintaining what is a very minor aspect of our godocs.  In fact, I think I started include the GitHub doc URLs somewhat on a lark, as just an extra "nice to have".
I think the number of changes in the doc URLs certainly shows how out of date they are, so I'm totally supportive of whatever tools make it easier to keep those up to date.  I'm just worried about the ongoing maintenance of it.  What do you think of pulling it out of the main github folder, similar to what we did with the scrape package?  Maybe even make it its own go module, so it doesn't get downloaded as part of the main module when someone is just using go-github, but not actually contributing to it?
wdyt?

wdyt?

Great idea!  I like it.  Thanks, @willnorris !  I'll work on that and update this PR when I've got it moved.

Tool moved to "update-urls" subdir with its own "go.mod" and "go.sum".  PTAL (low priority).
Thank you, Will!

That makes sense, Will... thank you!
I'll pull those breaking changes out into a separate PR and not merge this one yet until that one is in.

This PR is now on hold until #1487 has been merged, which is a prerequisite for tests to pass.

This PR now contains only changes to comments and the addition of the update-urls tool.
Merging.


Spinning food model

New functionality

A 3d model of a bowl of ramen is rendered on the front page

By default, the bowl alternates between slightly rotating clockwise and slightly rotating counter-clockwise
While the lottery is in progress, the bowl spins around counter-clockwise
When the lottery results are shown, the bowl is hidden until the lottery is restarted

Notes

The p5 script can be found in packages/app/src/scripts/food.js
The code is well commented to explain what is happening - for more information, refer to the p5.js reference
When the restaurant result is shown, the canvas's CSS display property is set to none to avoid re-loading the assets for every consecutive lottery shuffle
When leaving the front page, the canvas is destroyed - this means that the assets will have to be loaded again when returning to the front page
The assets are served from packages/app/src/data/assets/
Basic tests check that the canvas is visible/hidden when it should be.

Administrative tasks

Created a small wrapper to avoid repeated code when creating p5 canvases
Moved all p5 related React components to components/p5Components/
Created a scripts/ directory to the front-end root for p5 scripts - after all, they are not services and should not be in the services/ directory
Updated the #root element's background color to match the animation's background. This was done purely to make the visual outlook unified for now, the colours will likely be adjusted later on.

Well, everything looks OK. I just don't like the way a single model has to be split to 9 parts for coloring things with different colors. See proof-of-concept of flattened, textured model on how this could be handled in the model instead of in the code.
What I've done there is:

I imported the model to blender and joined all parts into a single object with multiple materials.
Changed to blender's image editor view. There I created a new black texture with size 1024x1024 pixels.
Changed to the texture node editor and assigned a dummy image texture node (not connected to anything but selected) with the image created in step 2. selected. This setup is the very unintuitive way of telling blender where to paint the colors of those materials later on. Note that the selection of the image texture node remains even after you change to another material or view.
Now, I marked seams on the model and UV unwrapped it
When happy with UV layout, I changed the renderer from EEVEE to Cycles and baked the diffuse colors. For this, I selected diffuse from the dropdown and selected only colors to be baked. (Make sure to select all image texture nodes in individual materials' texture node editors and the model to be baked. Then go to render tab, disable selected to active and hit bake) As we "assigned" our target image to materials in step 3., the colors have now magically appeared on our image texture.
Created clone of the model and cleared its materials, created a single new material and assigned it the generated image texture.
Presto! Now we have a textured model with a single material.
P5.js for some reason uses inverted UV-coordinates so the image still needs to be vertically flipped in image editor before saving to file.
Now, load the single model and the texture with P5.js and assign the texture when rendering.

Notes:

The UV layout is really lazy on the POC-branch, the noodles take the most of the texture space even though they could very well be overlaid on top of each other to make manual edits with Photoshop more pleasant experience.
Having to manually flip the image texture is kind of annoying but I didn't find a quick-fix for this from P5 docs. There most likely is some configuration flag or property somewhere, but I didn't find it.
We lose the ability to adjust specularity on per-part basis. Either everything is specular or nothing is specular. For refined specularity adjustment, a separate specularity map should be baked.


A bunch of letterforms are missing from defaultLetterMap

As near as I can tell, there are quite a few letterforms missing from the defaultLetterMap.
λ> sort $ M.elems $ OCR.getLetterMap OCR.defaultLetterMap
"AABBCCEEFFGGHHIJJKKLLNOPPRX"

It looks like it's missing one of the Rs, the U, the Y, and both Zs.

I think the problem is that spurious spaces are getting into the string literal somehow:
λ> raw1 = bimap id (filter (/=' ')) $ OCR.rawLetterforms1
λ> sort $ M.elems $ OCR.getLetterMap $ uncurry OCR.parseLetterMap raw1
"ABCEFGHIJKLOPRUYZ"

Maybe a problem with heredoc?

Ah, thank you...this is definitely interesting, and concerning!  I wonder what is going on here...it does sound like it could be a problem with the heredoc.  I should probably also build some fuzz testing for this instead of relying on some built in forms.

I've added a filter to the heredocs, this seems to resolve the issue for now. But I do wonder what's happening.

Ah! It might be a locale thing, since checking defaultLettermap on my machine gives the full alphabet.  What OS are you using?

I'm on MacOS 10.13.

hm..definitely interesting.
i'm closing this issue because I've implemented the filtering change mentioned, but it would be interesting to investigate this further on the heredoc side. thanks opening!


helm: add "inifnity" as an option to scheduler requeue count docs

closes reanahub/reana#502

Codecov Report

Merging #645 (3f6050e) into master (2ef1175) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 3f6050e differs from pull request most recent head 78d00c3. Consider uploading reports for the commit 78d00c3 to get more accurate results

@@           Coverage Diff           @@
##           master     #645   +/-   ##
=======================================
  Coverage   18.84%   18.84%
=======================================
  Files          26       26
  Lines        2138     2138
=======================================
  Hits          403      403
  Misses       1735     1735


Support for HEC raw endpoint

It would be neat if this also supported the HTTP Event Collector raw endpoint using parameterized values for sourcetype, host, index and source through the URL query as well as the channel ID.
See http://dev.splunk.com/view/event-collector/SP-CAAAE8Y
I have a client using Logback asking for this. They'd like to send properly formatted logs out of their Docker/Openshift env without going through the openshift stdout logs that make the logs look like gibberish.

Thanks @inspired, this is something we'd love to see but it will require a non-trivial engineering effort. That said, we are open to a pull request

https://github.com/splunk/splunk-library-javalogging/pull/75
This a port of https://github.com/splunk/splunk-library-javalogging/pull/58


I used the official demon to query the wallet address information incorrectly

Exception in thread "main" com.binance.dex.api.client.BinanceDexApiException: BinanceDexApiError[code=400,message=decoding bech32 failed: checksum failed. Expected a955xp, got 07f2d2.]
at com.binance.dex.api.client.BinanceDexApiClientGenerator.executeSync(BinanceDexApiClientGenerator.java:82)
at com.binance.dex.api.client.impl.BinanceDexApiRestClientImpl.getAccount(BinanceDexApiRestClientImpl.java:77)
at TestBinance.main(TestBinance.java:17)

Have you managed to get the sdk set up? we all have errors due to that missing ethereum jar. Could you share the file if you have it? thanks (see other open issues on the matter)

@appfile01

jcenter
https://maven.aliyun.com/repository/public

use this repository


Add API for setting client config

Based of #132

I am not a huge fan of describing this everywhere as a 'client config' API.  For instance, letting users specify the accent colour for a given room (SPEC-259, one of the use cases for this) is not really "client configuration", which to me implies that it's specific to a given client application.  Why can't we just call it private user data?

It might be nice to expose GET paths for:

/user/{userId}/rooms/{roomId}/config
/user/{userId}/config

As for naming... I don't really like "client config" since it's possibly the vaguest thing in the world ever (configuration per Client (device) or Client (user)? Config is wrt... your account? Your app? A room?). Whilst I prefer "private user data", I don't like the "private" part since it's not clear what about the user data is "private". The intention is that this is a user-specific store of data which is shared between all clients for a given user. Maybe something like:

personal user data?
account user data? (given it is tied to your account)
personal account data?
account data?

"Account data" sits well with me as we don't horrendously abuse the word "account" elsewhere (whereas "client" and "user" are always mixed up). It also emphasises that it is to do with, well, your account which has connotations of it being shared between your devices, private to you, etc.

Assigned to @ara4n for feedback.

I could live with account_data.  It feels unnecessarily complicated that it doesn't match the 'private_user_data' name that we receive in the sync response, but it's not the end of the world.

It feels unnecessarily complicated that it doesn't match the 'private_user_data' name that we receive in the sync response,

These should be consistent. If we call it account_data then the sync response should be account_data too.

Renamed private_user_data to account_data fcbb985, 40f7eab


Blur effect bug in the main Activity

Steps to reproduce
Open the main Activity to see the wallpaper in full
Lock the screen
Unlock again
The wallpaper gets blurred and stuck in that state.
After this it cannot be scrolled and double click does not work to unblur it.

Interestingly, I can't seem to reproduce this on the current master. Please reopen it if you see it crop up in the future (preferably in a beta build so that it can be fixed before going to production).

I can reproduce it with the latest beta. When is the next beta release from Master?

Muzei 2.5 Beta 1 is out now!

I got that too! But then if you switch to the Sources tab and back to Muzei, the blur goes away

Could it be device dependant? I just updated to latest beta and the bug is still here. I have OnePlus 5 with Oreo.

I'm not having any luck reproducing this or figuring out how this is happening. Could you try (in this order):

Restarting your device
Switching to a different wallpaper, then re-activating Muzei

Did both, still there. I will check with a different  device


Add shortcut to take screenshot for the whole window

Primary+Shift+P should take the screenshot and store it in the image's directory.
Ideally, it should also store the file full path name in the clipboard.


Fix minimum charge value validation on Voucher edition

This is a bug that happens when you are logged in on Backend with German language selected.
Shopware allows you to use comma on minimum charge value. That means if you are creating a voucher with an Absolute value, you can configure the minimum charge < voucher value, and this isn't valid.

Example:
Considering the conditions above described, you can create a Voucher with minimum charge = 3,95 and value = 150. That means you just need 3,95€ on your cart to apply the voucher of 150€.

Questions
Answers

Why?
It's an insecure behavior, enabling customers to apply vouchers in inconsistent conditions. It broke the cart logic on frontend side, when you apply a voucher with this kind of configuration and you cart total amount is less than voucher value

BC breaks?
no

Tests exists & pass?
yes

Related tickets?

How to test?
Loggin on backend with German language selected. Create a Voucher with minimum charge = 3,95 and value = 150 (invalid voucher) Go to frontend do some shop and apply the voucher to your cart when you have more than 3,95€ and less than 150€

Requirements met?
Does your PR fulfil our contribution requirements?

Hello,
thank you for creating this pull request.
I have opened an issue on our Issue Tracker for you. See the issue link: https://issues.shopware.com/issues/SW-18550.
Please use this issue to track the state of your pull request.

Thanks @nevoeiro, can you please squash your commits into one?

This PR was closed due to inactivity. If this change is still important to you, feel free to create a new pull request.
For more information about our contribution guidelines, see https://developers.shopware.com/contributing/contribution-guideline/.


[Dropdown] multiple selection labels not showing up on samsung internet

Tested on:

Android 7 - default samsung internet browser

Using the default samsung internet browser, labels in multiple selection are not showing up.
On every selected element, it produces:

Transition: Element is no longer attached to DOM. Unable to animate.  Use silent setting to surpress this warning in production. scale

Any news on this one?

Hi @leonidas-o, I don’t own a Samsung, but I did a quick search, and it seems possible to somehow install it on any Android device. I’ll try to debug it this weekend.

okay, nice. Thank you

@leonidas-o I couldn’t reproduce. I grabbed the latest version of the browser that’s <IP_ADDRESS> (was released around 19th of July), and labels do show up. I was unable to setup remote devices in Chrome’s developer tools, because Internet was crashing upon accessing my development machine, so I don’t know if there were any console errors, but everything seemed to be working as expected.
What version of Samsung Internet were you using when you noticed that?

@Banandrew I got all updates installed, Neither Google's Play Store nor the Galaxy Apps App is showing any updates on the Samsung Internet Browser.
So I would say the official released version is the one I'm using:
version 5.4.00-75

@leonidas-o Confirming the issue on 5.4.00-75—I’m assuming I got an unstable version. The good news is that the issue will resolve itself once the stable branch is updated, the bad is that I can’t fix it on 5.4.00-75, because the browser crashes when I try to do remote debugging to have access to desktop’s developer tools.

@Banandrew Today I got an update of the Samsung Internet browser, Now it is on <IP_ADDRESS>. The issue is also on this version present. Maybe you are more lucky trying to debug this version.

@leonidas-o Sorry for the delay, this version stopped crashing, so I was able to connect to a PC, but not for long—crashing returned and I was unable to investigate the issue further. Samsung Internet does something interesting with the DOM, and the transition component can’t transition labels (as well as modals—#5341), because they’re “no longer attached to DOM”, even though I could see them in the DOM. Somehow, this might be the cause of crashing reported in #5341, as well as crashes that I’m experiencing while being connected to a PC.
I don’t know how to fix it, and I can’t even start debugging it. My brother owns a Samsung, I should be able to meet him soon, and I’ll try to use his phone to do the same. If it’ll be crashing, I don’t think we’ll ever be able to fix it, unless Samsung updates their browser.

Any update on this issue? This issue is pretty bad :( , since a lot of users are using Samsung Internet.
It's not possible to debug using http://developer.samsung.com/remotetestlab/rtlDeviceList.action# ?

@Banandrew Just a hint, to debug on Samsung Internet, you can install it on non-Samsung devices:
https://www.androidauthority.com/samsung-internet-browser-android-793983/
https://play.google.com/store/apps/details?id=com.sec.android.app.sbrowser&hl=en
or preview the changes on Samsung Developers
http://developer.samsung.com/remotetestlab/rtlDeviceList.action#

Hi @aguyinmontreal, that’s what I did. I didn’t have a chance to use my brother’s Samsung for debugging, so my last post lead to nothing. I was planning to try their remote test lab, but haven’t had the time to do it; I will if I won’t be busy until January.
Since we’re talking about it again, what’s the current version of the browser in Google Play, did they update it?

@Banandrew   it doesn't say the version, but it says it has been last updated on October 25, 2017.  There is also a "beta" version on GooglePlay https://play.google.com/store/apps/details?id=com.sec.android.app.sbrowser&hl=en which is version <IP_ADDRESS> https://play.google.com/store/apps/details?id=com.sec.android.app.sbrowser.beta&hl=en

@Banandrew Maybe you would want to explain the issue here:  https://github.com/SamsungInternet/support/issues  (Samsung browser issues)

@aguyinmontreal Thank you, I’ll see if I can gather more information and will go from there.


Add setting for tmux.conf

The reload config item defaults to ~/.tmux.conf, though the world seems to be moving to using $XDG_CONFIG_HOME/tmux/tmux.conf (a.k.a ~/.config/tmux/tmux.conf usually). It would be nice to be able to set this instead of typing it every time (or if possible, inferring it from the current tmux session, but I don't know if that' possible).

makes sense, will do it a config item within a day or so

I have updated this to support the requested feature, check the README and search for "Pointer to config file" for details.


[Event Request] Codeunit 80 Sales Post - OnBeforeRunWithCheck2

Hi, would it be possible to get an event OnBeforeRunWithCheck2 in Codeunit 80 Sales Post, procedure PostItemChargePerOrder, like this

                        ItemJnlLine2."Location Code" := NonDistrItemJnlLine."Location Code";
                        NonDistrItemJnlLine."Quantity (Base)" -= ItemJnlLine2."Quantity (Base)";
                        NonDistrItemJnlLine.Amount -= ItemJnlLine2.Amount;
                        NonDistrItemJnlLine."Discount Amount" -= ItemJnlLine2."Discount Amount";
                    end else begin // the last time
                        NonDistrItemJnlLine."Quantity (Base)" := -"Quantity (Base)";
                        NonDistrItemJnlLine."Invoiced Qty. (Base)" := -"Quantity (Base)";
                        NonDistrItemJnlLine."Unit Cost" :=
                          Round(NonDistrItemJnlLine.Amount / NonDistrItemJnlLine."Invoiced Qty. (Base)",
                            GLSetup."Unit-Amount Rounding Precision");
                        NonDistrItemJnlLine."Item Shpt. Entry No." := "Item Ledger Entry No.";
                        NonDistrItemJnlLine."Applies-to Entry" := "Item Ledger Entry No.";
                        NonDistrItemJnlLine.CopyTrackingFromSpec(TempTrackingSpecificationInv);

                        IsHandled := false;
                        OnBeforeRunWithCheck2(IsHandled,SalesLine,NonDistrItemJnlLine);
                        if not IsHandled then
                            ItemJnlPostLine.RunWithCheck(NonDistrItemJnlLine);

                        NonDistrItemJnlLine."Location Code" := ItemJnlLine2."Location Code";
                    end;
                until Next() = 0;

    [IntegrationEvent(false, false)]
    local procedure OnBeforeRunWithCheck2( IsHandled : Boolean; SalesLine : Record "Sales Line", NonDistrItemJnlLine : Record "Item Journal Line")
    begin
    end;

?
Thanks

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.