write_metadata in DataFrame.to_parquet kills worker

While executing
df.to_parquet(
    path,
    engine="pyarrow",
    compression="gzip",
    partition_on=["part_col"],
    append=False,
    write_index=False,
)

on a dask.DataFrame, the write_metadata task repeated fails as it crashes the worker where it is executed and triggers a recalculation of any dask.delayed results held on that worker.  After exceeding the retry limit, the function throws an exception.
The DataFrame is constructed from a set of 2500 dask.delayed partitions.  The partitions all fit easily within memory; however, when the write_metadata task is submitted, it appears to kill the worker due to the memory limit being exceed.  There is nothing explicitly logged when the worker dies in the worker or the scheduler as to what killed the worker.  The target file system is s3; however, it does not seem like a likely explanation for this issue.
My expectation is that {"schema": t.schema, "meta": _meta} which is submitted to write_metadata is reasonably small; however, this does not seem to be the case. It doesn't seem like desirable behavior to pass  the full list all a once when these could easily processed incrementally.

Do you have a reproducible example @mcguipat? See http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

My expectation is that {"schema": t.schema, "meta": _meta}

That's the intent (though I haven't verified that's how things are implemented).
cc @rjzamora.

I'd be happy to look into this if I had a reproducer.  Perhaps something that that demonstrates that {"schema": t.schema, "meta": _meta} is too large - even if it requires a print or pdb statement to actually observe the issue.  I know it may be tricky to come up with something simple, but it would be super helpful.

I've spent some time here attempting to make a reproducible example; however, it's proving meaningfully more difficult than I had anticipated.  The original issue is present when running against a KubeCluster; however, when attempting to execute against a LocalCluster of similar spec, it does not seem to occur.  I'm continuing to try to find a way to reproduce/investigate the root cause.

@mcguipat you might find this page helpful: https://docs.dask.org/en/latest/debugging.html#logs

I just hit this in the last couple of days -- I suspect it's due to schema mismatch when using pyarrow
@mcguipat -- does the same thing happen if you add schema='infer' to your to_parquet call?

@gforsyth I've been investigating this and have run into a potentially related issue.
I don't see memory explosion as detailed in the original report but I've identified a bug whereby the pyarrow write_metadata implementation either throws or segfaults when empty dask dataframe partitions are processed.
If a partition contains an empty data frame a "meta" value of None is returned from write_partition  and results an exception/segfault depending on parameters
I'm submitting a PR to remedy this shortly but it may be worth checking for these conditions in your application.
Reproducible examples:
from dask import delayed
import dask.dataframe as dd
import faulthandler
import pandas as pd

faulthandler.enable()

path = "."

df_a = delayed(pd.DataFrame.from_dict)({"x": [], "y": []}, dtype=("int", "int"))
df_b = delayed(pd.DataFrame.from_dict)({"x": [1, 1, 2, 2], "y": [1, 0, 1, 0]})
df_c = delayed(pd.DataFrame.from_dict)({"x": [1, 2, 1, 2], "y": [1, 0, 1, 0]})

# 1. raises AttributeError
df = dd.from_delayed([df_a, df_b, df_c])
df.to_parquet(
    path,
    engine="pyarrow",
    partition_on=["x"],
    append=False,
    compute_kwargs={"scheduler": "synchronous"},
)

# 2. cores
df = dd.from_delayed([df_b, df_c, df_a])
df.to_parquet(
    path,
    engine="pyarrow",
    partition_on=["x"],
    append=False,
    compute_kwargs={"scheduler": "synchronous"},
)

https://github.com/dask/dask/pull/6741

I just hit this in the last couple of days -- I suspect it's due to schema mismatch when using pyarrow
@mcguipat -- does the same thing happen if you add schema='infer' to your to_parquet call?

I've been looking into this further and I cannot get to_parquet to complete with the following job:
Cluster:
10 workers @ 15 procs, 1 thread, 2GB mem limit
scheduler mem limit 20GB (grasping at straws here)
Dataframe with 2529 partitions
The issue is that the job makes progress until it's about 95% complete at which point it stalls indefinitely. You can see in the attached screenshot that the job makes progress then hangs for 6 hours... Worker memory usage increases continually as the job progresses and worker logs show lots of memory leak errors (which I expect is smoking gun).
I have attempted this with schema as None, "infer" and explicitly passed. I've also tried with both pyarrow and fastparquet engines none of which make any difference...
The only way I've managed to get the job to complete is to batch the partitions and call to_parquet repeatedly. Ideally we wouldn't just increase worker memory allowance.
Re. pyarrow engine:
I would expect that on each completion of a delayed write_partition call most memory would be released (loaded pyarrow table etc.) leaving only schema and meta objects on workers ready for write_metadata. I've also attempted pickling -> unpickling the schema and meta objects and returning those copies in write_partition on the off chance this would release any dangling references to mem in the underlying Cython classes, this had no effect.
Apologies for not providing a reproducible example but this issue only occurs at scale, any thoughts would be much appreciated.

Working on a similar dataset to @gforsyth and experiencing the same apparent issue, the worker, performing the write_metadata task, using increasing amounts of memory until the worker OOM terminates and the worker and task restart. The dataset we originally encountered this issue with has thousands of columns, 10s of thousands of partitions, and millions of rows running on a cluster with TBs of memory (64GiB workers). We speculate the OOM issue is a function of the number of columns and the number of partitions. See reproducer below, where to_parquet succeeds if the dataframe with 2,000 columns has one partition (10,000 rows) but fails if it has 1,000 (10 rows/partition).
dask version 2.30.0
distributed version 2.30.1
pyarrow version 1.0.1
import os
import shutil
import pandas as pd

os.environ['DASK_DISTRIBUTED__WORKER__MEMORY__TARGET'] = "False"
os.environ['DASK_DISTRIBUTED__WORKER__MEMORY__SPILL'] = "False"
os.environ['DASK_DISTRIBUTED__WORKER__MEMORY__PAUSE'] = "0.93"
os.environ['DASK_DISTRIBUTED__WORKER__MEMORY__TERMINATE'] = "0.95"
os.environ['DASK_DISTRIBUTED__SCHEDULER__ALLOWED_FAILURES'] = "20"
os.environ['DASK_DISTRIBUTED__COMM__TIMEOUTS__CONNECT'] = "5"
os.environ['DASK_DISTRIBUTED__COMM__RETRY__COUNT'] = "10"

import dask
import dask.dataframe as dd
from distributed import Client, LocalCluster
from sklearn.datasets import make_classification

cluster = LocalCluster(
    n_workers=4,
    processes=True,
    threads_per_worker=1,
    memory_limit="3GiB"
)

c = Client(cluster)

n_samples_list = [10_000]
n_features_list = [2_000]
npartitions_list = [1, 1_000]

path = "./test.parquet"
engine = "pyarrow"

for n_samples in n_samples_list:
    for n_features in n_features_list:
        for npartitions in npartitions_list:
            print(f"Testing n_samples={n_samples} n_features={n_features} npartitions={npartitions}")
            print(f"Rows/partition={n_samples/npartitions}")

            print("Creating dataframe")
            X, _ = make_classification(n_samples=n_samples, n_features=n_features)
            df = pd.DataFrame(X)
            ddf = dd.from_pandas(df, npartitions=npartitions)
            print("Created dataframe")

            print("Writing parquet")
            shutil.rmtree(path, ignore_errors=True)
            ddf.to_parquet(path, engine)
            shutil.rmtree(path, ignore_errors=True)
            print(f"Wrote {path}, removing")

            print("Restarting Dask Client\n")
            c.restart()

Output follows:
Testing n_samples=10000 n_features=2000 npartitions=1
Rows/partition=10000.0
Creating dataframe
Created dataframe
Writing parquet
Wrote ./test.parquet, removing
Restarting Dask Client

Testing n_samples=10000 n_features=2000 npartitions=1000
Rows/partition=10.0
Creating dataframe
Created dataframe
Writing parquet
distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting
distributed.nanny - WARNING - Restarting worker

Workers will restart until parent process is killed, as expected.

Thanks for the reproducer @dankerrigan !   I get the same behavior on my machine.  It certainly seems like the problem occurs when all the metadata is aggregated onto a single worker to write the "_metadata" file.  I'll try to dig in and see if we are doing anything silly here.

Awesome! Thanks @dankerrigan
@rjzamora I'm glad you can see the issue locally. Let us know if you need any help.

I'm glad you can see the issue locally. Let us know if you need any help.

I don't have a solution yet, but I did poke around a bit.  The failure in this reproducer is definitely due to the total size of the metadata being sent to the final write_metadata task....  Every write_partition is sending along a ~1.3MB object containing the schema as well as the parquet metadata for the row-groups written by that task.  We can avoid sending the schema in this object (only one of the partitions is used to write the _common_metadata file anyway), but that still leaves about 940KB of metadata from each partition.  Therefore, a 1000-partition write will need to result in a single task aggregating almost 1GB of data to write the _metadata file.  This can easily cause issues on a memory-constrained system.
Thoughts on possible solutions:

As already mentioned above, we can avoid sending the same schema from every write_partition task.  This will not fix the general scaling issue, but will avoid some of the memory overhead.
Although the total parts list is ~1GB in size, the final metadata blob is about 4x smaller, because the metadata for every partition includes data that will be thrown away within _append_row_groups.  This means we could probably reduce the memory pressure a bit further by using multiple intermediate tasks to aggregate the metadata.
I suspect that the only way to make metadata writing completely scalable is to work out a mechanism to append row-groups to the _metadata file without needing to aggregate them in memory first.  This shouldn’t be too difficult, but may require changes in pyarrow.

I am also facing a similar issue with large files, do we have any plan to fix this.

I'm seeing this also, with 2,500 partitions of a 500 column 80m record dataset. Everything processes well within memory constraints until the final metadata write, which fails. Annoyingly there's plenty of memory on the server for this last job, just not within a single worker with its fixed proportion of the total server memory.
I don't suppose there's a way of getting this to operate within a different memory constraint or on a dedicated, large memory worker as a temporary work-around?

So, I see a couple ways that we can solve this problem, neither is very hard:

Add the ability to turn off writing metadata.  This isn't strictly necessary for Parquet to work.  It's just a convenience and results in some efficiencies when reading.  We should have a flag to turn this off.  This shouldn't be too hard to add
Is the actual resulting metadata object huge, or is it just having a single task that has 1000's of dependencies?   If the latter (which I hope) we should solve this with a tree reduction.  We should accumulate the metadata in intermediate stages.  I suspect that this will also help the scheduling heuristics.

@rjzamora @jorisvandenbossche any thoughts on these?

Add the ability to turn off writing metadata. This isn't strictly necessary for Parquet to work. It's just a convenience and results in some efficiencies when reading. We should have a flag to turn this off. This shouldn't be too hard to add

There is indeed a write_metadata_file option in to_parquet

Is the actual resulting metadata object huge, or is it just having a single task that has 1000's of dependencies? If the latter (which I hope) we should solve this with a tree reduction. We should accumulate the metadata in intermediate stages. I suspect that this will also help the scheduling heuristics.

Yes- this does help a little bit (note that the create_metadata_file utility uses a tree reduction).  However, for very large-scale writes, the size of the metadata itself can be a real problem.

Looking here it looks like we drop everything into a single task
https://github.com/dask/dask/blob/ad6f837ea11f03a32c7bb51d6e33516d93e85148/dask/dataframe/io/parquet/core.py#L701-L714
I do see the slick tree reduction code in create_metadata_file but that doesn't appear to be active in to_parquet?  It looks like it's used in arrow when reading, if I understand correctly.
I've cleaned up the graph structure a bit in the no write_parquet case in https://github.com/dask/dask/pull/7968 .  Something there isn't right though, if you can take a look (or indeed take over that PR) @rjzamora I would appreciate it (no expectations though)

I do see the slick tree reduction code in create_metadata_file but that doesn't appear to be active in to_parquet? It looks like it's used in arrow when reading, if I understand correctly.

Right - That code was a bit "experimental" at first, but the approach should likely be used more broadly.

Something there isn't right though, if you can take a look (or indeed take over that PR)

Cool - I'll take a look, but I probably can't "do" much in the short term

Cool - I'll take a look, but I probably can't "do" much in the short term

In theory everything is done.  However tests aren't passing, so I probably missed some key somewhere.  I suspect that you are more capable of interpretting failed tests here than I am

While @rjzamora and I chat here, it's worth pointing out that anyone who runs into this problem can probably avoid most of the issue by setting write_metadata_file=False in the to_parquet call.
df.to_parquet(write_metadata_file=False)

2. Is the actual resulting metadata object huge, or is it just having a single task that has 1000's of dependencies? ...

... for very large-scale writes, the size of the metadata itself can be a real problem.

Indeed the actual metadata can get relatively large, if you have lots of row groups / columns. Small illustration:
In [1]: table = pa.table(pd.DataFrame(np.random.randn(10000, 10), columns=list("abcdefghij")))

In [2]: import pyarrow.parquet as pq

In [3]: pq.write_table(table, "test_md_size.parquet")

In [4]: !ls test_md_size.parquet -sh
960K test_md_size.parquet

In [5]: col = []

In [6]: pq.write_table(table, "test_md_size.parquet", row_group_size=10, metadata_collector=col)

In [7]: !ls test_md_size.parquet -sh
3,6M test_md_size.parquet

In [11]: col[-1].write_metadata_file("metadata_only.parquet")

In [12]: !ls metadata_only.parquet -sh
980K metadata_only.parquet

So for 1000 row groups with 10 columns, you already get a 1 MB file, and the in-memory object will probably be several times bigger (and we need it in-memory to append them, as written Parquet files can't be appended).
The size can be limited a bit by not writing statistics (write_statistics=False), but that of course also defeats part of the benefit of having a _metadata file.
(it might be interesting to explore other "metadata" file formats that are more flexible (eg appendable), like Apache Iceberg)


getattr- getattr : signature does not have a field dayOfWeek

Dear authors,
I found OpenDF is trying to repeatedly call  getattr while the signature validation failed:
Dialog 54
GUID: 096db3b5-b0cf-4268-8841-04936f86ac2b OpenDF:

U1: Did I host a graduation party last year?
A1: Yes, I found one event matching "graduation party". It was "Graduation Party" on May 19, 2019 from 5:00 to 5:30 PM.
U2: What did I do after the graduation party last year?
A2: I found one event on May 19, 2019 after 5:30 PM. It was "Graduation After-Party" from 7:00 to 7:30 PM.
U3: What day was the graduation party event last year?
A3: On May 19, 2019, it was Sunday.

Program:
 :dayOfWeek(
    :date(
       :start(
          FindEvents(
             AND(
                at_time(
                   FullYearofYear(
                      LastYear(
                         ) ) ) ,
                has_subject(
                   graduation party ) ) ) ) ) )

Graph:

The cause was that dayOfWeek is not an input of Date. fixed in commit e7cc2dbd

Fix confirmed, thank you.


Update abort-controller import

Motivation
See https://github.com/mysticatea/abort-controller/issues/32
Due to a missing export statement in the abort-controller package, using the named export AbortController will break when run in the browser.
Description
Update all named import AbortController with the default import, named the same.
This goes against one of our linter rules import/no-named-as-default, so every import is accompanied with an eslint-disable statement.
We should discuss whether or not this is the best course of action, considering the underlying issue may come back when we use AbortController in the future (and use the named export).

should we rather fork and use our fork until they accept and release our fix?

I just released @chainsafe/abort-controller v3.0.1 with the fix
Closing this in favor of using that for now


Add source for commit message guidelines

Add source for a good commit message guidelines.
Checklist:

[ ] I have read freeCodeCamp's contribution guidelines.
[ ] My pull request has a descriptive title (not a vague title like Update index.md)
[ ] My pull request targets the master branch of freeCodeCamp.
[ ] All the files I changed are in the same world language, for example: only English changes, or only Chinese changes, etc.

Closes #XXXXX

Thank you for opening this pull request.
This is a standard message notifying you that we’ve reviewed your pull request and have decided not to merge it. We would welcome future pull requests from you.
Thank you and happy coding.


DINT-485: release processes documentation

I think we should keep the descriptions of the other packages in the README. The original intention for the other packages were to remove any magento references in the code. The older packages were merged later into this repo to keep up with backwards compatibility.
The original plan was that the commerce-events packages should eventually be used in the internal EPC extension. I think when they move the extension from the Magento namespace to the Adobe namespace, we should do the same with the packages being used in the extension.
We will need to make sure all functionality is the same though, so extensive unit testing should be implemented to make sure all events are being covered correctly.


Support top of nested block for SMARTBLOCK command

Currently always inserts to the bottom of the nested block/page. Should possibly implement index

From Matt Brockwell on Slack

I think we have enough sibling and other button params to cover this


Added a  chromeUseTransform flag to config object.

This pull request add a new chromeUseTransform to the config object.
This flag is used only when the user agent is chrome, to switch from using css zoom attribute to transform css attribute.
This flags default to false, so it do not change previous behaviors.
I personally need this, because when using the css zoom approach, zoomed out content get squished after a certain level ( i mean this kind of artifact: https://cdn.css-tricks.com/wp-content/uploads/2013/01/squeeze-content.gif )
When using transform (with this chromeUseTransform flag set to true)  i get a nice zoom effect where content is zoomed uniformly not matter what the zoom level is.

Thanks. Looks nice :)
I'll merge it down, and I'll make a minor version version bump ASAP.
Cheers,
/Martin

You are welcome :) !

Thanks for the version bump !


Update README.md with new installation flow

Redo #5271 to pull in right email with CLA signed

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Hi Stanley,
You probably know this, but it might be good to be aware of this test result:
https://github.com/grpc/grpc/issues/5280#issuecomment-185204848
Paul


Shutdown droplets by default. Support poweroff with vagrant halt -f.

This patch uses the 'shutdown' action rather than poweroff when using vagrant halt, as i think would be expected by the user.
This also fixes an inconsistency where the "reboot" action performs a graceful restart, while "halt" powers off.

Looks great! For reference this aligns with http://docs.vagrantup.com/v2/cli/halt.html and http://docs.vagrantup.com/v2/cli/reload.html


improve masking function memory/speed

Currently we use Stream.cycle/1 and Enum.reduce_while/3 for the implementation of the masking function.
This is probably very suboptimal in terms of speed and memory consumption. If I understand Elixir/Erlang performance correctly, binary matches are always to be preferred for speed/memory. Also we're only doing the xor of 1 byte (unsigned big integer) at a time and we could be doing it with up to four bytes (the size of the mask)

after giving this a glance http://erlang.org/doc/efficiency_guide/myths.html#myth--tail-recursive-functions-are-much-faster-----than-recursive-functions
I tried it with a body-recursive implementation and wow that's slow
Operating System: Linux
CPU Information: Intel(R) Core(TM) i7-9700KF CPU @ 3.60GHz
Number of Available Cores: 8
Available memory: 31.30 GB
Elixir 1.12.1
Erlang 24.0

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 2 s
parallel: 1
inputs: Large, Medium, Small
Estimated total run time: 2.80 min

Benchmarking body with input Large...
Benchmarking body with input Medium...
Benchmarking body with input Small...
Benchmarking compile with input Large...
Benchmarking compile with input Medium...
Benchmarking compile with input Small...
Benchmarking match with input Large...
Benchmarking match with input Medium...
Benchmarking match with input Small...
Benchmarking stream with input Large...
Benchmarking stream with input Medium...
Benchmarking stream with input Small...

##### With input Large #####
Name              ips        average  deviation         median         99th %
compile         99.81       10.02 ms     ±3.17%       10.03 ms       10.72 ms
match           73.71       13.57 ms     ±3.24%       13.54 ms       14.65 ms
stream          13.54       73.86 ms     ±4.06%       74.25 ms       78.82 ms
body           0.0152    65666.52 ms     ±0.00%    65666.52 ms    65666.52 ms

Comparison:
compile         99.81
match           73.71 - 1.35x slower +3.55 ms
stream          13.54 - 7.37x slower +63.84 ms
body           0.0152 - 6554.43x slower +65656.50 ms

Memory usage statistics:

Name       Memory usage
compile        23.84 MB
match          38.15 MB - 1.60x memory usage +14.31 MB
stream        206.00 MB - 8.64x memory usage +182.15 MB
body           34.62 MB - 1.45x memory usage +10.78 MB

**All measurements for memory usage were the same**

##### With input Medium #####
Name              ips        average  deviation         median         99th %
compile        5.34 K      187.26 μs    ±33.76%      191.52 μs      338.91 μs
match          3.77 K      265.34 μs    ±25.08%      258.18 μs      409.38 μs
body           1.11 K      896.89 μs     ±4.81%      882.93 μs     1026.21 μs
stream         0.83 K     1205.38 μs    ±28.13%     1196.79 μs     1932.71 μs

Comparison:
compile        5.34 K
match          3.77 K - 1.42x slower +78.08 μs
body           1.11 K - 4.79x slower +709.63 μs
stream         0.83 K - 6.44x slower +1018.12 μs

Memory usage statistics:

Name       Memory usage
compile       245.31 KB
match         391.83 KB - 1.60x memory usage +146.52 KB
body          411.38 KB - 1.68x memory usage +166.06 KB
stream       2111.56 KB - 8.61x memory usage +1866.25 KB

**All measurements for memory usage were the same**

##### With input Small #####
Name              ips        average  deviation         median         99th %
compile      196.45 K        5.09 μs  ±2237.32%        1.57 μs        4.11 μs
match        155.04 K        6.45 μs  ±1752.01%        1.95 μs        5.37 μs
body         125.88 K        7.94 μs  ±1503.12%        2.65 μs        7.25 μs
stream        46.30 K       21.60 μs   ±513.21%        7.01 μs      433.76 μs

Comparison:
compile      196.45 K
match        155.04 K - 1.27x slower +1.36 μs
body         125.88 K - 1.56x slower +2.85 μs
stream        46.30 K - 4.24x slower +16.51 μs

Memory usage statistics:

Name       Memory usage
compile         3.61 KB
match           5.11 KB - 1.42x memory usage +1.50 KB
body            5.32 KB - 1.47x memory usage +1.71 KB
stream         23.28 KB - 6.45x memory usage +19.67 KB

**All measurements for memory usage were the same**

bench file:
Mix.install([:benchee])

defmodule Foo do
  def stream_mask(payload, <<a, b, c, d>>) do
    [a, b, c, d]
    |> Stream.cycle()
    |> Enum.reduce_while({payload, _acc = <<>>}, fn
      _mask_key, {<<>>, acc} ->
        {:halt, acc}

      mask_key, {<<part_key::integer, payload_rest::binary>>, acc} ->
        {:cont, {payload_rest, <<acc::binary, Bitwise.bxor(mask_key, part_key)::integer>>}}
    end)
  end

  def match_mask(payload, mask, acc \\ <<>>)

  def match_mask(payload, nil, _acc), do: payload

  # n=4 is the happy path
  # n=3..1 catches cases where the remaining byte_size/1 of the payload is shorter
  # than the mask
  for n <- 4..1 do
    def match_mask(
          <<part_key::integer-size(8)-unit(unquote(n)), payload_rest::binary>>,
          <<mask_key::integer-size(8)-unit(unquote(n)), _::binary>> = mask,
          acc
        ) do
      match_mask(
        payload_rest,
        mask,
        <<acc::binary, Bitwise.bxor(mask_key, part_key)::integer-size(8)-unit(unquote(n))>>
      )
    end
  end

  def match_mask(<<>>, _mask, acc), do: acc

  @compile {:inline, compile_mask: 2, compile_mask: 3}
  def compile_mask(payload, mask, acc \\ <<>>)

  def compile_mask(payload, nil, _acc), do: payload

  # n=4 is the happy path
  # n=3..1 catches cases where the remaining byte_size/1 of the payload is shorter
  # than the mask
  for n <- 4..1 do
    def compile_mask(
          <<part_key::integer-size(8)-unit(unquote(n)), payload_rest::binary>>,
          <<mask_key::integer-size(8)-unit(unquote(n)), _::binary>> = mask,
          acc
        ) do
      compile_mask(
        payload_rest,
        mask,
        <<acc::binary, Bitwise.bxor(mask_key, part_key)::integer-size(8)-unit(unquote(n))>>
      )
    end
  end

  def compile_mask(<<>>, _mask, acc), do: acc

  def body_mask(payload, mask)

  def body_mask(payload, nil), do: payload

  # n=4 is the happy path
  # n=3..1 catches cases where the remaining byte_size/1 of the payload is shorter
  # than the mask
  for n <- 4..1 do
    def body_mask(
          <<part_key::integer-size(8)-unit(unquote(n)), payload_rest::binary>>,
          <<mask_key::integer-size(8)-unit(unquote(n)), _::binary>> = mask
        ) do
      <<
        Bitwise.bxor(mask_key, part_key)::integer-size(8)-unit(unquote(n)),
        body_mask(payload_rest, mask)::binary
      >>
    end
  end

  def body_mask(<<>>, _mask), do: <<>>
end

mask = :crypto.strong_rand_bytes(4)

Benchee.run(
  %{
    "stream" => fn payload -> Foo.stream_mask(payload, mask) end,
    "match" => fn payload -> Foo.match_mask(payload, mask) end,
    "body" => fn payload -> Foo.body_mask(payload, mask) end,
    "compile" => fn payload -> Foo.compile_mask(payload, mask) end
  },
  time: 10,
  memory_time: 2,
  inputs: %{
    "Small" => :crypto.strong_rand_bytes(100),
    "Medium" => :crypto.strong_rand_bytes(10_000),
    "Large" => :crypto.strong_rand_bytes(1_000_000)
  }
)


Buffer Overrun When Ending Command With ""

Description
When ending any command with a backslash \ while using Lamp as my command framework, an ArgumentParseException is thrown with a message indicating a buffer overrun while parsing arguments.
Environment

Server: Paper 1.20.4
Lamp Version: 3.1.8

Steps to Reproduce

Execute a command that ends with a backslash \.
Observe the error in the server console.

Expected Behavior
Commands ending with a backslash should be processed without causing any errors.
Actual Behavior
An error is thrown in the console with the following stack trace: https://gist.github.com/DoudGeorges/a599c49d3247d0e140ed5a6e1cde2c79

Good catch, this bug has already been reported, but I forgot to patch it. Thanks for reporting.


Multiple Page Submissions on Match Up Page

what if a child submits multiple pages?
It would be ideal to only show page1 in the modal on the match up page and when a user clicks on the modal you can carousel / see all pages submitted.
The modals on the match up page may need to accept an array that allows us to map through the entire list of submissions and display accordingly in the modal.

-Fixed. Modals now have conditional rendering based on page # of stories.


ctr: failed to prepare extraction snapshot xxx failed to create sandbox layer xxx

Description

Steps to reproduce the issue:

download from  https://github.com/containerd/containerd/releases/download/v1.4.4/cri-containerd-cni-1.4.4-windows-amd64.tar.gz
extract it
run containerd
run

PS G:\Downloads\containerd> .\ctr.exe image pull docker.io/library/hello-world:latest
docker.io/library/hello-world:latest:                                             resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:308866a43596e83578c7dfa15e27a73011bdd402185a84c5cd7f32a88b501a24:    exists         |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:b0408a0f1d74eced127a2658429f336ed8fa48e36d59878f155b19865e5dbd70: exists         |++++++++++++++++++++++++++++++++++++++|
config-sha256:545e4e38d9db3fd6d543f0b4498a83e91f17166ed9280830a84f984cf22a0149:   exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:a11c4ca8f1ba16b6e1395d4523be2215bd0871c2e3e4695b9f2b87bd7472be52:    exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:d76481af78e2b81f9ea02fc66c1950cf204cbb93cd49fd1751695d62ec81d9f9:    exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:b70d36b981c5bce88c8132b2a7248b435709cf03477d08e8e7f3752cc9f0eaa6:    exists         |++++++++++++++++++++++++++++++++++++++|
elapsed: 10.1s                                                                    total:   0.0 B (0.0 B/s)
unpacking windows/amd64 sha256:308866a43596e83578c7dfa15e27a73011bdd402185a84c5cd7f32a88b501a24...
ctr: failed to prepare extraction snapshot "extract-525933300-z-ZK sha256:fb9c5a79707ee6af09795e104e283faa4bb833ee5f4d6a6d83abfec6f85f7df4": failed to create sandbox layer: hcsshim::CreateScratchLayer - failed failed in Win32: The specified module could not be found. (0x7e): unknown

Describe the results you received:
Describe the results you expected:
What version of containerd are you using:
$ containerd.exe --version
containerd github.com/containerd/containerd v1.4.4 05f951a3781f4f2c1911b05e61c160e9c30eaa8e

Any other relevant information (runC version, CRI configuration, OS/Kernel version, etc.):
WINDOWS 10 Pro

runc --version
$ runc --version

crictl info
$ crictl info

uname -a
$ uname -a

I think to run windows containers on windows you will  need the right version of hcshim installed @kevpar

I have a similar problem

@cnhuye Is the containers feature installed on your machine?

@cnhuye hi, i got the same problem, have you resolved this problem?


missing M5 from using latest pbmm2 Docker code

I used pbmm2 to map BAM to the assembled genome and polish by SMRT6.0 polishing. This is https://quay.io/repository/biocontainers/pbmm2?tab=tags 0.12.0--ha87ae23_0 which is the latest and less than 1 month. However, the M5 field is missing from the minimap output, so I can't use them for polishing. Do you know how to fix this? Are you strictly update only the Conda one?
The bioconda page https://bioconda.github.io/recipes/pbmm2/README.html mentioned that we can docker pull from Quay. I think it would be good if Pacbio also supports the docker as it is increasingly popular in the bioinformatics community.

You can't mix and match latest software from conda with official releases. Please stick to one release channel. If you use GenomicConsensus with pbmm2 from conda, polishing will succeed.
What "support" would you expect for docker? Most of our tools are on bioconda, what you make out of it, is up to you.

@armintoepfer We have some PYTHONPATH issue in our setup, so probably have to go with Bioconda route. I'm very surprised though that the official release and Conda are not compatible. Would Pacbio plan to maintain these two parallel version?

because keeping them aligned would require tons more resources. Bioconda has no service-level agreements, and we provide this service to the community, without any concrete promises.

Those are two different worlds. One is cut fresh from development HEAD, the other goes through official vetting and consequently can lag behind quite a bit. Cherry picking works sometimes, but no promise, because interfaces will break and if, we make sure that software from the same source works together.
The choice is yours. You pick the blue pill, you get stable releases with official tech support. You pick the red pill, you stay in conda land, get the latest and greatest versions, but might trigger a few more bugs here and there.
That's all I have to say about this topic.


docs and API refinement

I'm currently diving into IAPs and was wondering about some topics I explain in the following. My config is:

react-native-iap: 3.3.9
react-native: 0.60.4
platform: Android

typeof Product

[ ] The docs say that each field is of type string. Contrary to that I get type Number for price and originalPrice
[ ] Contrary to the docs I get a number for price and not localizedPrice
[ ] For title I get a concatenated string of a) the title I entered in the Play Console item setup and b) the title of the app itself. I expected to get only a).
[ ] introductoryPrice includes the currency chars and thus maybe should be renamed to introductoryLocalizedPrice
[ ] maybe unify naming for: there is getSubscriptions and requestSubscription for subscriptions but getProducts and requestPurchase

deprecated

[ ] update deprecated function names in the docs by their successors p.ex.:

Once you have called getProducts(), and you have a valid response, you can call buyProduct()

Finishing a Purchase

[ ] The docs say that I have to call acknowledgePurchaseAndroid or (should call) finishTransactionIOS. I think it would be a good idea for a future release to provide a single  acknowledgePurchase to unify both platforms.

Q&A

[ ] First question mentions fetchProducts. Is the hole answer for version 2 or should get Products be used instead?

@voxspox Looks like you have a great insight on improving react-native-iap. Would you like to give us a PR for some of those you've listed?
I think I can design the Finishing a Purchase section and answer to Q&A.
 First question mentions fetchProducts. Is the hole answer for version 2 or should get Products be used instead?

The answer is currently, they do the same thing but I'd liked to ensure that it won't work as a promises.

maybe this issue should be split into separate issues

any updates on this?


[Pending Review]Update how-to-reprint-the-reports-created-during-checkbook-reconcilia…

…tion.md
Pull request guidance
Thank you for submitting your contribution to our support content! Our team works closely with subject matter experts in CSS and PMs in the product group to review all content requests to ensure technical accuracy and the best customer experience. This process can sometimes take one or more days, so we greatly appreciate your patience.
We also need your help in order to process your request as soon as possible:

We won't act on your pull request (PR) until you type "#sign-off" in a new comment in your pull request (PR) to indicate that your changes are complete.

After you sign off in your PR, the article will be tech reviewed by the PM or SME if it has more than minor changes. Once the article is approved, it will undergo a final editing pass before being merged.

sounds great


Migration to Godot 4.0

I successfully migrated this plugin to Godot 4.0. I'm posting the changes to the two files radial_progress.gd and RadialProgressBar.gd below, in diff format.
If you're not familiar with diff format: The lines in the Godot 3 files are prefixed with <, and the lines in the Godot 4 files are prefixed with >.
diff -r radial_progress.gd.3 radial_progress.gd.4
1c1
< tool
---
> @tool

diff -r RadialProgressBar.gd.3 RadialProgressBar.gd.4
1c1
< tool
---
> @tool
5,12c5,11
< export var max_value = 100;
< export var radius = 120;
< export var progress = 0;
< export var thickness = 20 ;
< export var bg_color : Color  = Color(0.5,.5,0.5,1);
< export var bar_color :Color =  Color(0.2,.9,0.2,1);
< var angle = 0.0 ;
< var tween = Tween.new();
---
> @export var max_value = 100;
> @export var radius = 120;
> @export var progress = 0;
> @export var thickness = 20 ;
> @export var bg_color : Color  = Color(0.5,.5,0.5,1);
> @export var bar_color :Color =  Color(0.2,.9,0.2,1);
> var angle = 0.0 ;
14d12
< 	add_child(tween);
23c21
< 	update();
---
> 	queue_redraw();
28c26
< 	update();
---
> 	queue_redraw();
33c31
< 	update();
---
> 	queue_redraw();
43c41
< 	 update();
---
> 	queue_redraw();
45a44
> 	var tween: Tween = create_tween()
54c53
< 	var points_arc = PoolVector2Array()
---
> 	var points_arc = PackedVector2Array()
56c55
< 	var colors = PoolColorArray([color])
---
> 	var colors = PackedColorArray([color])
58,59c57,58
< 	    var angle_point = deg2rad(angle_from + i * (angle_to - angle_from) / nb_points - 90)
< 	    points_arc.push_back(center + Vector2(cos(angle_point), sin(angle_point)) * radius)
---
> 		var angle_point = deg_to_rad(angle_from + i * (angle_to - angle_from) / nb_points - 90)
> 		points_arc.push_back(center + Vector2(cos(angle_point), sin(angle_point)) * radius)

With the changes noted above, this plugin works in Godot 4.2-dev5.

I submitted the Godot 4 version of this plugin to the Asset Library: https://godotengine.org/asset-library/asset/2193


Removing dependency and use of parametric programs

Right now both VQE and QAOA return and use parametric program objects from pyquil.  These objects are used as they are intended to be efficient templates for variational quantum programs.
That said, the parametric.py library is still very experimental.  This means that it can be pretty unintuitive to use and still has several bugs and strange edge cases.
Due to this, we should remove the dependency of pyqve and pyqaoa on these objects until they are more stable.  In the meantime they can be replaced with basic python functions that return pyQuil programs.
@jotterbach Is going to start taking a look at this.

@willzeng
I looked through the code and I think the problem has an upside and a downside.

On the upside: VQE only has functions of the form f: list -> PyQuil.function which is exactly what we want. Hence there is no direct dependency on pyquil.ParametricFunction

On the other hand pyqaoa.qaoa has a dependency on various parts od pyquil.Paulis, which hence indirectly introduces a dependency to ParametricFunctions through Paulis.exponential_map that returns a decorated program.

As far as I can tell from a first pass Paulis.exponential_map is the only point where we induce ParametricFunction into grove. There is no direct import of ParametricFunction in the grove library (except some ipynb) and the name of parametric_state_evolve is misleading. In this light it might be worthwhile to try and refactor Paulis to purge the dependency. What's our thought?
cc @ncrubin as he wrote most of qaoa.py according to the git annotate.

Yep that's correct @jotterbach.  As part of this issue exponential_map should be changed to return a python function rather than a parametric program. I also agree that changed should be made to make function behavior clear if their output type has changed

this was fixed by #22


Win32Exception when path to vlc-dlls contains special characters

We encountered a Win32Exception when loading the libvlc dlls (at Vlc.DotNet.Core.Interops.VlcInteropsManager..ctor) in case the path to the dlls contains any special character like &-äöüâ^°汉语.
To fix this issue, we added the CharSet = CharSet.Unicode property to the [DllImport] Attribute of the LoadLibraryFunction (at Vlc.DotNet.Core.Interops.Win32Interops). Now it looks like this:
[DllImport("kernel32", SetLastError = true, CharSet = CharSet.Unicode)]
public static extern IntPtr LoadLibrary(string lpFileName);

I think its a good idea in general to add this property to the DllImport. At least in case the arguments could contain special characters. What do you think?
Thanks

Charset Will set to Ansi

Closing this for now, feel free to reopen if this is still an issue.


use youtube links with start times

I went through the videos that had start times as text and got them mapped to the link itself. This will help anyone that is clicking in the link and wants to see the actual section (instead of searching for the start time)

awesome!! thank you for this fix!


[RESOLVED] Interface eth0(): High error rate (>2 for 5m)

Problem started at 09:21:29 on 2023.04.10
Problem name: Interface eth0(): High error rate (>2 for 5m)
Host: S100-Primario
Severity: Warning
Operational data: errors in: 3, errors out: 0
Original problem ID: 6802
Event details in Zabbix: http://<IP_ADDRESS>/tr_events.php?triggerid=23483&eventid=6802

Problem has been resolved in 6m 0s at 09:27:29 on 2023.04.10
Problem name: Interface eth0(): High error rate (>2 for 5m)
Host: S100-Primario
Severity: Warning
Original problem ID: 6802


bedtools: 2.29.2 -> 2.30.0

Motivation for this change
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

This is a semi-automatic executed nixpkgs-review which is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 110735 run on x86_64-darwin 1

1 package built:

bedtools

This is a semi-automatic executed nixpkgs-review which is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 110735 run on x86_64-darwin 1

1 package built:

bedtools


Show meaningful error in case of doubled schema

Earlier it returned information about id being defined twice, which could be confusing for the users as in most cases this field wasn't defined anywhere. Now it will raise RuntimeError with meaningful error that contain line with first schema.

:green_heart: :blue_heart: :purple_heart: :yellow_heart: :heart:

It probably should be expanded to the embedded_schema as well, but I do not use them, so I have forgot about it.


Basic auth properties for Gradle wrapper aren't applied

Steps to Reproduce

Create a new project with flutter create my_app
Set a custom distributionUrl in /android/gradle/wrapper/gradle-wrapper.properties that requires basic authentication.
Define the credentials in /android/gradle.properties as follows:

systemProp.gradle.wrapperUser=your-username
systemProp.gradle.wrapperPassword=your-password

Run a command with ./android/gradlew

Expected results: The Gradle wrapper downloads the defined Gradle version and executes the command.
Actual results: The server throws a 401 HTTP exception.
Workaround: Swap /android/gradle/wrapper/gradle-wrapper.jar with the generated gradle-wrapper.jar from a (new) native Android project. The Gradle wrapper now downloads the defined Gradle version by correctly applying the basic auth credentials.
flutter doctor -v output:
[✓] Flutter (Channel stable, 2.10.2, on macOS 12.2.1 21D62 darwin-x64, locale
    en-GB)
    • Flutter version 2.10.2 at /Users/frederik.dumazy/fvm/versions/stable
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 097d3313d8 (11 days ago), 2022-02-18 19:33:08 -0600
    • Engine revision a83ed0e5e3
    • Dart version 2.16.1
    • DevTools version 2.9.2

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Users/frederik.dumazy/Library/Android/sdk
    • Platform android-31, build-tools 30.0.3
    • ANDROID_HOME = /Users/frederik.dumazy/Library/Android/sdk
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.11+0-b60-7590822)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.2.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.10.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio
    • Android Studio at /Applications/Android Studio Preview.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6842174)

[✓] Android Studio (version 2021.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.11+0-b60-7590822)

[✓] IntelliJ IDEA Community Edition (version 2021.1.1)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.64.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.34.0

[✓] Connected device (2 available)
    • macOS (desktop) • macos  • darwin-x64     • macOS 12.2.1 21D62 darwin-x64
    • Chrome (web)    • chrome • web-javascript • Google Chrome 98.0.4758.109

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

I could not specify the exact url and credentials since this is sensitive information, but the distributionUrl points to an Artifactory instance.
Defining the basic auth within the distributionUrl directly like<EMAIL_ADDRESS>also didn't work, as well as defining the wrapperUser|wrapperPassword properties in $USER/.gradle/gradle.properties.
I would assume that the gradle-wrapper.jar that is created with flutter create is outdated.
Swapping it out with one from a new native Android project seems to fix this issue and allows to use basic auth for the distributionUrl

cc @blasten

Yes, the wrapper needs to be updated.
We have a script to do this: https://github.com/flutter/flutter/blob/master/dev/tools/repackage_gradle_wrapper.sh

cc @GaryQian


Convert HTML to Gomponents

This is my attempt to solve #167. I didn't use x/net/html because... well I forgot to look into it. The XML parser in the standard library seems to work fine though once configured to parse HTML.
The way I decided to format the output was as such:

Attributes are on the same line
g.Text and other elements take up a new line
Whitespace is ignored
Tabs are used instead of spaces (as per Go's standard)
It's standard to have one "page" per file, so this code outputs an entire new file with the imports and the package (default "ui") rather than outputting a single function

Here's an example:
<!DOCTYPE html>

	<meta charset="UTF-8">
	<meta name="viewport" content="width=device-width, initial-scale=1.0">
	<title>Document</title>

How are you doing?

	<unknown required data-crazy="lol">wow</unknown>

	<form action="/test" method="get">
		<input type="text" required>
	</form>

package ui

import (
	g "github.com/maragudk/gomponents"
	. "github.com/maragudk/gomponents/html"
)

func Page() g.Node {
	return Doctype(
		HTML(Lang("en"),
			Head(
				Meta(Charset("UTF-8")),
				Meta(Name("viewport"), Content("width=device-width, initial-scale=1.0")),
				TitleEl(
					g.Text("Document"),
				),
			),
			Body(
				P(
					g.Text("How are you doing?"),
				),
				g.El("unknown", Required(), Data("crazy", "lol"),
					g.Text("wow"),
				),
				A(Aria("hidden", "true"), Data("wow", "some field"), g.Attr("unrecognized", "crazy"), g.Attr("anotherunrecognized", "anotherunrecognized")),
				Form(Action("/test"), Method("get"),
					Input(Type("text"), Required()),
				),
			),
		),
	)
}

(Note: the output contains trailing commas, not seen here, but these are easily removed by running go fmt, which is integrated into most editors)
Usage: go run . (-p PACKAGE NAME) [html file]
There is still lots of work left, such as integrating SVG and HTMX, running automated tests, supporting globbing, but I think this lays a good foundation.
Apologize for the older commits as well. I had an older fork of this repo and I didn't know how to squash them excerpt by deleting my fork and starting a new one, and at that point I had already started working on it.
Feel free to move this off of main.

Tests are failing because I accidentally used go 1.20 features :skull: (strings.CutPrefix)

Hey @amrojjeh, thanks for having a go at this! I'll see if I can get around to having a look at it before the start of my summer holidays at the end of this week. 😊

Hi @amrojjeh . Thanks again for having a go at this!
Since then, @PiotrKowalski made an online version at https://htg.piotrkowalski.me, with source available at https://github.com/PiotrKowalski/html-to-gomponents . I think I'm going with pointing to that for now, since it's the least amount of work for me and looks like it's working fine. If I'm adding anything to the core library, I'd have to make sure to understand it and be responsible for maintaining it, and unfortunately I don't want to take that on right now.
But perhaps it would make sense for you to publish this as an independent CLI tool?


Ditch kiterc overrides in favor of environment variables

Resolves #31.
Couldn't test with docker yet. Copying stage under docker and doing a simple docker build succeeded, but then run failed with:
./lynxkite-sphynx: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by ./libnetworkit.so).
I suspect this is not due to this PR. ;)

This works for me! This change is live on PizzaBox now.


Encrypting the backup dump of postgresql db in MinIO

I close this to keep track of it in one place (#903)


Python 3.11?

It looks like naplam-panos has an issue running within Python3.11. Is there going to be a update to get it supported for 3.11?

Yea, will take a look


Organize helper functions / utilities

Move some reused R code into hewr and depend on it in pipeline scripts
Clean up some python pipelining code and restore a needed helper function for parsing directories.

Flagging that this will impact  https://github.com/CDCgov/pyrenew-hew/pull/136


Current project status

Hi, what is the current status of this project? Is it still actively maintained?

Good to hear. Am I correct though that beginning with Node 10 and stream.pipeline pump is not longer required ? If so i think it would be appreciated if the README commented on to why or why not ?

@matthiasg you are correct - the following quote from the Node docs confirms it:

pump is only necessary for Nodejs 8.x or earlier, as for Node 10.x or later version, pipeline is introduced to replace for pump. This is a module method to pipe between streams forwarding errors and properly cleaning up and provide a callback when the pipeline is complete.

Yup, in Node.js 10 pipeline was added (by me :)) to have pump in core. I still use pump personally whenever I want to support Node.js <10

@mafintosh @lukesims sorry had actually read it in the node docs, but had forgotten about it. It would be sensible to mention in this projects README as well though, to close the loop for people coming from other projects (like I did) trying to figure out whether its required. good work btw

I agree with @matthiasg as it would save future users from making the same side-trip to the Node docs that we all did to reach this conclusion. This is the PR if anyone else is interested - good work @mafintosh :)


rest: introduce transaction resolve and simulation endpoints

Description
This stack of commits introduces a transaction resolve and simulation endpoint to the REST api service.
More documentation, usage examples as well as a stabilized format for an unresolved transaction JSON payload still need to be done/decided.
At a high level the resolve api will allow someone to provide a partially filled in PTB and have the Fullnode:

select versions
mutability of the objects
budget determination
fill in reference gas price
basic gas selection/gas smashing

Test plan
How did you test the new or updated feature?

Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

[ ] Protocol:
[ ] Nodes (Validators and Full nodes):
[ ] Indexer:
[ ] JSON-RPC:
[ ] GraphQL:
[ ] CLI:
[ ] Rust SDK:
[ ] REST API:

Ok I've updated the PR to account for the provided feedback. There are still a few outstanding things that I think, and would prefer, to do in a follow up PR specifically:

Updating the UnresolvedTransaction format to provide less information about inputs (in addition to general tweaks to the JSON format)
handle basic type inference and BCS serialization of pure args

The REST service now has the concept of "stable" and "unstable" apis and this one will remain "unstable" till those TODOs are closed out

Also to answer this:

Do we need both resolve and simulate at the REST level? It seems like simulate is offering a strict subset of the features that resolve offers today.

Yes I think it makes sense to provide both, resolve only can really accept a json UnresolvedTransaction format (given everything we want to do with it) while simulate provides a way to do a simulation of a fully resolved transaction provided via bcs.


Rethink pagination in search results

Pagination no longer works for me

We should probably simply limit the number of search results and show on a single page. You then need to search to refine what you are looking for. We could add search results as you scroll down or simply show a next button that shows the next results.
Alternative would be to abbreviate with an ellipsis...

Fixed by #47


Registration with existing email address raises Integrity exception

Hi. Thanks for the great app!
It appears that the registration endpoint does not check for existing email address. The following exception is raised:
django.db.utils.IntegrityError: duplicate key value violates unique constraint "user_user_email_key"
DETAIL:  Key<EMAIL_ADDRESS>already exists.

This can be fixed via altering the create endpoint like this.
        try:
            user = self.perform_create(serializer)
        except django.db.utils.IntegrityError:
            return Response(status=status.HTTP_400_BAD_REQUEST)

Am I missing something or this is really an issue?

I also encountered this error and it seems it's because in the registration serializer, the validate_email function checks whether the email exists AND is verified, however it doesn't handle the issue where the email exists but is not verified.
You can override the serializer:
from allauth.account.adapter import get_adapter
from allauth.account.models import EmailAddress
from dj_rest_auth.registration.serializers import RegisterSerializer
from django.conf import settings
from rest_framework import serializers

class UserRegistrationSerializer(RegisterSerializer):
    def validate_email(self, email):
        email = get_adapter().clean_email(email)
        if settings.ACCOUNT_UNIQUE_EMAIL:
            if email and EmailAddress.objects.filter(email__iexact=email).exists():
                raise serializers.ValidationError(
                    'A user is already registered with this e-mail address.',
                )
        return email

and then add it in settings.py:
REST_AUTH = {
     ...
    'REGISTER_SERIALIZER': 'your_module_name.serializers.UserRegistrationSerializer'
}

it's because in the registration serializer, the validate_email function checks whether the email exists AND is verified, however it doesn't handle the issue where the email exists but is not verified.

That is exactly the case. Thank you for providing a quick fix.
I think we can call probably call it a bug as the case of the email existing but is not verified should be gracefully handled instead of code breaking.

This is a bug and the fix is okay. But to be compatible with django-allauth we should do something like:
    def clean_email(self):
        value = self.cleaned_data["email"].lower()
        value = get_adapter().clean_email(value)
        if value and app_settings.UNIQUE_EMAIL:
            value = self.validate_unique_email(value)
        return value

    def clean_email2(self):
        value = self.cleaned_data["email2"].lower()
        return value

    def validate_unique_email(self, value):
        adapter = get_adapter()
        assessment = flows.manage_email.assess_unique_email(value)
        if assessment is True:
            # All good.
            pass
        elif assessment is False:
            # Fail right away.
            raise adapter.validation_error("email_taken")
        else:
            assert assessment is None
            self.account_already_exists = True
        return adapter.validate_unique_email(value)

(taken from the BaseSignupForm from django-allauth)


How do APIs work?

IP API:
https://geo.ipify.org/docs
Due data I've to show the best call will be Type 2

Map API:
https://leafletjs.com/
First steps. Index.html ->
https://leafletjs.com/examples/quick-start/

Due leaflet is installed with npm is not necessary to import it at index.html


Feature additions

"Pending" or "Contingent" property searches
Last X months ago search on Sold properties

@Amir-SDSU


Add A2C implementation

Reminder todo after internship.
Mostly for meta-bandit and gridworld tasks

Suggestion: you could probably just implement it as PPO with fixed parameters (gae=1, no advantage normalization, 1 epoch, 1 minibatch) as per "A2C is a Special Case of PPO"

Good point, I didn't know about this equivalence. For the meta-RL setups I may have to write some extra logic but will try to keep things minimal.


Release Apache ORC 1.7.9

[x] GitHub Action CI Check (Java 8/11/17) and Ubuntu/MacOS/Windows.

https://github.com/apache/orc/actions/workflows/build_and_test.yml?query=branch%3Abranch-1.7

[ ] Docker Tests (CentOS7, Debian10/11, Ubuntu18/20/22, Fedora37)
[x] Apache Spark 3.3.3-SNAPSHOT Integration Test with 1.7.9-SNAPSHOT
[x] Apache Iceberg 1.3.0-SNAPSHOT Integration Test with 1.7.9-SNAPSHOT
[x] Vote Tag Creation: v1.7.9-rc1
[x] Upload RC1 Vote Artifacts: https://dist.apache.org/repos/dist/dev/orc/v1.7.9-rc1/
[x] Send RC1 Vote Email: https://lists.apache.org/thread/bk86fd8pm013b9q2qfj989jsvn5mmxc2
[x] Apache Spark 3.3.3-SNAPSHOT Integration Test with RC1
[x] Apache Iceberg 1.3.0-SNAPSHOT Integration Test with RC1
[ ] Publish to Apache Download: https://downloads.apache.org/orc/orc-1.7.9/
[ ] Publish to Maven Central: https://repo1.maven.org/maven2/org/apache/orc/orc-core/1.7.9/
[ ] Send RC1 Vote Result:
[ ] Release Tag Creation: https://github.com/apache/orc/releases/tag/v1.7.9
[ ] Create next milestone:
[ ] Register 1.7.9 to https://reporter.apache.org/addrelease.html?orc

Could you help participate in the vote please? Thanks in advance!
@dongjoon-hyun @williamhyun @guiyanakuang @pavibhai

Hi, @wgtmac . According to your Spark integration PR, https://github.com/wgtmac/spark/pull/1, it seems that you are not aware of the relationship of the release cycles of Apache Spark and ORC yet. We intentionally have been keeping that Apache ORC releases are aligned with Apache Spark releases.

Apache ORC 1.6.x <-> Apache Spark 3.2.x
Apache ORC 1.7.x <-> Apache Spark 3.3.x
Apache ORC 1.8.x <-> Apache Spark 3.4.x
Apache Spark 1.9.x <-> Apache Spark 3.5.x (TBD)

Please refer the previous @williamhyun 's Apache ORC 1.7.8 Release Issue. He used Apache Spark branch-3.3.

https://github.com/apache/orc/issues/1376
https://github.com/williamhyun/spark/pull/1

Hi @dongjoon-hyun, could you help me run the docker tests? I constantly get connection errors and cannot finish this work item. Thank you in advance!

Sorry but I cannot help you at this time due to the my personal environment migration as of today, @wgtmac .

Sorry but I cannot help you at this time due to the my personal environment migration as of today, @wgtmac .

Thank you for the reply. I think it is fine to skip the docker tests as this is really a minor release which does not change much from the previous release.
I have finished all other tasks except creating the next milestone. My question is whether we have the plan to release 1.7.10? And if yes, what about the due date?
Again, thank you for providing me with the detail worklog for the release process! @dongjoon-hyun

For the record, it is one of pre-tasks which release manager must do before cut a RC tag.

I think it is fine to skip the docker tests as this is really a minor release which does not change much from the previous
release.

Since it was the first release of yours, I understand it was missed mistakenly, but I'd cast -1 if the future release manager make a RC without testing it.

According to the speed and content of commits in branch-1.7, I don't expect much this year. Given that, could you create a milestone for 1.7.10 on November (6 month later)?

My question is whether we have the plan to release 1.7.10? And if yes, what about the due date?

I have created the milestone for 1.7.10 and thus close this issue. Thanks @dongjoon-hyun

Thank you!


GRPCRoute Support

Proposed changes
Problem: As a user of NGF with applications that require GRPC traffic
I want NGF to support GRPCRoute Exact Method Matching, Header Matching, and Listener Hostname Matching
So that I have a method to ingress GRPC traffic to my applications.
Solution: Add GRPCRoute support, reusing logic from HTTPRoute where possible and where it makes sense. The initial ticket was for only Hostname and Backend Ref matching, but it made sense to add the other support as so much of the validation and logic can be reused from HTTPRoute, instead of adding logic to reject those routes now.
Note: The only solution I have for turning on http2 is to hardcode it into the base NGINX conf files. I don't think there is a way to make this configurable. Do we foresee any potential issues with this? UPDATE: This will be addressed in a future PR by offering an option to disable http2
Testing: Ran conformance tests with GRPCRoutes locally (GRPCRoute conformance tests are in main branch only) , unit testing, manual testing
--- PASS: TestConformance/GRPCExactMethodMatching (1.02s)
    --- PASS: TestConformance/GRPCExactMethodMatching/2_request_to_'/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/EchoThree'_should_receive_a_Unimplemented_(12) (0.00s)
    --- PASS: TestConformance/GRPCExactMethodMatching/0_request_to_'/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_should_go_to_grpc-infra-backend-v1 (0.02s)
    --- PASS: TestConformance/GRPCExactMethodMatching/1_request_to_'/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/EchoTwo'_should_go_to_grpc-infra-backend-v2 (0.02s)
--- PASS: TestConformance/GRPCRouteHeaderMatching (0.01s)
    --- PASS: TestConformance/GRPCRouteHeaderMatching/6_request_to_'/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_with_headers_'{Color:blue}'_should_go_to_grpc-infra-backend-v1 (0.02s)
    --- PASS: TestConformance/GRPCRouteHeaderMatching/7_request_to_'/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_with_headers_'{Color:green}'_should_go_to_grpc-infra-backend-v1 (0.02s)
    --- PASS: TestConformance/GRPCRouteHeaderMatching/2_request_to_'/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_with_headers_'{Color:orange,Version:two}'_should_go_to_grpc-infra-backend-v1 (0.01s)
    --- PASS: TestConformance/GRPCRouteHeaderMatching/0_request_to_'/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_with_headers_'{Version:one}'_should_go_to_grpc-infra-backend-v1 (0.06s)
    --- PASS: TestConformance/GRPCRouteHeaderMatching/5_request_to_'/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_with_headers_'{Some-Other-Header:one}'_should_receive_a_Unimplemented_(12) (1.05s)
    --- PASS: TestConformance/GRPCRouteHeaderMatching/10_request_to_'/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_with_headers_'{Color:purple}'_should_receive_a_Unimplemented_(12) (1.05s)
    --- PASS: TestConformance/GRPCRouteHeaderMatching/9_request_to_'/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_with_headers_'{Color:yellow}'_should_go_to_grpc-infra-backend-v2 (1.06s)
    --- PASS: TestConformance/GRPCRouteHeaderMatching/4_request_to_'/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_with_headers_'{Color:orange}'_should_receive_a_Unimplemented_(12) (1.08s)
    --- PASS: TestConformance/GRPCRouteHeaderMatching/8_request_to_'/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_with_headers_'{Color:red}'_should_go_to_grpc-infra-backend-v2 (1.09s)
    --- PASS: TestConformance/GRPCRouteHeaderMatching/1_request_to_'/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_with_headers_'{Version:two}'_should_go_to_grpc-infra-backend-v2 (1.09s)
    --- PASS: TestConformance/GRPCRouteHeaderMatching/3_request_to_'/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_with_headers_'{Color:blue,Version:two}'_should_go_to_grpc-infra-backend-v2 (1.10s)
--- PASS: TestConformance/GRPCRouteListenerHostnameMatching (2.12s)
    --- PASS: TestConformance/GRPCRouteListenerHostnameMatching/6_request_to_'foo.com/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_should_receive_a_Unimplemented_(12) (0.01s)
    --- PASS: TestConformance/GRPCRouteListenerHostnameMatching/7_request_to_'no.matching.host/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_should_receive_a_Unimplemented_(12) (0.01s)
    --- PASS: TestConformance/GRPCRouteListenerHostnameMatching/0_request_to_'bar.com/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_should_go_to_grpc-infra-backend-v1 (0.01s)
    --- PASS: TestConformance/GRPCRouteListenerHostnameMatching/1_request_to_'foo.bar.com/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_should_go_to_grpc-infra-backend-v2 (0.01s)
    --- PASS: TestConformance/GRPCRouteListenerHostnameMatching/3_request_to_'boo.bar.com/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_should_go_to_grpc-infra-backend-v3 (0.03s)
    --- PASS: TestConformance/GRPCRouteListenerHostnameMatching/5_request_to_'multiple.prefixes.foo.com/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_should_go_to_grpc-infra-backend-v3 (0.03s)
    --- PASS: TestConformance/GRPCRouteListenerHostnameMatching/2_request_to_'baz.bar.com/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_should_go_to_grpc-infra-backend-v3 (0.04s)
    --- PASS: TestConformance/GRPCRouteListenerHostnameMatching/4_request_to_'multiple.prefixes.bar.com/gateway_api_conformance.echo_basic.grpcecho.GrpcEcho/Echo'_should_go_to_grpc-infra-backend-v3 (0.04s)

Closes #1783 #1790 #1765 #1768 #1838 #1837
Checklist
Before creating a PR, run through this checklist and mark each as complete.

[x] I have read the CONTRIBUTING doc
[x] I have added tests that prove my fix is effective or that my feature works
[x] I have checked that all unit tests pass after adding my changes
[x] I have updated necessary documentation
[x] I have rebased my branch onto main
[x] I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes
If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

Initial support for GRPCRoute, including Exact Method Matching, Header Matching, and Listener Hostname Matching

Sort of going back on my original comment, it's probably worth having an example grpcroute and app in the examples folder, just no need for the README guide because we can write a doc instead.

Sort of going back on my original comment, it's probably worth having an example grpcroute and app in the examples folder, just no need for the README guide because we can write a doc instead.

@sjberman The issue is the app though - for the other examples, we're using a sample NGINX so it's pretty straightforward, but for a gRPC app, we'll need to provide an image. There is a basic hello world test image here that we could use, but I'm not sure we should. The image I've been using for testing isn't ours, it's the test image from the conformance suite, and we should not use that in user facing examples. Without a working sample app, we're not offering anything over what's already available.

@ciarams87 I just want devs to be able to easily test things out. Maybe we include the grpcroute example configs, and then the README in there can link to the test app, without us directly providing it?

@ciarams87 I just want devs to be able to easily test things out. Maybe we include the grpcroute example configs, and then the README in there can link to the test app, without us directly providing it?

That makes sense. I've created a new PR for the example though, just to allow feedback there with requiring a full re-review of this one from Kate and Michael.


Activerel class array

Brings array options to ActiveRel's (from|to)_class
Deprecates all ActiveRel query methods.

Because query methods are deprecated, I did not modify them to support Array from/to.
If you are using these methods and want to disable the deprecation warning, set ActiveSupport::Deprecation.silenced = true.

Hi @subvertallchris! Maybe I am wrong, but this implementation feels broken...
As I understand, the Array will allow something like this:
                                -----> Apple
                               /
                              /
Basket ----- [:CONTAINS]-----
                              \
                               \
                                -----> Banana

I think, the matching ruby code would look like this:
class Basket
  include Neo4j::ActiveNode
  has_many :out, :apples, type: CONTAINS, model_class: Apple
  has_many :out, :bananas, type: CONTAINS, model_class: Banana
end

class Apple
  include Neo4j::ActiveNode
  property :weight, type: Integer
  has_one :in, :basekt, origin: :apples, model_class: Basket
end

class Banana
  include Neo4j::ActiveNode
  property :weight, type: Integer
  has_one :in, :basekt, origin: :apples, model_class: Basket
end

class ContainsRelation
  include Neo4j::ActiveRel
  from_class Basket
  to_class [ Apple, Banana ]
  type :CONTAINS
end

Now, i want to use the relation to put one apple and one banana into the basket.
basekt = Basket.create

apple = Apple.create(weight: 10)
banana = Banana.create(weight: 20)

ContainsRelation.create(from_node: basket, to_node: apple)
ContainsRelation.create(from_node: basket, to_node: banana)

But I get an error :open_mouth:
Neo4j::ActiveRel::Persistence::ModelClassInvalidError:
  Node class was Apple (54190140), expected [
    Apple(weight: Integer),
    Banana(weight: Integer)
  ] (55617160)

Tested with 67e11bd2.
If I take a look into #valid_type? in lib/neo4j/active_rel/persistence.rb i see this:
def valid_type?(type_object, node)
  case type_object
  when false, :any
    true
  when Array
    type_object.each { |c| return false unless valid_type?(c, node) }
    true
  else
    node.class.mapped_label_names.include?(type_object.to_s.constantize.mapped_label_name)
  end
end

For me, there are at least two bugs in this method...

I don't get how the else-part should work... It checks if an array contains another array. This won't work in this case: [:Apple].include?([:Apple]) # false
Shouldn't the Array-part use #any? to check if any specified class matches the given node!?

I think, the Array-part should look like this:
def valid_type?(type_object, node)
  # ...
  when Array
    type_object.any? { |c| valid_type?(c, node) }
  else
  # ...
end

Yup, looks like my tests didn't go far enough. I get really distracted by trying to write tests that didn't duplicate and clearly lost focus a little. I'll be home in three hours and will fix it first thing. Thanks for tracking that down.

Also not a huge deal but we normally recommend using symbols rather than the class constants just in case there's issues with the load order of your classes.

Huge +1 to that. I'd love to find a better way of handling that.

@cheerfulstoic: Yeah, i know! With rails, it works fine, without --> chicken-and-egg problem :smile:

@janpieper That fix will be merged in a sec, I had a lull at work so just popped it in and fixed the test.


authorizeURLHandler: How do I know when OAuthSwift has sent the user to Safari?

Description:
In my app, I want to get a token, so I'll send my user to the OAuth provider, I'm doing this by redirecting them to Safari:
OAuthSwift.authorizeURLHandler = OAuthSwiftOpenURLExternally.sharedInstance
Everything works perfectly, but one question: How do I know when my app has sent the user to Safari?
OAuth Provider (Twitter, Github, ..):
Not important.
OAuth Version:

[ ] Version 1
[X] Version 2

OS (Please fill the version) :

[x] iOS :
[ ] OSX :
[ ] TVOS :
[ ] WatchOS  :

Installation method:

[ ] Carthage
[X] CocoaPods
[ ] Manually

Library version:

[X] head
[ ] v1.0.0
[ ] v0.6
[ ] other: (Please fill in the version you are using.)

Xcode version:

[X] 8.0 (Swift 3.0)

[ ] 8.0 (Swift 2.3)

[ ] 7.3.1

[ ] other: (Please fill in the version you are using.)

[ ] objective c

the oauth flow with OAuthSwiftOpenURLExternally :

safari is launched
safari open the login page for the specific provider

safari have its own cookies, so user could be already logged

after success full login, callback url is call (and must open your app using url scheme with token as  query or fragment parameters)

there is no user information sent from the app to safari

@phimage Correct, but when using OAuthSwiftOpenURLExternally,  the app will automatically send the user to Safari, I don't need any user data. I just want to know when OAuthSwift has redirected to Safari. Is there a way to do that?

there is no "user" send to safari
just url open with on iOS : UIApplication.shared.openURL(url)
If you use OAuth version 2, just after calling authorize safari is open immediately
So you know when it's done, because you call authorize
for oauth version 1 is more complicated, there is before a request to the server
Then if you want a clean way to do this, implement your own OAuthSwiftURLHandlerType and use it
open class MonitoredHandler: OAuthSwiftURLHandlerType {

    public static var sharedInstance : MonitoredHandler = MonitoredHandler()

    @objc open func handle(_ url: URL) {
        // TODO : here send event : safari will open
         OAuthSwiftOpenURLExternally.handle(url)
       // TODO : here send event : safari did open
    }
}

just for information, OAuthSwiftOpenURLExternally is not recommend, apple do not allow it I think
so for ios9 there is SafariURLHandler with a delegate to know where safari is initialized

@phimage OK, so there is a way to open URLs in Safari, that would allow me to get approved by the App Store:
UIApplication.shared.open(NSURL(string:"http://www.reddit.com/") as! URL, options: [:], completionHandler: nil)

How would I set the URL handler to work with that?

not in Safari but SFSafariViewController using OAuthSwift  SafariURLHandler
and for your question, I don't really understand why you want to open in safari a reddit url... nothing in relation with OAuth
UIApplication.shared.open(url) is equivalent to OAuthSwiftOpenURLExternally.handle(url)

@phimage What would I put in url? The example I found had the reddit url in it.

@phimage So I would do this, right?:
oauthswift.authorizeURLHandler = OAuthSwiftOpenURLExternally.handle(url)
but what would I put in URL?


Fix error handling of libpsl-native

This removes a bunch of incorrect code which attempted to map Linux errno to Windows error codes. With .NET Core 1.0.0, this is unncesssary. They've implemented new Win32Exception(Marshal.GetLastWin32Error()) to work correctly on Linux, despite the odd name. Marshal.GetLastWin32Error() retrieves errno, and the constructor of Win32Exception calls strerror_r to get the associated POSIX error message.
In the process, I also refactored and simplified a fair bit of the native library calls to remove the need for extra abstraction functions.
I added a GetErrorCategory() native function that maps an errno to a PowerShell ErrorCategory on a best effort basis; this is primarily so that the existing error handling in the file-system provider can be reused, resulting in much better error messages (where native code is used, like with symlinks).
Currently, when New-Item -ItemType fails because the user doesn't have permissions, the error looks like this:
> mkdir /tmp/inaccessible
> chmod a-wrx /tmp/inaccessible
> New-Item -ItemType SymbolicLink -Path /tmp/inaccessible/link -Target .
New-Item : Input/output error
At line:1 char:1
+ New-Item -ItemType SymbolicLink -Path /tmp/inaccessible/link -Target  ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-Item], Win32Exception
    + FullyQualifiedErrorId : System.ComponentModel.Win32Exception,Microsoft.PowerShell.Commands.NewItemCommand

With these changes, it looks like this:
> New-Item : Administrator privilege required for this operation.
At line:1 char:1
+ New-Item -ItemType SymbolicLink -Path /tmp/inaccessible/link -Target  ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : PermissionDenied: (/home/andrew/src/PowerShell:String) [New-Item], UnauthorizedAccessException
    + FullyQualifiedErrorId : NewItemSymbolicLinkElevationRequired,Microsoft.PowerShell.Commands.NewItemCommand

Currently, when Set-Date fails because the user doesn't have permissions, the error looks like this:
> Set-Date (Get-Date)
Set-Date : Unix.NonWindowsSetDate error: 5
At line:1 char:1
+ Set-Date (Get-Date)
+ ~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Set-Date], InvalidOperationException
    + FullyQualifiedErrorId : System.InvalidOperationException,Microsoft.PowerShell.Commands.SetDateCommand

Note the absolutely useless message "Unix.NonWindowsSetDate error: 5".
With these changes, it looks like this:
> Set-Date (Get-Date)
Set-Date : Operation not permitted
At line:1 char:1
+ Set-Date (Get-Date)
+ ~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Set-Date], Win32Exception
    + FullyQualifiedErrorId : System.ComponentModel.Win32Exception,Microsoft.PowerShell.Commands.SetDateCommand

Where "Operation not permitted" was the POSIX error message for the errno set by settimeofday in the native library.
For comparison, it looks like this on Windows:
PS C:\Users\andschwa> Set-Date (Get-Date)
Set-Date : A required privilege is not held by the client
At line:1 char:1
+ Set-Date (Get-Date)
+ ~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Set-Date], Win32Exception
    + FullyQualifiedErrorId : System.ComponentModel.Win32Exception,Microsoft.PowerShell.Commands.SetDateCommand

Where A required privilege is not held by the client was returned by Windows.
Also, while the changes I made in f21e1230f4306424585f8e90681128dce1cb8883 I reverted in 1f8d40354fd559168701bf25a78bca4ebba05af4, I wish to keep them in the history as the addition of a configure script to the CMake system for the library was non-trivial, and will likely prove useful again.
I'm sure I've not caught every use of the native library where error handling can be improved, but these were the most glaring problems.

Hi @andschwa, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

It looks like you're a Microsoft contributor (Andy Schwartzmeyer). If you're full-time, we DON'T require a Contribution License Agreement. If you are a vendor, please DO sign the electronic Contribution License Agreement. It will take 2 minutes and there's no faxing! https://cla.microsoft.com.

TTYL, MSBOT;

@vors For the InformationRecord.NativeThreadId the test addition in Write-Information Cmdlet context passed on both platforms (it pretty much just asserts it wasn't 0, which it was before).

@mirichmo take a look

@vors tests added.


queries with timestamp values fail when using multistatements

have found an issue with the way the library currently handles prepared statements.
'use strict';
var Client = require('mariasql');

var c = new Client();

c.connect({
  host: "localhost",
  user: "root",
  password: "foo",
  db: "foobar",
  multiStatements: true
});

c.query("UPDATE a SET s = :s, d = DATE_FORMAT(DATE_ADD(CURDATE(),INTERVAL 60 DAY), '%Y-%m-%d 23:59:59') WHERE id = :id; CALL sp (:id, null, null, null);", {"s":"A", "id": 123}, function(err, rows) {
  if (err)
    throw err;
  console.dir(rows);
});

c.close();

you will notice the prepared statement is incorrectly parsed. the value of :id is added to the end of the string, causing the query to fail.
also was there any reason that you decided to prepare the query in javascript instead of using mysql_prepare? The prepare function appears that there could be edge cases where it fails.
anyways i manage to solve this problem by moving '%Y-%m-%d 23:59:59' to a named placeholder so this can be a workaround for now.

The reason the async prepared statements API is not used is because it's a real pain in the ass to work with and is why I never got anywhere with it. If you or anyone else wants to take a swing at it, I am definitely open to a PR to add it in. Once we have something like that, I will gladly get rid of the emulated prepared statements functionality.


Merge Failed.
This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.

Merge Failed.
This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.

Starting check jobs.

Starting check jobs.

recheck

Starting check jobs.


3496: Vendoring pwasm-utils 0.16 under the name wasm-utils

The pwasm-utils crate is not maintained anymore (the repository is archived), so this commit vendors it under the name wasm-utils. The code is simplified as well, with the ability to use no-std being removed since it is not used in the execution_engine crate.

PR no more relevant since it's now forked and lives in its own repo.


Consider keeping a line number or something in the Span object

I've started a crummy linter. Not much good yet; it's just got the one checker so far and that is the you're-actually-using-the-variable-before-it's-even-been-initialized checker. But it's finding problems where I put them, so.
The thing is I'd like to report to my end user something like "I found an issue on line 57". But information about line numbers is actually discarded by the time I figure out there's even a problem to report. I have reference to a Span object though, which so far as I can figure out, lets me know the (preprocessed) character count. I wonder if we can add the line number in there.

There is a helper module that can convert byte offsets into file name and line number. The reason not to store this in the Span directly, is because there are a lot of spans in the AST tree, so it pays to keep these objects as small as possible.

Do you know how get_location_for_offset is meant to work?
As the first parameter I've tried passing the filename, and I've tried passing the file's contents. In both cases, it panics in my case with a message like the following
thread 'main' panicked at 'byte index 68 is out of bounds of `#include <stdio.h>
int main() {
        int i;
        putchar(i);
        int x = i;
}
`', library/core/src/str/mod.rs:107:9

The documentation is a little unclear, but I figured out that get_location_for_offset does not expect the first parameter to be a file name, but the contents of the file. Do I need to expand the #includes and other directives myself?


LPFG-989: Update Learner UI to register learner

Requires learner-record running branch LPFG-989 to run locally.

Just need to rebase @robleney-kainos


Add ENVIRONMENT GHOST_URL to allow overwrite the default http://local…

…host:2368
The docker run -e url=http://www.example.com ... don't work with this image, because of the manual set via --url on the config command.

I've tried, and I can't seem to reproduce a failure with -e url=...: :confused:
$ docker run -it --rm --name ghost -e url=http://example.com:2368/ghost ghost
[2017-08-10 17:50:37] INFO Creating table: posts
[2017-08-10 17:50:37] INFO Creating table: users
[2017-08-10 17:50:37] INFO Creating table: roles
[2017-08-10 17:50:37] INFO Creating table: roles_users
[2017-08-10 17:50:37] INFO Creating table: permissions
[2017-08-10 17:50:37] INFO Creating table: permissions_users
[2017-08-10 17:50:37] INFO Creating table: permissions_roles
[2017-08-10 17:50:37] INFO Creating table: permissions_apps
[2017-08-10 17:50:37] INFO Creating table: settings
[2017-08-10 17:50:37] INFO Creating table: tags
[2017-08-10 17:50:37] INFO Creating table: posts_tags
[2017-08-10 17:50:37] INFO Creating table: apps
[2017-08-10 17:50:37] INFO Creating table: app_settings
[2017-08-10 17:50:37] INFO Creating table: app_fields
[2017-08-10 17:50:37] INFO Creating table: clients
[2017-08-10 17:50:37] INFO Creating table: client_trusted_domains
[2017-08-10 17:50:37] INFO Creating table: accesstokens
[2017-08-10 17:50:37] INFO Creating table: refreshtokens
[2017-08-10 17:50:37] INFO Creating table: subscribers
[2017-08-10 17:50:37] INFO Creating table: invites
[2017-08-10 17:50:37] INFO Creating table: brute
[2017-08-10 17:50:37] INFO Model: Post
[2017-08-10 17:50:37] INFO Model: Tag
[2017-08-10 17:50:37] INFO Model: Client
[2017-08-10 17:50:38] INFO Model: Role
[2017-08-10 17:50:38] INFO Model: Permission
[2017-08-10 17:50:38] INFO Model: User
[2017-08-10 17:50:38] INFO Relation: Role to Permission
[2017-08-10 17:50:39] INFO Relation: Post to Tag
[2017-08-10 17:50:39] INFO Relation: User to Role
[2017-08-10 17:50:39] INFO Finished database migration!
[2017-08-10 17:50:44] INFO Ghost is running in production...
[2017-08-10 17:50:44] INFO Your blog is now available on http://example.com:2368/ghost/
[2017-08-10 17:50:44] INFO Ctrl+C to shut down
[2017-08-10 17:50:44] INFO Ghost boot 4.742s

(Which also forces me to access it via /ghost/ in my browser, which is a neat feature I didn't realize Ghost had. :smile:)

This solution won't work. the environment variables in a dockerfile are evaluated during docker build, so essentially you end up with the same result of ghost config being run with --url http://localhost:2368.
Like I said in #77 -> running docker with the environment variable is the correct way to go about this.


can't open file 'audioAnalysis.py': [Errno 2] No such file or directory

So I'm facing this issue, and it's kind of driving me nuts. I've tried every variation of editing my bashrc file, and I still get
audioAnalysis.py': [Errno 2] No such file or directory
My current path where everything is installed is /Users/davegoldblatt/pyAudioAnalysis. That said, there is a pyAudioAnalysis directory inside of /Users/davegoldblatt/pyAudioAnalysis. That path is located at /Users/davegoldblatt/pyAudioAnalysis/pyAudioAnalysis.
I've tried
export PATH="/users/davegoldblatt/pyAudioAnalysis/:$PATH"
export PATH="/users/davegoldblatt/:$PATH"
PYTHONPATH=$PYTHONPATH:"/users/davegoldblatt/pyAudioAnalysis"
PYTHONPATH=$PYTHONPATH:"/users/davegoldblatt/"

Did I git clone incorrectly? does my bashrc file have a syntax error?

Same problem here. On a Mac. It could be related to brew?

Hi, I am facing the same issue.
python: can't open file 'AudioAnalysis.py': [Errno 2] No such file or directory
Please help


.net Core Tiered Compilation seems to cause lags

[EDIT]:
See https://github.com/dotnet/runtime/issues/13196
A provisionally fix seems to be to disable Tired Compilation.
If Tired Compilation is on it can be observed (Process Monitor) that the process creates a lot of threads:

[EDIT]: I quickly tested it with a fixed velocity and no lags occured. May be an issue of the Input Implementation.
At the moment this is primarily used in TransformComponent.Rotate and TransformComponent.RotateAround.

Fixed as long as Tiered Compilation stays off.


The Frames Ratio of Caption Buttons ( minmaxclose )

Suggestion, please consider to make the aspect ratio of the frames of caption buttons ( minmaxclose ) square or 1:1, this will make the appearance of files explorer more nicer and attractive, not like Windows 11 22H2, its ratio is 1:2, 1:2,5 so that the spacing of caption buttons is far away. Thank you🙏

Thank you for your feedback, were using the default AppWindow setup and don't have plans to change this.


[test]

@maxday is this test still ongoing?

@maxday is this test still ongoing?

no, I'll close :)


Builder not putting instances in fvar table

This is happening in https://github.com/SorkinType/SplineSans. I'm investigating.

No instances in the gftools-builder generated designspace file, even though there are in the fontmake file.

Oh, I think it's a misconfiguration.


Increment versions for containerservicefleet releases

Increment package versions for containerservicefleet releases

API change check
API changes are not detected in this pull request.


refactor(syntax): remove ModuleRecord::export_default

This value can be derived.

#7578  👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.


Add |= and ^= operator tokens (use &= as a reference)

TODO:
Currently, we're missing the &= |= and ^= tokens, and using them causes a compile-time error.
The expression a &= b will be compiled as a = a & b and we already have the functionality for that. Adding the tokens for the operators is sufficient.
How to implement

I've made a walkthrough on how to do it in #69 use it for a reference.
see it's diff and complete the todos

Finally
Don't forget to add some tests (at basics.pk).

Fixed by: #71


Separate Command Not Working.

Hello,
It seems that the /separate command is broken. When you use the command it will tell you "These players are now ignoring each other", but sadly that's not the case once I ran the command the players continued to be able to talk to one another.

Make sure to delete your bannedWords file in config/GriefPrevention so it generates the defaults correctly.


Extendable Criteria object for building DSLs to find particular entities

Splitting discussion from #14 into separate discussions. Paging @dig412.
I've got something similar that I've been working on for this as well, but it's not ready for prime time. I'll clean it up a bit and post it as a gist some time in the next week or two. It might be interesting to see if we can combine the ideas we've had separately.

This is a class that builds the parts of the WHERE clause through various method calls - it mostly differs from the built-in Criteria class in that it holds the clauses in an array until it is compiled so that you can combine them with ->and()-> and ->or()->.
We use it to allow good control over retrieving objects out of a repository without having to know many/any of the implementation details of the object itself.
For example, this class:
class EventCriteria extends AmissCriteria
{
	public function forVenue(Venue $venue)
	{
		$this->where("venueId", "=", $venue->id);
		return $this;
	}

	public function forEventIds(array $eventIds)
	{
		$this->where("id", "IN", $eventIds);
		return $this;
	}

	public function excludeHidden()
	{
		$this->where("hidden", "=", "0");
		return $this;
	}

	public function excludePastEvents()
	{
		$this->where("enddate", ">=", "DATE(NOW())");
		return $this;
	}

Is used in various places like this:
$criteria = (new EventCriteria())
	->excludeHidden()
	->forVenue($venue)
	->where("startdate", "=", $startDate->format("Y-m-d"));

$events = $manager->getList("Event", $criteria->compile());

It can be used directly, but I tend to extend it to provide model-specific methods. I've also implemented a debug() method that generated a SELECT query for testing purposes, which is really useful when getList returns unexpected results.
Shall I put the AmissCritera class up in a gist?

The ORM I was working on before I changed course and wrote Amiss used the 'lots of little method calls' pattern to build clauses. I ended up abandoning it because it didn't profile favourably. Having said that, I really like the look of this part of your example:
``php
$criteria = (new EventCriteria())
->excludeHidden()
->forVenue($venue)
That's quite elegant.

This is really a philosophical issue at its core, so I think I should explain where I'm coming from a bit first. When I looked at the state of PHP ORMs back when I first made Amiss (including the crappy precursor one I was working on at the time), I did some extensive ORM profiling and was (quite frankly) disgusted at the performance. The ORMs (even with caching enabled) were doing so much work that the cost of the query was utterly dwarfed by comparison. It really put the sword to the old "all optimisations are premature when your database query is going to be the slowest thing in your app" idea that gets bandied about any time someone tries to suggest that code should go faster ;)

I experimented a lot and learned that in PHP, function calls are nowhere near as cheap as I thought they were, and that you can go a long, long way to avoiding thousands of them (particularly string ones) with careful use of isset and comparison operators. You've probably noticed in the Amiss code that there are lots of weird little idioms that seem to go one step too far to avoid a simple function call, but it is exactly this detail that makes it as fast as it is. The code is intentionally a little bit imperfect for the sake of pragmatism and we reap some pretty good raw speed benefits as a consequence considering the features that it offers.

And for all that effort, Amiss is *still* slower than I'd like it to be. v5 is slightly faster than v4 in most cases, but not by enough to be significant. I just profiled the `array/select` example now and the overhead of the `buildQuery` method alone is exactly the same as the entire interaction with `PDO`! And the worst part about that is that that's a good result for an ORM!

Speed is one of my main goals for Amiss - I think I'm really just trying to prove that you can have feature-rich PHP that is as fast as it can possibly be if you make some careful - and perhaps sometimes heretical - choices about the design of the code.

At this stage, I'm reluctant to include something like this because I've tried it before. I do think this definitely merits further discussion though as the more specific chaining that you're providing with it - the '->excludeHidden()->forVenue($venue)` stuff - looks quite nice to my eyes. Maybe there's a way we can have it both ways :D