Refactor ArrowEngine for better read_parquet performance

Refactoring ArrowEngine to improve general performance (with an emphasis on breaking dependencies between split_row_groups and gather_statistics).  The motivation is one of the performance issues mentioned in #6264 (specifically here)
Key Changes:

The split_row_groups default is now None.  The behavior will switch to True if there is a _metadata file available (or if there is only one file in the dataset)
The user may now pass an integer value to split_row_groups.  We will then pack this many row-groups (or fewer) into each output partition.  Note that this is different from chunksize because split_row_groups does not require statistics processing (it is also the number of row-groups, regardless of variations in byte size).
Statistics are only gathered for "expected" index columns.  Since parsing statistics in python is a bit slow, we only gather statistics for a user-specified index column, or for the index columns specified in the "pandas metadata".  We also avoid gathering statistics if there is not filtering and we do not have an index column to calculate divisions for.

I think the key change here is that we have separated the gather_statistics and split_row_groups logic.  We are no longer required to gather statistics when the user sets split_row_groups=True (or an integer value).
Just to add some more motivation...  Here is the benefit I am seeing for the 1TB criteo dataset:
# This PR
%time dd.read_parquet(path, engine="pyarrow", split_row_groups=True).npartitions
CPU times: user 459 ms, sys: 31.2 ms, total: 490 ms
Wall time: 488 ms
6337

# Before
%time dd.read_parquet(path2, engine="pyarrow", split_row_groups=True).npartitions
CPU times: user 39.8 s, sys: 1.05 s, total: 40.9 s
Wall time: 40.2 s
6337

Note that I am also hoping to continue cleaning up this PR and make it a bit easier/cleaner to plug in the pyarrow.datasets API (in place of pq.ParquetDataset).  It may also make sense to overhaul the chunksize implementation (especially since the byte-size statistics are certainly wrong for string columns), but I'll need to move that work to a separate PR.
TODO:

[ ] Get feedback on expanded split_row_groups behavior (and default behavior)
[ ] Get feedback on modified gather_statistics behavior
[ ] General cleanup
[ ] Decide if/how to align chunksize and split_row_groups>1 behavior
[x] Tests added / passed (add test coverage for integer split_row_groups)
[x] Passes black dask / flake8 dask

cc @martindurant @TomAugspurger

Wow, another big PR! Please give us some time...
Does this affect anything on the fastparquet side?

Wow, another big PR! Please give us some time...

Yeah - I'm sorry about doing this, but I fear the reorganization is probably necessary.  Most dask_cudf code (which uses ArrowEngine.read_metadata) includes gather_metadata=False, because the default behavior is much too slow.
I'm certainly not expecting a thorough review in the immediate future.

Does this affect anything on the fastparquet side?

I have not modified anything on the fastparquet side here.

@rjzamora there is quite some overlap with https://github.com/dask/dask/issues/6174 / https://github.com/dask/dask/pull/6172. What do you think is the best way forward? Because many of the things I need to use / change in https://github.com/dask/dask/pull/6172 are being refactored here as well (so it doesn't seem that useful to continue working on that until this is merged, or I could base my POC on this branch).
Personally, I would say: while refactoring this, let's directly start using the new datasets API (otherwise we need to do another refactor of this refactor later on), but I certainly understand that that is a less direct and short-term improvement for users, as it will need some iteration on both dask and pyarrow's side.
(we still have a short time to get as much as possible of the required features for dask into pyarrow 1.0, so as I said before, help with testing it out with dask (and dask_cudf) is very welcome)

On your actual key changes (didn't yet look at the code, only your description at the top): those sound good to me. I am also running into "vagueness" around when split_row_groups, gather_statistics, etc are exactly triggered and when not and how those different keywords influence each other, and making this more explicit / transparent to the user seems a good idea.

The split_row_groups default is now None. The behavior will switch to True if there is a _metadata file available (or if there is only one file in the dataset)

The main reason here is to avoid IO to get the number of row groups in case this information is not yet available (from the _metadata file) ?
For your specific example of the criteo dataset, what is the main driver for this huge performance difference?

Statistics are only gathered for "expected" index columns.

So this would mean that you no longer check all columns to see which ones are sorted and can potentially be inferred as index column, but only do this check if specified by the user? (or gathered from the pandas metadata)
That sounds good to me (the logic around index is also quite complicated right now)

One more question: how do you avoid gathering statistics when split_row_groups=True ? Because for splitting in row groups, you need to know how many row groups there are, for which you need to read the FileMetadata? (or is this only avoiding parsing the statistics, but still requires reading the metadata, in which case I suppose it's only a speed-up when there is a _metadata file available?)

@rjzamora there is quite some overlap with #6174 / #6172. What do you think is the best way forward? Because many of the things I need to use / change in #6172 are being refactored here as well (so it doesn't seem that useful to continue working on that until this is merged, or I could base my POC on this branch).
Personally, I would say: while refactoring this, let's directly start using the new datasets API (otherwise we need to do another refactor of this refactor later on), but I certainly understand that that is a less direct and short-term improvement for users, as it will need some iteration on both dask and pyarrow's side.

Totally agree - Perhaps I should have left the "[WIP]" label here.  My immediate plan for this PR was to incorporate pyarrow.dataset to make sure that backend will "work" for >0.17.1 (and leave in whatever "dataset" components that we are okay with CI missing for now).  I was actually starting this earlier today.
Since we are having performance problems with the existing ArrowEngine, I don't think we can avoid a refactor like this.  I'm sorry if it feels like I'm "pulling the rug out" under #6172 , but I do think your POC is an extremely valuable reference.  Do you think it is reasonable for us to use #6172 as a template/example for this PR (and probably a followup after pyarrow 1.0 lands).

The split_row_groups default is now None. The behavior will switch to True if there is a _metadata file available (or if there is only one file in the dataset)

The main reason here is to avoid IO to get the number of row groups in case this information is not yet available (from the _metadata file) ?

Right - I am usually working with a local filesystem myself, but my understanding is that the IO cost can be dramatic for large datasets on non-local filesystems.  Therefore, the default behavior is to avoid collecting metadata if there is no _metadata available.  The default is certainly not set in stone here, and I am very open to changing it.  For example, I'm already realizing we should probably use split_row_groups=True as the default if the user specifies gather_statistics=True.

For your specific example of the criteo dataset, what is the main driver for this huge performance difference?

There are two general reasons (and maybe others I am forgetting):

Splitting by row group currently requires gather_statistics=True and statistics are currently parsed/summarized for all columns (not just those needed for filtering and/or divisions) -- This PR does not require any statistics parsing to split row-groups, and it only parses statistics for columns that are completely necessary.
There is some inefficient code used for parsing/summarizing statistics.  For example, we are doing slow things like this.  Statistics parsing is improved in this PR, but is also completely avoided when there is no filtering or index to calculate divisions for.

So this would mean that you no longer check all columns to see which ones are sorted and can potentially be inferred as index column, but only do this check if specified by the user? (or gathered from the pandas metadata)
That sounds good to me (the logic around index is also quite complicated right now)

Yes - I think inferring the index by searching for a sorted column is obsolete now that use of "pandas metadata" is pretty well established.  If the pandas metadata is not included, I think it is fair to expect the user to specify the index explicitly :)

@jorisvandenbossche - On the topic of using parrow.dataset:  When the user does list(ds.get_fragments(filter=ds.field("a") < 10)), should the returned list of fragments already be filtered, or is the filtering done when the fragment is read in as a table?  Passing in a filter argument doesn't seem to change the list of fragments I get.  (Sorry if this is the wrong forum for this question)

When the user does list(ds.get_fragments(filter=ds.field("a") < 10)), should the returned list of fragments already be filtered, or is the filtering done when the fragment is read in as a table?

Yes, in general there is this capability, and filtering is done during get_fragments, but only for full fragments (it won't yet filter actual rows): so it will filter out the fragment if it knows the full fragment can be skipped based on partitioning information or the statistics.
However, from now trying our different examples, there seems to be some caveats to this:

Filtering on a partition column always works as expected in get_fragments
Filtering on a normal column only works when splitting+filtering row groups, so not in plain get_fragments but in split_by_row_group (one obvious reason is that we don't "merge" statistics to see if a fragment consisting of multiple row groups can be skipped, apart from it simply not being implemented, I think, even for a dataset consisting of fragments backed by a single

Small example:
In [3]: df = pd.DataFrame({"part": ['A', 'A', 'B', 'B'], "col1": [0, 0, 1, 1], "col2": range(4)})

In [4]: df.to_parquet("test_filter_fragments/", partition_cols=["part"])

In [10]: filter_partition = ds.field("part") == "A"

In [11]: filter_column = ds.field("col1") >= 1

In [12]: dataset = ds.dataset("test_filter_fragments/", format="parquet", partitioning="hive")

# no filter -> 2 fragments
In [14]: len(list(dataset.get_fragments()))
Out[14]: 2

# filtering on partition -> works
In [15]: len(list(dataset.get_fragments(filter=filter_partition)))
Out[15]: 1

# filtering on columns -> doesn't work here directly ..
In [16]: len(list(dataset.get_fragments(filter=filter_column)))
Out[16]: 2

# .. but does work when further filtering on row groups
In [17]: len([rgfragment for fragment in dataset.get_fragments(filter=filter_column) for rgfragment in fragment.split_by_row_group(filter_column)])
Out[17]: 1

So: this is another reason that the split_row_groups can be useful. But, if that would be useful, we can also see if we can do more filtering without splitting by row groups (JIRAs are welcome ;))

Since we are having performance problems with the existing ArrowEngine, I don't think we can avoid a refactor like this. I'm sorry if it feels like I'm "pulling the rug out" under #6172 , but I do think your POC is an extremely valuable reference.

No problem! I am also happy that you are refactoring this, as doing it with the current codebase in #6172 is not that easy. We then just need to communicate good on how to approach this together.
So to make it explicit: you prefer to finish this refactor using the ParquetDataset so not having to rely on pyarrow master / future release for now? And then I can try to put my changes from #6172 on top of this PR?
Alternatively, I can also do a PR to your branch to directly include using pyarrow.dataset here in this PR. I am happy with both (or another) approaches.

and probably a followup after pyarrow 1.0 lands

Yep, only one aspect I am trying to say: ideally we shouldn't wait on testing this with pyarrow.dataset until pyarrow 1.0 lands, because then we discover a whole set of bugs/missing features, and need to wait until 1.1 to use it, etc ;)
(I know it's a bit annoying we can't write something that also works with released arrow, but, we have nightly conda packages to make it a bit easier to test with master!)

So to make it explicit: you prefer to finish this refactor using the ParquetDataset so not having to rely on pyarrow master / future release for now? And then I can try to put my changes from #6172 on top of this PR?
Alternatively, I can also do a PR to your branch to directly include using pyarrow.dataset here in this PR. I am happy with both (or another) approaches.

My initial intention for this PR was too add partial support for the pyarrow.dataset API (when specifically requested by the user), and then coordinate with you on the best way to port more functionality in a followup PR.  Note that I started experimenting with partial support in (a related branch)[https://github.com/dask/dask/compare/master...rjzamora:pyarrow-dataset].  However, I am noticing that we still need to work out what parts needs to look like to support partitioned datasets.
Something that worries me is that I had not previously considered that the actual fragment object might need to be added to every read_partion task to handle partitioning or full filtering support - My hope was that we could ultimately define an element of parts as: (<path>, <row-groupindex-list>, <partitioning-info>), and then a single fragment (or group of fragments) could be quickly constructed in the read_partition task.  My motivation is that dask_cudf is currently using ArrowEngine to process the metadata and construct the parts.

Yep, only one aspect I am trying to say: ideally we shouldn't wait on testing this with pyarrow.dataset until pyarrow 1.0 lands, because then we discover a whole set of bugs/missing features, and need to wait until 1.1 to use it, etc ;)
(I know it's a bit annoying we can't write something that also works with released arrow, but, we have nightly conda packages to make it a bit easier to test with master!)

Good point - If there are nightly conda packages, testing shouldn’t be much of a burden :)

There seems to be quite a lot of conversation and ideas here, that I have not had the chance to go through. Would it be useful to have some live time to talk next week? We could call it a get-together of the dask-IO group.

Good suggestion @martindurant - A live conversation sounds like a great idea to me :)

A live conversation sounds good for me as well!

However, I am noticing that we still need to work out what parts needs to look like to support partitioned datasets.

In my attempt, I am using the Fragments itself as parts (which are similar in idea as the ParquetDatasetPiece), instead of destructuring it in its components (the file path and filesystem, partition information, and potentially row group information) and then reconstructing it in the worker for read_partition.
Now, I think both approaches could be fine.
You mention dask_cudf as a motivation for the "decomposed" way: I was (maybe naively) thinking that in the dask_cudf's overridden CudfEngine.read_partition it could still get the path and partition information from the fragment there (to get the information dask_cudf needs)?
(in any case, I don't think it is realistic to keep parts exactly the same as is when using pyarrow.dataset, so starting to use that will require some changes in dask_cudf as well).
Also please feel free to chime in at https://issues.apache.org/jira/browse/ARROW-8655 with use cases for a more feature-rich partitioning information.

My initial intention for this PR was too add partial support for the pyarrow.dataset API (when specifically requested by the user), and then coordinate with you on the best way to port more functionality in a followup PR. Note that I started experimenting with partial support

After thinking about his a bit more (and reviewing the PR): this PR is already doing quite a lot, so adding even more might not be the best idea? (and we are also aready mixing discussions about the refactor itself with discussions about how to fit in pyarrow.datasets ..., not making it easier to follow this)
Now, I think it depends on whether you want to initially support this refactored parquet reader for both using the existing ParquetDataset as for the new pyarrow.dataset. When initially wanting to support both, it might make the most sense to focus in this PR on the refactor using ParquetDataset, and then have another PR to add support for pyarrow.datasets. If only supporting the new pyarrow.dataset version in the refactored engine is fine, we can directly add it to this PR.
This is of course something for dask to decide, but I see potential arguments for both ways: if this refactored engine replaces the existing ArrowEngine (as done now in this PR), we of course need to keep support for released pyarrow initially, and thus support ParquetDataset/pyarrow.dataset side by side. On the other hand, since this refactored engine is not fully backwards compatible (regarding splitting row groups/gathering statistics + regarding index inference), you migth want to have it opt in initially (?), and then directly doing it only using pyarrow.dataset would make sense I think.
@rjzamora what was your idea here regarding how to land it in dask?

In my attempt, I am using the Fragments itself as parts (which are similar in idea as the ParquetDatasetPiece), instead of destructuring it in its components (the file path and filesystem, partition information, and potentially row group information) and then reconstructing it in the worker for read_partition.
Now, I think both approaches could be fine.
You mention dask_cudf as a motivation for the "decomposed" way: I was (maybe naively) thinking that in the dask_cudf's overridden CudfEngine.read_partition it could still get the path and partition information from the fragment there (to get the information dask_cudf needs)?
(in any case, I don't think it is realistic to keep parts exactly the same as is when using pyarrow.dataset, so starting to use that will require some changes in dask_cudf as well).

I think we may need to follow your example (of passing the row-group fragments directly).  One difference is that we will need to be able to include multiple row-group fragments in a single read_partition.  You are correct that dask_cudf can just get the path information from the fragment object(s).  The only possible problem is if the fragment objects are too large to inject into the task graph for huge datasets.  That said, I don’t expect this to be a problem (it was when we originally passed around the pq.ParquetDataset object, but I’d expect those objects to be much larger).

Also, note that I do plan to chime in a lot more on Jira starting next week - However, I was a little unsure of the most-critical functionality to push on.

This is of course something for dask to decide, but I see potential arguments for both ways: if this refactored engine replaces the existing ArrowEngine (as done now in this PR), we of course need to keep support for released pyarrow initially, and thus support ParquetDataset/pyarrow.dataset side by side. On the other hand, since this refactored engine is not fully backwards compatible (regarding splitting row groups/gathering statistics + regarding index inference), you migth want to have it opt in initially (?), and then directly doing it only using pyarrow.dataset would make sense I think.
@rjzamora what was your idea here regarding how to land it in dask?

I would like pyarrow.dataset to be leveraged directly within Arrowengine (if possible).  This is why I would like this PR to be “workable” for pyarrow.dataset (even if integration isn’t actually included here).  I would like to avoid the need for a complete “overhaul” in a followup PR… With that goal in mind,  I am certainly feeling open minded about how we land things in Dask.

I spent a bit of time trying to work in the pyarrow.dataset API here.  It may make sense to roll back some of these changes, but I did want to explore the implications of the refactor.  I only added two simple tests (since the dataset api is only used if/when the user passes dataset={"pa_dataset": True}), but the new approach does seem to "work."
With that said, the uncomputed read_parquet benchmark I used above does suggest that processing "metadata" with the legacy API is a bit faster than processing a list of row-group fragments:
%time dd.read_parquet(path, engine="pyarrow", split_row_groups=True, dataset={"pa_dataset": False}).npartitions

CPU times: user 462 ms, sys: 31.6 ms, total: 493 ms
Wall time: 492 ms
6337

%time dd.read_parquet(path, engine="pyarrow", split_row_groups=True, dataset={"pa_dataset": True}).npartitions

CPU times: user 1.81 s, sys: 17.4 ms, total: 1.82 s
Wall time: 1.82 s
6337

Besides the slight performnace hit (which is likely my own fault), there are a few other challenges I ran into:

The legacy ParquetDataset API provided a lot of information about the partitioning of a dataset.  Having access to the partition names and partition keys is useful for DataFrame metadata creation.  We can handle partitioning with the dataset API, but it still feels a bit "hacky" - Hopefully we can make this feel a bit cleaner.
The new dataset API will perform filtering on partitioned data, but the filters need to be defined in a specific expression format (list of tuples and/or list of list of tuples are no longer supported - I think).  It would be nice to have a way to convert a pure-python representation of a filter to the pyarrow supported format.  Sorry if such a utility already exists, or if I am missing something here.
It seems that the dataset API does not provide min/max statistics for string columns (while the metadata accessible with the ParquetDataset API does).  These stats are needed to set a string column to an index with known divisions.
[Already mentioned elsewhere] We don't have the "total_byte_size" metadata/statistics for each row-group, so the chunksize option is currently off the table.  It is unclear to me if this option is ever used in practice (prior to this PR, it may have been too slow to use anyway).
[Already mentioned elsewhere]  It would be ideal to have support for all fsspec file-system specifications.

I can raise ARROW/Jira issues about (3) and (4),  but I'm not sure of the most appropriate "ask" for the others.

The legacy ParquetDataset API provided a lot of information about the partitioning of a dataset.

Yes, I agree it would be nice to expose more of this information, this is already tracked in https://issues.apache.org/jira/browse/ARROW-8655. Some feedback on that issue about what information exactly is required for dask would be welcome.

The new dataset API will perform filtering on partitioned data, but the filters need to be defined in a specific expression format. .. It would be nice to have a way to convert ..

See my POC at https://github.com/dask/dask/pull/6172/files#diff-931b3b9d202e0bdb2270b99af80f7427R101-R107. This is using a private utility from pyarrow.parquet right now, but so we should probably consider making this public.

It seems that the dataset API does not provide min/max statistics for string columns

This is going to be fixed shortly: ARROW-9108 / https://github.com/apache/arrow/pull/7623 (I plan to merge that once CI passes)

We don't have the "total_byte_size" metadata/statistics for each row-group

Should be easy to add, as mentioned in the Arrow PR, and opened https://issues.apache.org/jira/browse/ARROW-9346 for this

It would be ideal to have support for all fsspec file-system specifications.

This is already working on pyarrow master (https://github.com/apache/arrow/pull/7395). I am using this in https://github.com/apache/arrow/pull/7545/files#diff-e7af02a1b90798e9844faabd241a28dfR72-R77, so we would need to do something similar here to wrap an fsspec filesystem in a pyarrow handler (or actually, I should probably add this to the pyarrow.dataset API, so dask doesn't need to care about it for now).

processing "metadata" with the legacy API is a bit faster than processing a list of row-group fragments:

Did you do any profile to see which part in specific is the possible culprit?

processing "metadata" with the legacy API is a bit faster than processing a list of row-group fragments:

Did you do any profile to see which part in specific is the possible culprit?

The case you were timing might be a large dataset with a _metadata file? That's something that still needs to be handled specifically in dask, as the pyarrow.dataset API right now offers this as a separate function: parquet_dataset("path/to/_metadata") (so you were timing the case where all imetadata were read from the individual parquet files)

The case you were timing might be a large dataset with a _metadata file? That's something that still needs to be handled specifically in dask, as the pyarrow.dataset API right now offers this as a separate function: parquet_dataset("path/to/_metadata") (so you were timing the case where all imetadata were read from the individual parquet files)

That makes sense - I will try targeting the _metadata when available and profile a few different cases (with and without _metadata).
Thanks for the quick/throrough response to all of my challenges @jorisvandenbossche - You are a pleasure to work with!

With that said, the uncomputed read_parquet benchmark I used above does suggest that processing "metadata" with the legacy API is a bit faster than processing a list of row-group fragments:

Coming back to this difference in time to "discover" the dataset and create the task graph (what we were discussing yesterday on the call as well).
The difference might indeed be due to parsing statistics or not. So even when splitting row groups (for which you need to open the parquet FileMetadata to get the number of row groups), not parsing the statistics seems to give quite a speed-up.
I was testing locally with a part of the NYC taxi data (for 2.5 years (2016-07 - end 2018), one file per month, total disk size of 4.3 GB):
In [1]: import pyarrow.dataset as ds

In [2]: dataset = ds.dataset("notebooks-arrow/nyc-taxi-data/original-partitioned/", format="parquet", partitioning=["year", "month"])

In [3]: fragments = list(dataset.get_fragments())

In [4]: len(fragments)
Out[4]: 30

In [5]: %time [frag.ensure_complete_metadata() for frag in fragments]

Timing results of the last line of master vs commenting out parsing statistics / num_rows when collecting the metadata:
In [5]: %time [frag.ensure_complete_metadata() for frag in fragments]
# using https://github.com/apache/arrow/pull/7692
CPU times: user 3.27 s, sys: 12.2 ms, total: 3.28 s
Wall time: 3.38 s

# using https://github.com/apache/arrow/pull/7692 but without parsing statistics
CPU times: user 340 ms, sys: 0 ns, total: 340 ms
Wall time: 339 ms

(this is with a dataset without a _metadata file so it is reading metadata from each individual file, but it are only 30 files and on local filesystem, so still a relevant timing I think)

@jorisvandenbossche - Thank you for investigating the performance discrepancy!  Your findings seem consistent with my limited understanding of gathering statistics.  Is it possible/reasonable to add an argument (to get_fragments/split_by_row_group) to be explicit about whether or not to collect statistics?  For example, it would be useful for Dask to explicitly avoid collecting statistics in cases where there is no need to filter or calculate index divisions (a pretty common use case).  Note that I am seeing the performance discrepancy even when a _metadata file is present.

I opened https://issues.apache.org/jira/browse/ARROW-9459 to discuss this (we will probably not be able to fix this for 1.0, though, unless it happens today ..).

Note that I am seeing the performance discrepancy even when a _metadata file is present.

Yes, I assume that the timing difference would almost be identical (since, as noted above, the timing was with only a limited number of files on a local filesystem, so file access itself shouldn't matter much).

@martindurant @jorisvandenbossche -  Note that I pulled out use of the pyarrow.dataset API to make this PR a bit easier to review.  We can plug that logic back in after we review/merge these (already significant) changes.

Thanks! I think that will help gettings this part of the refactor reviewed and merged first. Note that I won't be able to give feedback short-term, as I will be offline until the end of the month.

I experimented a bit with a simple benchmark to (1) validate that the data-reading component of read_parquet is not negatively impacted by these changes, and (2) that the performance of the metadata-processing component is improved.
bench.py:
import dask.dataframe as dd
from dask.datasets import timeseries
import glob
import time
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--write", action="store_true")
parser.add_argument("--split_row_groups", action="store_true")
parser.add_argument("--no_meta", action="store_true")
parser.add_argument("--skip_read", action="store_true")
args = parser.parse_args()

path = "test.bench.parquet"

if args.write:
    ddf = timeseries(
        start='2000-01-01', end='2000-03-01', partition_freq='1h'
    ).to_parquet(path, engine="pyarrow", row_group_size=1000)
    print("Dataset Written.")
else:
    if args.no_meta:
        path = glob.glob(path + "/*.parquet")
    start_t = time.time()
    ddf = dd.read_parquet(
        path,
        engine="pyarrow",
        split_row_groups=args.split_row_groups,
        gather_statistics=args.split_row_groups,
    )
    metatime = time.time() - start_t
    if not args.skip_read:
        ddf.compute(scheduler="synchronous")
    runtime = time.time() - start_t
    npartitions = ddf.npartitions

    print("Runtime [s]:", runtime)
    print("Reading Metadata [s]:", metatime)
    print("Reading Data [s]:", runtime - metatime)
    print("npartitions:", npartitions)

For a simple timeseries dataset with 5760 row-groups, the data-reading performance seems unchanged by this PR, while the metadata-processing component is clearly improved.  Note that the nearly 20x boost in ArrowEngine.read_metadata performance for splitting by row-group (with a _metadata file) was a key motivation for this PR...
 I'm not likely to have any minor comments, so I think that we go ahead and merge this and see how things go. Thoughts @martindurant?

OK, I had a look through here and I don't have any architectural concerns. I'm not likely to have any minor comments, so I think that we go ahead and merge this and see how things go. Thoughts @martindurant?

Thanks for reviewing @TomAugspurger !   I am certainly hoping to get this in asap, but I certainly want Martins blessing :)

Thanks for all the clarifications. I'm happy with what's here, however you decide to deal with those marginal edge cases.

Would you be able to summarize the status of this PR Rick for people who may be less plugged in?

Thanks for the ping @jakirkham - This PR is ready to go from my perspective.  It will break dask_cudf, but I am in the process of putting together a cudf PR to address that.

Sounds like we should merge this rather than xfailing as suggested in #6461. I'm going to press the button.

Great to see this merged!
@rjzamora do you plan to open a PR with the changes related to pyarrow.dataset that you took out of this PR?

Great to see this merged!

+1
It's been great to see the cross project collaboration and communication
here too
On Fri, Aug 7, 2020 at 6:48 AM Joris Van den Bossche <
<EMAIL_ADDRESS>wrote:

@jorisvandenbossche commented on this pull request.
In dask/dataframe/io/parquet/arrow.py
https://github.com/dask/dask/pull/6346#discussion_r467049116:

   base, fns = _analyze_paths(paths, fs)

   proxy_metadata = None

   if "_metadata" in fns:

       # We have a _metadata file. PyArrow cannot handle

       #  "_metadata" when `paths` is a list. So, we shuld

       # open "_metadata" separately.

       paths.remove(fs.sep.join([base, "_metadata"]))

       fns.remove("_metadata")

       proxy_metadata = (

           pq.ParquetDataset(

               fs.sep.join([base, "_metadata"]),

               filesystem=fs,

               filters=filters,

               **dataset_kwargs,

           )

           .pieces[0]

but I believe the missing filters argument means we cannot handle
filtering on partitioned columns.
But the _metadata file shouldn't be filtered out anyway, I think? So I
think your code snippet above might be a bit cleaner (although doesn't
matter much in practice)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/pull/6346#discussion_r467049116, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTBSCJPQP7NYM5UNZWLR7QALFANCNFSM4OIY4HAA
.

@rjzamora do you plan to open a PR with the changes related to pyarrow.dataset that you took out of this PR?

Yes - I do plan to resurect the original pyarrow.dataset-based changes from this PR in a new PR.  However, I am unlikely to have the time to push the changes into a merge-ready state until the week of August 16th.  I'd also be happy to let you take over if you are motivated to push on this sooner (no worries if not - I am hapy to do the work here).

No problem, I am also busy with other things right now (we are also adding dataset writing in Arrow), happy to collaborate on it later this month.


ArgumentError: argument --self_test: conflicting option string(s): --self_test

I just installed tensorflow and no problem when I enter python -m tensorflow.models.image.mnist.convolutional in console
But when I open convolutional.py on spyder, it appears that
File "/usr/lib/python2.7/argparse.py", line 1467, in _handle_conflict_error
    raise ArgumentError(action, message % conflict_string)
ArgumentError: argument --self_test: conflicting option string(s): --self_test

and other example have the same problem such as
 File "/usr/lib/python2.7/argparse.py", line 1467, in _handle_conflict_error
    raise ArgumentError(action, message % conflict_string)
ArgumentError: argument --batch_size: conflicting option string(s): --batch_size

anyone can solve it??

@keveman: Did you get a chance to look at this?

@shiuan5566 What platform are you trying this on?

@keveman  I use Kubuntu 15.10

I am not the best person to debug this. Throwing it over to @martinwicke so he can find someone with access to different platforms.

I don't know what spyder is, and the fact that it runs ok in plain python makes me think this is not actually our error.
These types of errors are caused by the flag initialization code such as flags.DEFINE_string('batch_size', ...) running twice, or two different parts of the program defining flags with the same name (for self_test that would be a pretty likely candidate, batch_size not so much.

Automatically closing because there was no response and this is an unsupported configuration. Please reopen if it is still an issue and you verify it is not caused by Spyder's execution environment running initialization code twice. Thank you.

I ran into the same problem. I'm running word2vec.py from https://github.com/tensorflow/models/tree/master/tutorials/embedding
I'm running it on a google datalab notebook, on debian.


1.11.5 Patch Release Checklist

[ ] Dapr maintainer(s) communicate out the discovery of issues that require a patch release and the reason why to the Discord Maintainers and Release channels. Patch releases are made for one of categories below:

[ ] Security vulnerability

https://github.com/dapr/dapr/pull/7117

[ ] Regression that does not have a work around:

https://github.com/dapr/dapr/pull/7002
https://github.com/dapr/dapr/pull/7018
https://github.com/dapr/dapr/pull/7057
https://github.com/dapr/dapr/pull/7068

[ ] Broken mainline scenario that has a missing test case:

[ ] Create v1.11.5-rc.3 Tag
[ ] Performance tests passing:
[ ] End to End tests passing on Linux and Windows:
[ ] New test case written to catch future occurrences
[ ] Notify users to try RC1
[ ] Update the longhaul tests to use v1.11.5-rc.3
[ ] Write release notes :  https://github.com/dapr/dapr/pull/7144
[ ] Review release notes [@dapr/maintainers-dapr ]
[ ] Create v1.11.5 Tag [@dapr/maintainers-dapr ]
[ ] Backport fixes into master branch [@dapr/maintainers-dapr ]
[ ] Update the documentation: Latest version & versions in supported releases
[ ] Push new tag in installer-bundle repo.
[ ] Announce the patch release on Discord Announcements channel

v1.11.5 released https://github.com/dapr/dapr/releases/tag/v1.11.5


typo ellipsis, added

source

You need to squash all these commits together, see http://docs.godotengine.org/en/latest/community/contributing/pr_workflow.html#mastering-the-pr-workflow-the-rebase

This is not really a typo, it's done on purpose since as of 2019, there is still no easy way for lambda computer users to input … on their keyboard. If you're a bit tech savvy you might know how to do it, or if you're a bit less tech savvy you'll type ... in an advanced editor like Word, let it convert it, and copy paste it everywhere... But it's definitely not a convenient workflow.
So now, I don't want to presume that translators will be able to do this extra work, so IMO it's better to simply use three dots in the source form. If we want the editor to then display an ellipsis character, that can be done programmatically.

Before spending too much time doing such changes, I'd advise discussing with upstream whether this is done on purpose and if it should be fixed. Otherwise it's potentially a lot of time lost for you if we end up rejecting the proposal.

Closing for now as per above comments. I'm not strongly against the change, but it has to be discussed beforehand and then handled properly in a single commit. We also have to take care not to invalidate hundreds of translations needlessly, so .po files should likely see the same substitution for languages where this character is relevant.


findBy* typings: Remove Error as a possible return value

What:

Remove | Error from findBy* return typings.

Why:

The findBy* functions throw errors, so an Error will never actually be returned. Similar to the getBy* functions.
See https://github.com/kentcdodds/dom-testing-library/issues/226#issuecomment-482396530 for more context.

How:

I updated each of the 4 return typings for the findBy* functions and removed | Error.

Checklist:

[ ] Documentation added to the docs site N/A
[x] Typescript definitions updated
[ ] Tests N/A
[x] Ready to be merged

https://github.com/Microsoft/TypeScript/blob/master/src/lib/es2015.promise.d.ts#L13
This looks correct, you can't type the rejected value of a Promise (thought you could 😅), and the error is not a return value, it's an exception

Codecov appears to be having issues https://status.codecov.io/ but this looks safe to merge when it's back up

Thanks!
@allcontributors[bot], please add @bopfer for test

@allcontributors[bot], please add @JaxCavalera for review

Whoops, let me try that again..
@allcontributors[bot], please add @JaxCavalera for review

One last time!
@allcontributors[bot], please add @JaxCavalera for review

:tada: This PR is included in version 3.19.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


无法进入安装器

您好
电脑是LG Gram 13Z990 i7-8550u 16G内存
用黑果小兵的镜像 做完USB后原封不动地替换使用您的EFI
现在无法进入安装器 直接尝试启动会黑屏无反应
打开 -v -x 选项启动看到出现memory panic
https://streamable.com/0qft2i
日志全录下了 求大神帮我看下问题出在哪里
试了整天了 这些全都尝试过了:
10.15.0 到 10.15.6 试了全部不行
删除所有分区并初始化SSD
BIOS更新到最新版本W1ZD1250
BIOS还原默认设置
Secure Boot已关
换了不同U盘
更新EFI里clover版本到最新
更新EFI里部分kext版本到最新 (我是小白,复杂的操作不太会)

Try turning off safe mode in bios

@jydjygv
Sorry, I do not know where is safe mode setting in BIOS, can you say more detail? If you mean secure boot, I turn it off already.

@jydjygv
Sorry, I do not know where is safe mode setting in BIOS, can you say more detail? If you mean secure boot, I turn it off already.

@jydjygv
Sorry, I do not know where is safe mode setting in BIOS, can you say more detail? If you mean secure boot, I turn it off already.

@jydjygv
Sorry, I do not know where is safe mode setting in BIOS, can you say more detail? If you mean secure boot, I turn it off already.

@jydjygv
Sorry, I do not know where is safe mode setting in BIOS, can you say more detail? If you mean secure boot, I turn it off already.

@jydjygv
Sorry, I do not know where is safe mode setting in BIOS, can you say more detail? If you mean secure boot, I turn it off already.

@jydjygv
Sorry, I do not know where is safe mode setting in BIOS, can you say more detail? If you mean secure boot, I turn it off already.

@jydjygv
Sorry, I do not know where is safe mode setting in BIOS, can you say more detail? If you mean secure boot, I turn it off already.

@jydjygv
Sorry, I do not know where is safe mode setting in BIOS, can you say more detail? If you mean secure boot, I turn it off already.

LG GRAM EFI.zip
try this EFI

yes secure boot is what i meant to say sorry

@jydjygv AMAZING I can boot to installer now. I did not solve for the whole day but you help me solve in a few minutes. Thank you so much, you are so great.

Installation finish perfectly, even trackpad works. Once again, very sincerely THANK YOU.


HDFS-17420. EditLogTailer and FSEditLogLoader support FGL

EditLogTailer and FSEditLogLoader support FGL.
Both EditLogTailer and FSEditLogLoader need Global lock.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 21s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ HDFS-17384 Compile Tests _

+1 :green_heart:
mvninstall
32m 41s

HDFS-17384 passed

+1 :green_heart:
compile
0m 47s

HDFS-17384 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
compile
0m 43s

HDFS-17384 passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
checkstyle
0m 40s

HDFS-17384 passed

+1 :green_heart:
mvnsite
0m 45s

HDFS-17384 passed

+1 :green_heart:
javadoc
0m 43s

HDFS-17384 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
1m  7s

HDFS-17384 passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
1m 53s

HDFS-17384 passed

+1 :green_heart:
shadedclient
21m 34s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 35s

the patch passed

+1 :green_heart:
compile
0m 38s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javac
0m 38s

the patch passed

+1 :green_heart:
compile
0m 36s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
javac
0m 36s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 30s

the patch passed

+1 :green_heart:
mvnsite
0m 37s

the patch passed

+1 :green_heart:
javadoc
0m 30s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
1m  2s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
1m 42s

the patch passed

+1 :green_heart:
shadedclient
21m 11s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
204m 40s

hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 30s

The patch does not generate ASF License warnings.

295m  8s

Subsystem
Report/Notes

Docker
ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6679/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6679

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 84bdbb5307df 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
HDFS-17384 / 806a5f1e1acba9d6cfa397471318e2d92a995953

Default Java
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6679/1/testReport/

Max. process+thread count
4439 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6679/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@ZanderXu Thanks for contribution. @huangzhaobo99 @RocMarshal Thanks for review. Merged.


https://elshor.github.io/dstools/ not working

The link https://elshor.github.io/dstools/
returns cannot get.

Sorry but this repo is not maintained anymore.
El
On Wed, May 26, 2021 at 6:07 PM jallmer @.***> wrote:

The link https://elshor.github.io/dstools/
returns cannot get.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/elshor/dstools/issues/5, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB2DJZHZOIXO27KYUDFWUGDTPUFBXANCNFSM45SIKLGQ
.


implementing an equation

Dear developpers,
I have my own FVM method for solving the eqaution below but it is slow (~10us per time step) because a lot of space is wasted as the solution is quite localized. Hence, I was hoping to use mesh adaptation in Trixi. BUT:
I am intimidated by the framework. Basically, I want to simulate
$$\partial_t \mu +\partial_x(F^1(x,y)\mu)+ \partial_x(F^2(x,y)\mu) + \partial_x^2\mu=0$$
Looking at the docs, I understand that I could use https://trixi-framework.github.io/Trixi.jl/stable/tutorials/adding_nonconservative_equation/
but this is written in non-conservative form. I guess my question is "Is it worth it?" Would it be faster than my in-house code (explicit euler with upwind fvm)?

I don't think that's a non-conservative from? That looks like conservative form with a parabolic (e.g., 2nd order derivative) term.
Just checking, is the sign on $\partial_x\mu$ correct? Usually I see a negative sign on that term.

yes it is conservative, I mentioned the link because the vector field is not constant
There is no sign - because mu is a measure but we can put the minus if this measure has a density.
Is this an equation handled by trixi?

Thanks for the clarification - yes, I think you'd need to follow the approach in the docs for spatially varying coefficients, which is to just include those as an extra set of unknowns. However, this isn't too difficult - you'd just create a set of equations with both mu and F as unknowns, and initialize them both in the initial_condition function. When you define the flux, you would just set the flux for F to zero so that it's not evolved.
In terms of speed, it probably depends on what your solution looks like. If it's smooth and differentiable, yes, I think Trixi should be faster than a 1st or 2nd order FVM for higher levels of accuracy. If it has a lot of shocks or discontinuities, I'm not as sure.

no it is smooth

Great, then I think it should do fairly well. Both using a higher order timestepper and using a higher order spatial discretization should improve accuracy.
If you plan on implementing this using Trixi, please don't hesitate to contact us if we can help clarify anything about the interface.

ok thanks, I will give it a try

This is tough...

I dont know how to write the function flux_conservative. If you can give me a push, it'd be lovely

using Trixi
using Trixi: AbstractEquations, get_node_vars
import Trixi: varnames, default_analysis_integrals, flux, max_abs_speed_naive,
              have_nonconservative_terms

# Since there is no native support for variable coefficients, we use 3
# variables: one for the basic unknown `u` and 2 for the coefficient `a`
struct IsolatedNeuron <: AbstractEquations{2 #= spatial dimension =#,
                                           3 #= 3 variables (u,F1,F2) =#}
end

varnames(::typeof(cons2cons), ::IsolatedNeuron) = ("scalar", "F1", "F2")

default_analysis_integrals(::IsolatedNeuron) = ()

# The conservative part of the flux is zero
flux(u, orientation, equation::IsolatedNeuron) = zero(u)

# Calculate maximum wave speed for local Lax-Friedrichs-type dissipation
function max_abs_speed_naive(u_ll, u_rr, orientation::Integer, ::IsolatedNeuron)
    _, F1_ll, F2_ll = u_ll
    _, F1_rr, F2_rr = u_rr
    return max(abs(F1_ll), abs(F2_ll), abs(F1_rr), abs(F1_rr))
end

# We use conservative terms
have_nonconservative_terms(::IsolatedNeuron) = Trixi.False()

function flux_conservative(u_mine, u_other, orientation,
                              equations::IsolatedNeuron)
    _, F1, F2 = u_mine
    scalar, _ = u_other
    return SVector(F1 * scalar + F2 * scalar, zero(scalar), zero(scalar))
end

I don't think this is a non-conservative equation, so I think you only need to construct flux, which I believe should look like
function flux(u, orientation, equations::IsolatedNeuron)
    scalar, F1, F2 = u
    if orientation == 1
        return SVector(F1 * scalar, zero(scalar), zero(scalar))
    else # if orientation == 2
        return SVector(F2 * scalar, zero(scalar), zero(scalar))
end

I also think have_nonconservative_terms defaults to False()

Thank you!
I now have the following but it complains LoadError: UndefVarError: solve
using Trixi
using Trixi: AbstractEquations, get_node_vars
import Trixi: varnames, default_analysis_integrals, flux, max_abs_speed_naive,
have_nonconservative_terms
Since there is no native support for variable coefficients, we use 3
variables: one for the basic unknown u and 2 for the coefficient a
struct IsolatedNeuron <: AbstractEquations{2 #= spatial dimension =#,
3 #= 3 variables (u,F1,F2) =#}
end
varnames(::typeof(cons2cons), ::IsolatedNeuron) = ("scalar", "F1", "F2")
default_analysis_integrals(::IsolatedNeuron) = ()
The conservative part of the flux is zero
flux(u, orientation, equation::IsolatedNeuron) = zero(u)
Calculate maximum wave speed for local Lax-Friedrichs-type dissipation
function max_abs_speed_naive(u_ll, u_rr, orientation::Integer, ::IsolatedNeuron)
_, F1_ll, F2_ll = u_ll
_, F1_rr, F2_rr = u_rr
return max(abs(F1_ll), abs(F2_ll), abs(F1_rr), abs(F1_rr))
end
We use conservative terms
have_nonconservative_terms(::IsolatedNeuron) = Trixi.False()
function flux(u, orientation, equations::IsolatedNeuron)
scalar, F1, F2 = u
if orientation == 1
return SVector(F1 * scalar, zero(scalar), zero(scalar))
else # if orientation == 2
return SVector(F2 * scalar, zero(scalar), zero(scalar))
end
end
using OrdinaryDiffEq
equation = IsolatedNeuron()
function initial_condition(x, t, equation::IsolatedNeuron)
scalar = exp( -((x[1]-1)^2+x[2]^2) / (2*0.3^2))
F1 = -x[1]
F2 = -x[2]
SVector(scalar, F1, F2)
end
Create a uniform mesh in 1D in the interval [-π, π] with periodic boundaries
mesh = TreeMesh((-2.,-2.), (2., 2.), # min/max coordinates
initial_refinement_level=4, n_cells_max=10^4)
Create a DGSEM solver with polynomials of degree polydeg
Remember to pass a tuple of the form (conservative_flux, nonconservative_flux)
as surface_flux and volume_flux when working with nonconservative terms
volume_flux  = (flux_central, flux)
surface_flux = (flux_lax_friedrichs, flux)
solver = DGSEM(polydeg=3, surface_flux=surface_flux,
volume_integral=VolumeIntegralFluxDifferencing(volume_flux))
Setup the spatial semidiscretization containing all ingredients
semi = SemidiscretizationHyperbolic(mesh, equation, initial_condition, solver)
Create an ODE problem with given time span
tspan = (0.0, 1.0)
ode = semidiscretize(semi, tspan)
Set up some standard callbacks summarizing the simulation setup and computing
errors of the numerical solution
summary_callback = SummaryCallback()
analysis_callback = AnalysisCallback(semi, interval=50)
callbacks = CallbackSet(summary_callback, analysis_callback)
OrdinaryDiffEq's solve method evolves the solution in time and executes
the passed callbacks
sol = solve(ode, Tsit5(), abstol=1.0e-6, reltol=1.0e-6,
save_everystep=false, callback=callbacks)

but OrdinaryDiffEq is in the code above....

I fixed it by importing it before Trixi.

The solve errors
1-element ExceptionStack:
LoadError: MethodError: objects of type Tuple{typeof(flux_central), typeof(flux)} are not callable
Stacktrace:
  [1] flux_differencing_kernel!
    @ ~/.julia/packages/Trixi/wpKZk/src/solvers/dgsem_tree/dg_2d.jl:262 [inlined]
  [2] flux_differencing_kernel!
    @ ~/.julia/packages/Trixi/wpKZk/src/solvers/dgsem_tree/dg_2d.jl:248 [inlined]
  [3] macro expansion
    @ ~/.julia/packages/Trixi/wpKZk/src/solvers/dgsem_tree/dg_2d.jl:236 [inlined]
  [4] #565
    @ ~/.julia/packages/Polyester/W70Gq/src/closure.jl:274 [inlined]
  [5] macro expansion
    @ ~/.julia/packages/Polyester/W70Gq/src/batch.jl:159 [inlined]
  [6] _batch_no_reserve
    @ ~/.julia/packages/Polyester/W70Gq/src/batch.jl:112 [inlined]
  [7] batch
    @ ~/.julia/packages/Polyester/W70Gq/src/batch.jl:231 [inlined]
  [8] macro expansion
    @ ~/.julia/packages/Polyester/W70Gq/src/closure.jl:387 [inlined]
  [9] macro expansion
    @ ~/.julia/packages/Trixi/wpKZk/src/auxiliary/auxiliary.jl:246 [inlined]
 [10] calc_volume_integral!(du::StrideArraysCore.PtrArray{Float64, 4, (1, 2, 3,

Try not to pass a tuple, but only
volume_flux  = flux_central
surface_flux = flux_lax_friedrichs

You need a Tuple for nonconservative equations. Since your equation is conservative only one flux is needed.

Great. The Laplacian can probably just use LaplaceDiffusion2D

Great. The Laplacian can probably just use LaplaceDiffusion2D though maybe the sign is wrong.

OK, I think I got it. But the mass is not conserved and the density becomes negative :(
using Trixi: AbstractEquations, get_node_vars
import Trixi: varnames, default_analysis_integrals, flux, max_abs_speed_naive,
              have_nonconservative_terms

# Since there is no native support for variable coefficients, we use 3
# variables: one for the basic unknown `u` and 2 for the coefficient `a`
struct IsolatedNeuron <: AbstractEquations{2 #= spatial dimension =#,
                                           3 #= 3 variables (u,F1,F2) =#}
end

varnames(::typeof(cons2cons), ::IsolatedNeuron) = ("scalar", "F1", "F2")

default_analysis_integrals(::IsolatedNeuron) = ()

# The conservative part of the flux is zero
flux(u, orientation, equation::IsolatedNeuron) = zero(u)

# Calculate maximum wave speed for local Lax-Friedrichs-type dissipation
function max_abs_speed_naive(u_ll, u_rr, orientation::Integer, ::IsolatedNeuron)
    _, F1_ll, F2_ll = u_ll
    _, F1_rr, F2_rr = u_rr
    return max(abs(F1_ll), abs(F2_ll), abs(F1_rr), abs(F1_rr))
end

# We use conservative terms
have_nonconservative_terms(::IsolatedNeuron) = Trixi.False()

function flux(u, orientation, equations::IsolatedNeuron)
    scalar, F1, F2 = u
    if orientation == 1
        return SVector(F1 * scalar, zero(scalar), zero(scalar))
    else # if orientation == 2
        return SVector(F2 * scalar, zero(scalar), zero(scalar))
    end
end

function initial_condition(x, t, equation::IsolatedNeuron)
    scalar = exp( -((x[1]-1)^2+x[2]^2) / (2*.2^2))
    F1 = -x[1]
    F2 = -x[2]
    SVector(scalar, F1, F2)
end

equation_transport = IsolatedNeuron()
equations_parabolic = LaplaceDiffusion2D((1.,1.), equation_transport);

# Create a uniform mesh in 1D in the interval [-π, π] with periodic boundaries
mesh = TreeMesh((-4.,-4.), (4., 4.), # min/max coordinates
                initial_refinement_level=4, n_cells_max=10^4)

# Create a DGSEM solver with polynomials of degree `polydeg`
# Remember to pass a tuple of the form `(conservative_flux, nonconservative_flux)`
# as `surface_flux` and `volume_flux` when working with nonconservative terms
volume_flux  = flux_central
surface_flux = flux_lax_friedrichs
solver = DGSEM(polydeg=3, surface_flux=surface_flux,
               volume_integral=VolumeIntegralFluxDifferencing(volume_flux))

# Setup the spatial semidiscretization containing all ingredients
semi = SemidiscretizationHyperbolic(mesh, equation, initial_condition, solver)

# Create an ODE problem with given time span
tspan = (0.0, 2.0)
ode = semidiscretize(semi, tspan)

# Set up some standard callbacks summarizing the simulation setup and computing
# errors of the numerical solution
summary_callback = SummaryCallback()
analysis_callback = AnalysisCallback(semi, interval=50)
callbacks = CallbackSet(summary_callback, analysis_callback)

# OrdinaryDiffEq's `solve` method evolves the solution in time and executes
# the passed callbacks
sol = solve(ode, Tsit5(), abstol=1.0e-6, reltol=1.0e-6,
            save_everystep=false, callback=callbacks)

plot(sol)

pd = PlotData2D(sol)

pd = PlotData2D(sol(0.), semi);plot(pd["scalar"], color = :viridis, title = "t=0")

pd = PlotData2D(sol(2.), semi);plot(pd["scalar"], color = :viridis, title = "t=2")

Given the vector field, the density should to a dirac at (x,y)=0 without the Laplacian. The Laplacian prevents this so I am expecting an invariant distribution around 0

Good question, at the moment we require that you do not add a type annotation for equations in the initial condition when using parabolic terms

OK

The current version of the code being:
using OrdinaryDiffEq, Trixi, Plots

using Trixi: AbstractEquations, get_node_vars
import Trixi: varnames, default_analysis_integrals, flux, max_abs_speed_naive,
              have_nonconservative_terms

# Since there is no native support for variable coefficients, we use 3
# variables: one for the basic unknown `u` and 2 for the coefficient `a`
struct IsolatedNeuron <: AbstractEquations{2 #= spatial dimension =#,
                                           3 #= 3 variables (u,F1,F2) =#}
end

varnames(::typeof(cons2cons), ::IsolatedNeuron) = ("scalar", "F1", "F2")

default_analysis_integrals(::IsolatedNeuron) = ()

# The conservative part of the flux is zero
flux(u, orientation, equation::IsolatedNeuron) = zero(u)

# Calculate maximum wave speed for local Lax-Friedrichs-type dissipation
function max_abs_speed_naive(u_ll, u_rr, orientation::Integer, ::IsolatedNeuron)
    _, F1_ll, F2_ll = u_ll
    _, F1_rr, F2_rr = u_rr
    return max(abs(F1_ll), abs(F2_ll), abs(F1_rr), abs(F1_rr))
end

# We use conservative terms
have_nonconservative_terms(::IsolatedNeuron) = Trixi.False()

function flux(u, orientation, equations::IsolatedNeuron)
    scalar, F1, F2 = u
    if orientation == 1
        return SVector(F1 * scalar, zero(scalar), zero(scalar))
    else # if orientation == 2
        return SVector(F2 * scalar, zero(scalar), zero(scalar))
    end
end

function initial_condition(x, t, equation)
    scalar = exp( -((x[1]-1.2)^2+(x[2]-4)^2) / (2*.2^2))
    # scalar = exp(-(x[1]-v0b)^2/(2 * σv0^2) -(x[1]-w0b)^2/(2 * σw0^2) )
    F1 = -x[1]
    F2 = -x[2]
    SVector(scalar, F1, F2)
end

equation_transport = IsolatedNeuron()
equations = LaplaceDiffusion2D((1.,1.), equation_transport);

# Create a uniform mesh in 1D in the interval [-π, π] with periodic boundaries
mesh = TreeMesh((-2.5,-2.), (5., 12.), # min/max coordinates
                initial_refinement_level=4, n_cells_max=10^5)

# Create a DGSEM solver with polynomials of degree `polydeg`
# Remember to pass a tuple of the form `(conservative_flux, nonconservative_flux)`
# as `surface_flux` and `volume_flux` when working with nonconservative terms
volume_flux  = flux_central
surface_flux = flux_lax_friedrichs
solver = DGSEM(polydeg=3, surface_flux=surface_flux,
               volume_integral=VolumeIntegralFluxDifferencing(volume_flux))

# Setup the spatial semidiscretization containing all ingredients
semi = SemidiscretizationHyperbolic(mesh, equations, initial_condition, solver)

# Create an ODE problem with given time span
tspan = (0.0, 4.0)
ode = semidiscretize(semi, tspan)

# Set up some standard callbacks summarizing the simulation setup and computing
# errors of the numerical solution
summary_callback = SummaryCallback()
analysis_callback = AnalysisCallback(semi, interval=50)
callbacks = CallbackSet(summary_callback, analysis_callback)

# OrdinaryDiffEq's `solve` method evolves the solution in time and executes
# the passed callbacks
sol = solve(ode, Tsit5(), abstol=1.0e-6, reltol=1.0e-6,
            save_everystep=false, callback=callbacks)

Taking a look...

Have you had a closer look at this elixir or a similar one? Maybe this helps as an inspiration.

Yeah, the elixir @JoshuaLampert linked would help. The main issue is that you are still using SemidiscretizationHyperbolic instead of SemidiscretizationHyperbolicParabolic (see https://github.com/trixi-framework/Trixi.jl/blob/dfdaef5adc243fcd6c55262a69ede59fe3a32a8f/examples/tree_2d_dgsem/elixir_advection_diffusion.jl#L45-L49)

Exactly. Then you can also just keep function initial_condition(x, t, equation::IsolatedNeuron), I guess.

I think you still need to remove the annotation from initial_condition because it will be called with both equation and equation_parabolic in different parts of the code (like callbacks)

Wait, I'm wrong - sorry @JoshuaLampert, you're correct, you can keep the type annotation in that case then.

Thank you for the elixr. It helped me with the boundary condition. I still have an error from solve. These are not easy to debug
using OrdinaryDiffEq, Trixi, Plots
# using AbbreviatedStackTraces

# using FHNMF
# neuron = FHNMF.NeuronParam2d(a = 0.01, b = 0.1, Iext = 4.0, J = 0.05)
# mesh_n = MeshPDE2d(Nw=300, param_neuron = neuron)

using Trixi: AbstractEquations, get_node_vars
import Trixi: varnames, default_analysis_integrals, flux, max_abs_speed_naive,
              have_nonconservative_terms

# Since there is no native support for variable coefficients, we use 3
# variables: one for the basic unknown `u` and 2 for the coefficient `a`
struct IsolatedNeuron <: AbstractEquations{2 #= spatial dimension =#,
                                           3 #= 3 variables (u,F1,F2) =#}
end

varnames(::typeof(cons2cons), ::IsolatedNeuron) = ("scalar", "F1", "F2")

default_analysis_integrals(::IsolatedNeuron) = ()

# The conservative part of the flux is zero
flux(u, orientation, equation::IsolatedNeuron) = zero(u)

# Calculate maximum wave speed for local Lax-Friedrichs-type dissipation
function max_abs_speed_naive(u_ll, u_rr, orientation::Integer, ::IsolatedNeuron)
    _, F1_ll, F2_ll = u_ll
    _, F1_rr, F2_rr = u_rr
    return max(abs(F1_ll), abs(F2_ll), abs(F1_rr), abs(F1_rr))
end

# We use conservative terms
have_nonconservative_terms(::IsolatedNeuron) = Trixi.False()

function flux(u, orientation, equations::IsolatedNeuron)
    scalar, F1, F2 = u
    if orientation == 1
        return SVector(F1 * scalar, zero(scalar), zero(scalar))
    else # if orientation == 2
        return SVector(F2 * scalar, zero(scalar), zero(scalar))
    end
end

function initial_condition(x, t, equation::IsolatedNeuron)
    scalar = exp( -((x[1]-1.2)^2+(x[2]-4)^2) / (2*.2^2))
    F1 = -x[1]
    F2 = -x[2]
    # F1 = FHNMF.f(x[1], x[2], neuron)
    # F2 = FHNMF.f(x[1], x[2], neuron)
    SVector(scalar, F1, F2)
end

equations_transport = IsolatedNeuron()
equations_parabolic = LaplaceDiffusion2D((1.,1.), equations_transport);

# Create a uniform mesh in 1D in the interval [-π, π] with periodic boundaries
mesh = TreeMesh((-2.5,-2.), (5., 12.), # min/max coordinates
                initial_refinement_level=4, n_cells_max=10^5)

# Create a DGSEM solver with polynomials of degree `polydeg`
# Remember to pass a tuple of the form `(conservative_flux, nonconservative_flux)`
# as `surface_flux` and `volume_flux` when working with nonconservative terms
volume_flux  = flux_central
surface_flux = flux_lax_friedrichs
solver = DGSEM(polydeg=3, surface_flux=surface_flux,
               volume_integral=VolumeIntegralFluxDifferencing(volume_flux))

# Setup the spatial semidiscretization containing all ingredients
# semi = SemidiscretizationHyperbolic(mesh, equations, initial_condition, solver)

boundary_condition_zero_dirichlet = BoundaryConditionDirichlet((x, t, equations) -> SVector(0.0))

boundary_conditions_hyperbolic = (; x_neg = boundary_condition_do_nothing,
                                    y_neg = boundary_condition_do_nothing,
                                    y_pos = boundary_condition_do_nothing,
                                    x_pos = boundary_condition_do_nothing)

boundary_conditions_parabolic = (; x_neg = boundary_condition_zero_dirichlet,
                                   y_neg = boundary_condition_zero_dirichlet,
                                   y_pos = boundary_condition_zero_dirichlet,
                                   x_pos = boundary_condition_zero_dirichlet);

semi = SemidiscretizationHyperbolicParabolic(mesh,
                                             (equations_transport, equations_parabolic),
                                             initial_condition, solver;
                                             boundary_conditions=(boundary_conditions_hyperbolic,
                                                                  boundary_conditions_parabolic))

# Create an ODE problem with given time span
tspan = (0.0, 4.0)
ode = semidiscretize(semi, tspan)

# Set up some standard callbacks summarizing the simulation setup and computing
# errors of the numerical solution
summary_callback = SummaryCallback()
analysis_callback = AnalysisCallback(semi, interval=50)
callbacks = CallbackSet(summary_callback, analysis_callback)

sol = solve(ode, Tsit5(), abstol=1.0e-6, reltol=1.0e-6,
            save_everystep=false, callback=callbacks)

The error being:
ERROR: MethodError: no method matching *(::Tuple{Float64, Float64}, ::SVector{3, Float64})

Closest candidates are:
  *(::Any, ::Any, ::Any, ::Any...)
   @ Base operators.jl:578
  *(::StaticArraysCore.StaticArray{Tuple{N, M}, T, 2} where {N, M, T}, ::StaticArraysCore.StaticArray{Tuple{N}, T, 1} where {N, T})
   @ StaticArrays ~/.julia/packages/StaticArrays/cZ1ET/src/matrix_multiply.jl:10
  *(::Union{StaticArraysCore.StaticArray{Tuple{s1, s2}, T, 2}, LinearAlgebra.Adjoint{T, <:StaticArraysCore.StaticArray{Tuple{s1, s2}, T, 2}}, LinearAlgebra.Diagonal{T, <:StaticArraysCore.StaticArray{Tuple{s1}, T, 1}}, LinearAlgebra.Hermitian{T, <:StaticArraysCore.StaticArray{Tuple{s1, s2}, T, 2}}, LinearAlgebra.LowerTriangular{T, <:StaticArraysCore.StaticArray{Tuple{s1, s2}, T, 2}}, LinearAlgebra.Symmetric{T, <:StaticArraysCore.StaticArray{Tuple{s1, s2}, T, 2}}, LinearAlgebra.Transpose{T, <:StaticArraysCore.StaticArray{Tuple{s1, s2}, T, 2}}, LinearAlgebra.UnitLowerTriangular{T, <:StaticArraysCore.StaticArray{Tuple{s1, s2}, T, 2}}, LinearAlgebra.UnitUpperTriangular{T, <:StaticArraysCore.StaticArray{Tuple{s1, s2}, T, 2}}, LinearAlgebra.UpperTriangular{T, <:StaticArraysCore.StaticArray{Tuple{s1, s2}, T, 2}}} where {s1, s2, T}, ::StaticArraysCore.StaticArray{Tuple{N}, T, 1} where {N, T})
   @ StaticArrays ~/

Would you mind posting the unabbreviated stacktrace?

I am not sure whether LaplaceDiffusion2D accepts a Tuple for the diffusivity. Looks like that could be the problem on the first glance.

One possible source of error:
boundary_condition_zero_dirichlet = BoundaryConditionDirichlet((x, t, equations) -> SVector(0.0))

should probably be
boundary_condition_zero_dirichlet = BoundaryConditionDirichlet((x, t, equations) -> SVector(0.0, 0.0, 0.0))

since you have 3 "variables" in your equation

Looks like it does not...

Is it easy to add just a diffusion on the x variable?

Yes, if you don't mind a hacky way, you could just redefine
https://github.com/trixi-framework/Trixi.jl/blob/dfdaef5adc243fcd6c55262a69ede59fe3a32a8f/src/equations/laplace_diffusion_2d.jl#L22-L29
so that it returns zero if orientation == 2

Judging from the opened issues, it seems I cannot use AMR with diffusion, is it true or not?

Unfortunately, yes, though we are very close to merging the PR for that https://github.com/trixi-framework/Trixi.jl/pull/1629

I have one more question if I may. That the equation is in fact
$$\partial_t \mu +\partial_x([F^1(x,y) + \int g\mu]\ \mu)+ \partial_x(F^2(x,y)\mu) + \partial_x^2\mu=0$$
you see that the vector field is modified by the integral of $g\mu$ for some known g. Would it be possible to tackle this with a callback for example? Or maybe adding a nonlinearity

Can this issue be closed or are there any open questions, @rveltz?

Yes we can close. It was really helpful.


Can't add controller

Would want to clear textfield

I agree with @Davete0302. I know it's trick because there's essentially many text fields, so a custom controller would do.

Will look into this. Should add a fix in a few days

Any update on this please

Hello
I need fix for this. Could you please add fix ASAP

I will look at it over the weekend. Sorry for the delay. Got so busy with work

@ramachandra9345 @phillipjacobs @Davete0302
Sorry for the delay. But I have updated the api.
You should be able to clear text now. ANd I have also exposed the textControllers for every textField


Add extension for gradle project doesn't work on Windows OS

I try to reproduce the issue 211 on my Windows OS.

Open the quarkus-gradle-project that we use for our test.
Call the Quarkus: Add extensions to current project command
Add an extension (ex. Agroal)

Terminal falis with:
Execution failed for task ':addExtension'.
> Could not find quarkus-universe-bom-1.2.0.Final.json (io.quarkus:quarkus-universe-bom:1.2.0.Final).
  Searched in the following locations:
      file:/C:/Users/azerr/.m2/repository/io/quarkus/quarkus-universe-bom/1.2.0.Final/quarkus-universe-bom-1.2.0.Final.json

I am getting a different problem when trying to add an extension to that project on Windows 10:
Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain

Our quarkus-gradle-project is used for test and I suppose it misses some files.
That's why I tried to create a project from scratch with vscode-quarkus command and add a new extension and it works pretty well.
I close this issue.


Token return Null Android 6

I follow Read Me and setup done , although see bug about folder debug ( delete , create ) , but app still running , when i check log-android OMG token always is null
who can help me .

I can run sucess , who can't run contact me<EMAIL_ADDRESS>

What was the solution? I have this problem on some devices. But I do not have access to the device to debug.

@EroticCode how do you do it?

please use grapp for recived google service or you can you physycal devices , sr i not online , my laptop have problem .

I'm having the same issue. not error in the log...


Bug fixes and unittests for kBE.

Fixed extraneous attributes when calling cons_fock from kBE modules
Added unit tests for kBE with chempot and dm
Fixed kbe_polyacetylene.py imports

Feel free to merge this - this all looks good to me.

@oimeitei Are you okay with giving me write access so I can merge?


Learning Pathways no longer available in the Look Book Service

Learning Pathways no longer available in the Look Book Service. Unable to provision

Hi @shirazgaff ,
Thanks for reporting the issue. This might be a temporary glitch with the provisioning engine, can you please try after few hours?
Thanks!

It is back online


Store: add failing test for proxy heap sorting issue

This test was created with production data from a user sent to me. It shows that despite getting sorted responses from store apis the end result of proxy heap is not sorted because of adverse store labels. The offender in this case is that receivers are not instructed to drop the "cluster" label so all series from the receivers are sorted before series from the store making them overall unsorted in the prometheu sense.

Store labels:
Receive labels:

The test failure is intended to show the issue

So i have a proposition how to solve all this:

we should establish the invariant that store apis always respond with sorted series; to do that in first iteration we pipe all sends through "resortingServer" at first
if we do it like that we can get rid of cuckoofilter stuff since we are ( at first ) always resorting to keep the invariant
if we have the invariant that store responses are sorted we can just use whole labelset in proxy heap Less and end up sorted in the expected way always; thus solving the problem
we can optimize the resorting away in some cases in followups if we can prove that the response is already sorted ( or still sorted after adding extlabels )

Continued in #6706


Post modal video uploads

Follow this checklist for submitting a pull request (PR):

[X] Your branch is up-to-date with the main branch.
[X] You are submitting your PR against the main branch of the repository you are contributing to from your fork.
[X] You have linked your GitHub issue at the Resolves Issue section if it exists.
[X] You have provided a brief description of the issue.

Description
Quick description of what this pull request does.

This pull request resolves the video upload issue, yet, i am not sure if it can break something or if there are any security problems, ill explain why i did everything so it can help to find remanescent bug faster

Problem with not deleting file from the database solved, also, the issue with the button not apearing in some articles was given because in the action for post with image the uuid was missing, also solved.

Resolves Issue: proudly closes #203 and #223

I wanna know if theres anything i can fix before merge

I wanna know if theres anything i can fix before merge

i was developing a player for one of my projects its not ready but i think we can use here

I'm running a codespace right now I have some questions but I might be able to answer them myself.

Oh my goodness @Ronnie018  you fixed it I could hug you right now

@Ronnie018 @ShashwatAgrawal20 @ashwaryy I have a zoom scheduled with Brian Hill (owner of Edovo) at 11 am, it's 12:39 right now. He wants to know what we need help with so what are some of the biggest issues we will not be able to figure out ourselves.  Also I had my mentorship meeting tonight and I learned about a few things we need to figure out at some point, like bottlenecking, if you think about our code as a pipeline between the frontend and backend and say the user push a lot of information at once, where is that pipe going to burst. I was thinking Sash might know a thing or 2 about testing the app for this stuff?

Also, I need to find a react library to call on the .json file for when I wright the code for the check boxes. So if any of you have an idea for that library I'm all ears. I'm going to try to work on this tomorrow.

https://www.youtube.com/watch?v=yVIO2yKe9o8

Also, I need to find a react library to call on the .json file for when I wright the code for the check boxes. So if any of you have an idea for that library I'm all ears. I'm going to try to work on this tomorrow.

You mean the code to feed the checkboxes components? If the json will be in the sourcecode you can read the file directly

?

Yes he is a decent guy, it cut a lot of our meeting short but he is going to reschedule with me and some of his IT will touch base with me within the next few days.
In regards to the json file, I found this site: https://www.learnbestcoding.com/post/81/3-ways-to-import-a-json-file-in-react
I'm working on that now


TypeScript types missing Promise support for .wrap function

Currently the type definition contain the following declaration for the wrap function:
export type WrapArgsType<T> =
  | string
  | ((callback: CallbackFunc<T>) => void)
  | CachingConfig
  | CallbackFunc<T>; // <- invalid type

export interface Cache {
  wrap<T>(...args: WrapArgsType<T>[]): Promise<T>;
}

However, this declaration doesn't support promises. I think the correct definition should be something like this:
export type WrapArgsType<T> =
  | string
  | ((callback: CallbackFunc<T>) => void)
  | CachingConfig
  | (() => PromiseLike<T> | T); // <- correct type

export interface Cache {
  wrap<T>(...args: WrapArgsType<T>[]): Promise<T>;
}

(Note I replaced CallbackFunc<T> with () => PromiseLike<T> | T)
How can we fix this?

Hi. The TypeScript types are managed in a different repo by someone else: https://github.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/cache-manager

PR opened here: https://github.com/DefinitelyTyped/DefinitelyTyped/pull/58974


Publish to Hex

Figure out why this happens:
===> Status Code: Forbidden (403)
Hex Error: account not authorized for this action

there is an elixir version of this with the same name lol...


Missing LICENSE file

Greetings!!
Really nice implementation.
I was wondering, under what license does this repo can be used?

Added licence https://github.com/J-Zeitler/geometry-clipmaps/blob/d0682fa8b3e98c4a70495be89de0280fb9b61e90/LICENSE
heads up: this repo contains (a very old version of) threejs which should also have MIT license
the entire project is very old, but I think this is what it does: https://www.youtube.com/watch?v=4D01JDwdfmI

I am updating my terrain system and I found this repository. I wanted to update my previous CDLOD tiled terrain implementation with a new one that worked similar to Geometry Clipmaps but that used only one  quad, or similar to the Nested Grids but that required no heightmap fetch to morph between levels, that is when I found your code!!!! you have saved me a lot of headaches.

The image above is from GodotEngine 4.2

Looks great! Good that this old code can be of use :D


Error in step 'fillChains'

Hi ,
I have been facing a problem with make_chains.py. When I align Target genome with query genome A (contigs 186, genome lenght 600Mb) worked perfectly fine. But thenwhen I want to align a query B (contigs 1500, 700 mb) against the same target, I always found an error in step fillChains. Species A and B are closely related. I was wondering what is wrong. I am attaching the .log file.
Thanks in advance,
Diego
make_chains.log

I think I am missing  RepeatFiller.py

Maybe you have an old git clone. We added RepeatFiller as git submodule (https://github.com/hillerlab/GenomeAlignmentTools) 6 months ago.
Pls do a clean git pull and try again.

If it's still relevant, please try the updated pipeline version (2.0.0) which will be released within next 24 hours.
Please create a new issue if you experience any troubles.


Add CRUD endpoints for reefs

This PR adds CRUD endpoints for reefs.

@vickykont-programize Let's just make sure the tests pass before merging - it looks like the Reefs controller needs a few more dependencies mocked before it gets injected.


Added a new algorithms book under "Learn More"

This book is different from other algorithms books for its rich illustrations. It's immensely helpful for those who are visual learner like myself.

Sounds good. Thanks for the recommendation 👍.
Happy New Years!


Support tilde ~ in key file path

Add suport  tilde ~ in key file path.
If keyFile begin with tilde followed by / slash or \ backslash depends on os, like
~/path-to-key.asc

should be replace by
${user.home}/path-to-key.asc

In class PGPKeyInfo we need to change method keyFromFile and confirm by unit tests

connected with #81

I think the challenge here is that the configured paths already come as java.io.File type values.
One not-so-clean option is to unpack the File to name and create another File instance with modified file path.


'Reserved' field in IPv4 Header

"What is 'reserved' field in ipv4 header format? Also "options" field is missing in the header."

Ipv4 packet defines flags field as 3 bits: reserved, dont_fragment, more_fragments
Rather than having users figure out what bit to set in a single flags field the field is broken out into those 3 fields.
If you require options for a specific use case please add a separate issue.

Add descriptions to ipv4 packet header fields.

options will not be included at this time as there is no customer request for this field.

implemented 3d4579c6201eb7af90076088ad3517f4bdacbccc


Use ensure_ndarray to view chunk as an array

Follow-up to PR ( https://github.com/zarr-developers/zarr/pull/352 )
Simplifies the process of constructing an ndarray to view the chunk data when writing the result to a destination. Also removes the last usage of np.frombuffer from library code (still used in examples, tests, and docs).
TODO:

[x] Add unit tests and/or doctests in docstrings
[x] Add docstrings and API docs for any new/modified user-facing classes and functions
[x] New/modified features documented in docs/tutorial.rst
[x] Changes documented in docs/release.rst
[x] Docs build locally (e.g., run tox -e docs)
[x] AppVeyor and Travis CI passes
[x] Test coverage is 100% (Coveralls passes)

Great, thanks!


[CE1.x] SEA with node20

The results of the experiment are pending because the obstacle still exists.


[Pygen] Add Command Line Support

Command-line support for Pygen.

@googlebot I consent

Looks good to me!

lgtm, thanks for adding initial cmdline support!


Properties editor should have the ability to disable the save button if there are required parameters not filled

This will be a configuration option
Provide a way for consumers to know if all the required fields are filled or not. This will be useful in cases where they are not using the properties editor buttons
Allow consumers to enable/disable the Save button through the properties controller.
Add a callback once “properties loaded” (after set form). This will let apps know to check if required fields are empty or not

A new configuration option disableSaveOnRequiredErrors has been added to the CommonProperties config object. If this is set to true, the "Save" button will be disabled if there are any Required errors.
New functions are added into the propertiesController for applications to set the disabled state for the "Save" button. These functions only set the state of the button for the main editor panel, and not the subPanel views.
/*
 * Set the main "save" button to disabled(true) or enabled(false)
 * @param saveDisable (boolean)
 */
setSaveButtonDisable(saveDisable)

/*
 * Returns the true if the main "save" button is disabled, false otherwise
 * @return boolean
 */
getSaveButtonDisable()

The existing getErrorMessage(s) functions in the propertiesController have been updated with a new parameter filterDisplayError = true
/*
 * Returns current list of error messages
 * @filterDisplayError (boolean) optional. If true, will not return error messages that are not displayed in the editor
 */
getErrorMessages(filteredPipeline, filterHiddenDisable, filterSuccess, filterDisplayError = true)
getErrorMessage(propertyId, filterHiddenDisable = false, filterSuccess = false, filterDisplayError = true)

New functions have been added to the propertiesController to get error messages
getAllErrorMessages()
getRequiredErrorMessages()

There is a new callback through the propertyListener(data) where data = { action: "PROPERTIES_LOADED" }. This callback will be invoked once all the data has been loaded in CommonProperties. At this point, all the property values and error messages have been set and evaluated.


on change event not firing

Can you show me how to trigger the onChange event with 'form-control' class?
There seems to be a problem triggering the on change event when dropdown.js is used.

It should already trigger the change event.
But is bugged... lol
https://github.com/FezVrasta/dropdown.js/blob/master/jquery.dropdown.js#L113
I've written .change() instead of .trigger('change').
If you wanna send a PR is welcome, otherwise I'll do it when I can.

My apologies, I wasn't having the latest dropdownjs. In the previous version, '.change()' didn't exist. Now it is working fine. Thanks!


lid.project_onto_image return zero arrays for uv and colors.

I tested this code in two part of the datasets boreas-2021-01-26-10-59 and boreas-2021-10-05-15-35, which return a zero arrays for uv and colors.
https://github.com/utiasASRL/pyboreas/blob/eeb2b1bb302f5386eff7198858846f6d89675fda/pyboreas/tutorials/lidar_camera_projection.ipynb#L114
my code
for idx in range(len(seq.lidar_frames)):
            try:
                cam = seq.get_camera(idx)
                lid = seq.get_lidar(idx)
                lid.remove_motion(lid.body_rate)
                T_enu_camera = cam.pose
                T_enu_lidar = lid.pose
                T_camera_lidar = np.matmul(get_inverse_tf(T_enu_camera), T_enu_lidar)
                lid.transform(T_camera_lidar)
                lid.passthrough([-75, 75, -20, 10, 0, 40])  # xmin, xmax, ymin, ymax, zmin, zmax
                # Project lidar points onto the camera image, using the projection matrix, P0.
                uv, colors, _ = lid.project_onto_image(seq.calib.P0)
                fig = plt.figure(figsize=(24.48, 20.48), dpi=100)
                ax = fig.add_subplot()
                ax.imshow(np.zeros(cam.img.shape, dtype=np.uint8)) # you can change it to this `np.zeros(cam.img.shape, dtype=np.uint8)` for black background
                ax.set_xlim(0, 2448)
                ax.set_ylim(2048, 0)
                ax.scatter(uv[:, 0], uv[:, 1], c=colors, marker=',', s=3, edgecolors='none', alpha=0.7, cmap='jet')
                ax.set_axis_off()

I downloaded the full dataset for this two parts.
If any more information is require, please tell me.

I found the same problem in boreas-2021-11-28-09-18 part too

Hey, what you're trying to do will not work for those sequences because those sequences are in the test set, where no ground truth poses are provided. So, when you do cam.pose or lid.pose, they're just returning an identity matrix.
You can try this again with a sequence that is a part of the training set. You can find out which sequences are a part of the training / test splits here:
https://github.com/utiasASRL/pyboreas/blob/eeb2b1bb302f5386eff7198858846f6d89675fda/pyboreas/data/splits.py#L42-L56


Improve "Migrating Configuration" with descripting of all common cases

Configuration is a big topic. In a common web app on old ASP.NET we'd have roughly the following stuff in web.config:

appSettings with simple key-value settings of the app and libs
standard sections from .NET like system.diagnostics
settings for asp.net - system.web
settings for web server - system.webServer
custom sections - sections of 3rd party registered in configSections element (please pay attention that these libs can be not related with asp.net at all, just some libs)

The article only talks about the first one - appSettings. Yeah, they are easily converted to anything as it's just key-value pair.
But what to do with the others?
The most unclear thing for me is custom sections. Just imagine you used some library. And that library supports configuration - it uses ConfigurationManager which just uses "system configuration file". And now it's empty. We could say something like "do not expect the lib to configure itself from configuration file, configure it on your own via its api". It makes sense indeed. But it breaks my app. It breaks any app. Moreover the lib could not support the same abilities via Api as via config.
I'm not sure that here is a good place to discus the design "deprecating System.Configuration". But anyway consequences of that design should be clearly stated in the docs.
Thanks.

+1

Here's a use-case close to MS. Right now I try to use ASP.NET 5 on top of .net 4.5.2 but with EF6, since EF7 is looking to not be production-ready for a long while. Try to use and Oracle provider.
Now you're in a very bad place. EF6 insists on using DbProviderFactories and the classes that could reconfigure that are internal. DbProviderFactories cannot really be registered any other way than with a section from ConfigurationManager. Ouch.
This is an example, coming right from MS, that some libraries really need to fall back to ConfigurationManager in clr runtime.

There's really 2 questions here with deepest to migration, The .NET Framework story and the .NET Core story. For .NET Framework, we'll try to be as compatible as possible so the libraries "just work". For .NET core libraries can't depend on configuration manager because there is none. They'll need to resort to their own configuration file,  or better, take IConfiguration in their APIs.

@davidfowl Totally agree. For Core libs need to be adapted and since ConfigurationManager is not there they have to fallback to IConfiguration or whatever.
But for the full Framework the story is not great today. I had to rely on a dirty hack using private reflection to reset the internal ConfigurationManager state and load web.config (as I said, the goal was to configure an Oracle DbProvider). If there is a better way let us know!

I just filed this bug https://github.com/aspnet/dnx/issues/2194

Thanks for contacting us.
We don’t have the resources to invest in this area, so we are closing the issue. Should your request generate enough  👍 responses, we’ll reconsider.


About RGB888

Here is not an issue or bug.
In JPEGDEC i only could see code about RGB565, and whether support RGB888 in future?

It would not be difficult to add support for RGB888, but it will take time. My purpose in writing JPEGDEC was for microcontrollers and RGB565 SPI LCD displays. I don't have spare time to do the work right now, but it is not difficult. I can only ask you to make the change and then do a pull request.


actions: manifest: Update to include fix to impostor commits.

More details in:
https://github.com/zephyrproject-rtos/action-manifest/pull/13

(dnm since it points to a PR head right now)

DNM removed, now pointing to https://github.com/zephyrproject-rtos/action-manifest/releases/tag/v1.3.1

hotfix?


Typescript types for ClientPrincipal

It would be helpful if typescript types were provided in some way for the Client Principal Data

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 89705f24-082e-d4e9-547c-1992db4470cb
Version Independent ID: 5df6b7d0-7e8a-f6c5-ea28-289b9a0ca84e
Content: Accessing user information in Azure Static Web Apps
Content Source: articles/static-web-apps/user-information.md
Service: static-web-apps
GitHub Login: @craigshoemaker
Microsoft Alias: cshoe

@njrardin
Thanks for your feedback! We will investigate and update as appropriate.

Thanks for the feedback @njrardin. Seeing that this doc only includes JavaScript and C#, TypeScript may be beyond the scope of this document, which is simply providing guidance on how one could access user info in a Static Web App.
Beyond that, TypeScript type references can be found at Reference | Microsoft Learn. Regarding ./auth/me I'm not sure offhand which type specifically is returned but I do believe it's one of these types.
In the meantime, I'll assigned the issue to content author so that they can make an assessment to see if either changing JavaScript to TypeScript or adding TypeScript is prudent.

@njrardin I've discussed with the content author, and this is feedback we'll add to the backlog.

Thanks for your feedback and your contribution to Azure docs.
Feedback for this repository is moving away from GitHub to a system specific to the Microsoft Learn platform. Issues for this repository will soon be disabled, and additional comments from GitHub will no longer be possible. However, we are now tracking and triaging this issue in the new feedback system.
To learn more about our feedback systems, please see Provide feedback for Microsoft Learn content.
#please-close


Pandas 2.0?

Scivision currently depends on pandas~=1.5 - can we relax this to support pandas 2.0?

https://pandas.pydata.org/docs/dev/whatsnew/v2.0.0.html


Update clusterapi

Closes: https://github.com/networkservicemesh/integration-k8s-packet/issues/398

we know that vl3 tests are unstable due to rework with ipam logic. So failure doesn't related to this PR. Merging.


Add Pastbin "Password" and "Burn after read" 😄

Hey everyone,
My first question, does pastebin-api work under javascript?
And, can you add the two new functions of Pastebin if their API allows it?

Thanks for everything!

My first question, does pastebin-api work under javascript?

Yes, ofcourse!

And, can you add the two new functions of Pastebin if their API allows it?
image

Last time I checked these weren't documented yet, but I can take another look today.

Looks like it's not supported by their API. The password field isn't documented either.
"Bad API request, Burn After Read paste can not be created via API"


RESTARTS count error when "get pods --watch"

I use /hack/local-up-cluster to start a local k8s.  Here is result:
./cluster/kubectl.sh get pods --watch
NAME             READY     STATUS    RESTARTS   AGE
my-nginx-kn1zy   0/1       Running   0          6s
my-nginx-msjcr   1/1       Running   0          6s
NAME             READY     STATUS    RESTARTS   AGE
my-nginx-kn1zy   1/1       Running   0          10s
my-nginx-kn1zy   0/1       Running   1         51s
my-nginx-kn1zy   1/1       Running   1         1m
my-nginx-kn1zy   0/1       Running   2         1m
my-nginx-kn1zy   1/1       Running   2         1m
my-nginx-kn1zy   0/1       Running   3         3m
my-nginx-kn1zy   1/1       Running   3         3m
my-nginx-kn1zy   1/1       Running   2         4m

The "RESTARTS" is supposed to be 3. But it will be changed to 2 at last . Is there reason which will cause RESTARTS reduce?

Thanks for reporting the issue. There has been a regression of restart count recently. We are planning to fix this shortly.
Closing this as a duplicate of #11948


Implement simple DNS based discovery.

What change does this introduce?
This PR implements "simple" DNS based discovery.  This is not "DNS-SD", but rather a mechanism that queries a single host and treats each address in the result as a separate target.
Why make this change?
This is part of migrating the discovery system from dogmatiq/configkit.  This particular discoverer is useful under Docker: It is possible to give multiple separate services the same alias on a network. Performing a DNS query against that alias name returns the addresses of all of the containers running across all of those services.
Is there anything you are unsure about?
No
What issues does this relate to?
Partially addresses #1
Based on the branch of PR #7

Sorry, I jumped the gun on this one - it should still be a draft because it depends on #7 being merged first.

Thanks man.


Consider reverting "Simplify DateTime api"

In many cases I (and I guess many other users, too), operate on UNIX timestamps, only converting the ts some local time zone to present the date to a user. It's useful to have a tinier UTC-only type for this use case. So at least in my opinion the API was better before 7e38724f66fed3b370e8a5763a3fca15ef56f000.

Fixed in 2cf0a22.


fix run-and-move-down error (fixes #16)

Fixes #16.
Changed an includes-statement to an actual comparison.

@mbroedl
This includes all your previous commits. I think you might need to rebase from master.

@kylebarron Thanks! I just realised that, but did not know how to do that so I faffed around a bit.
Would this be fine or should I delete the PR and make a new one after having rebased? :/

I'm not completely sure... This can be a tricky part of git (you'll find a lot of results when googling git rebase help haha).
Especially since your changes are tiny, you can basically just do:
git clone https://github.com/nikitakit/hydrogen-python
cd hydrogen-python

Then make your changes again, then do
git remote add mbroedl https://github.com/mbroedl/hydrogen-python
git push mbroedl master -f

Note that the last command will overwrite the history on your master branch on Github. Alternatively, you could create a new branch, push the new change to a new branch of your repository on Github, and then make a new PR from the new branch.
I hope that makes sense?


Drop BaseRef

...and just use Ref
Reviewer checklist

[ ] Ensure relevant issues are linked (description should include text like "Fixes #")
[ ] Ensure any appropriate documentation has been added or amended

Coverage remained the same at 98.292% when pulling d09dec608c7a13b69acc8edd76d48b0b8c61a557 on ref into 656a02b53ed661fa1894b7222da39ec2e591876d on main.


Official package for Meteor

Hi, we are interested in loading Knwl.js into our project:
https://github.com/RocketChat/Rocket.Chat
It is a cool chat app built using www.meteor.com so I was looking for a package and only found the following one:
https://github.com/Agnito/meteor-knwl/
Thats says: "This project is no longer recieving updates from the original package creator, Load Five."
Soooo.. we've already helped other cool js libs to add the /meteor directory with the right files so the project is automatically available as an official package at https://atmospherejs.com
Would you guys be willing to accept a Pull Request for that?

+1

+1

+1 :)

+1

@engelgabriel are you aware of retext? Seems more alive than this.

+1

Loadfive seems to be either dead or in a koma without a farewell note and no contact information. Does anybody have an idea on how to reach them? Would anyone be up to fork and continue this repository?

@FranzSkuffka I'm sorry to all for the lack of involvement recently. LoadFive is a project of mine in itself. Still being in school, I do this as a side project in my free time. Loadfive, in fact, is neither dead or in a coma; however, I apologize for the absence of updates in recent months. If you would like to contact me personally, my email is<EMAIL_ADDRESS>:-)
-- Ben

thanks for the update @loadfive

@loadfive if you need a maintainer, you can add me.


Silence Xcode 6.3 project settings warning

Xcode 6.3 shows a warning when you open the MMMarkdown project. This PR removes this warning by letting Xcode update the project settings.

Thanks for opening a pull request! :sparkles: