Pressing Return after changing value in the Inspector causes a node to enter Rename mode. macOS

Godot version
4.0 beta 12 and 13
System information
macOS 13 Ventura, Macbook Pro 2019, 16-inch
Issue description
When I press the return key after making changes in the Inspector panel, it causes the current note in the Scene tree to enter Rename mode.
Steps to reproduce
Select the node and in the Inspector panel change any value, such as scale or size, and press return.
Minimal reproduction project
N/A

I can't seem to reproduce the issue on Linux, might be macOS specific. CC @bruvzg

Duplicates:  #71605 #69160
For Linux the problematic behavior was a bit different and got fixed in #69234

@JohnCWakley since this was closed as a duplicate, it is very likely, that your comment will be forgotten. May I suggest, that you create a new issue for the problem, that you have identified.


queries with order by have no effect on the chart

I'm using a bar chart, because I want my chart to be ordered by month so I added orders by month in my query but the result doesn't have any effect.
this is my result

and this is result of sql lab

yup, order by doesn't work in nvd3 bar chart...not sure that's a historical issue or regression(unlikely)..
We won't be fixing it at this point since Echarts Bar will be available soon, but a quick fix PR is still welcome.


Fix pyconvert cache update bugs by clearing the whole cache after any rule is added

Fixes #364. Fixes #336.
It would be possible to do a more sophisticated partial cache clear, but IMO that would be premature optimization at this point.

Thank you!


Feature: Schema Registry support

Some Kafka libraries like confluent-kafka-python,  https://github.com/marcosschroh/python-schema-registry-client and https://github.com/lsst-sqre/kafkit provide various schema registry integration capabilities (HTTP client for schema registry, caching schema registry requests, serialization/deserialization, Avro, Protobuf, JSON-schema support)
It would be nice to have such capabilities built-in with declarative style API.
Further more, I suggest providing both schema-first (use model.avro files) and code-first implementations (generate and register avro schema from Pydantic models or Dataclasses using https://github.com/marcosschroh/dataclasses-avroschema) of schema registry integration.
Using default code example, I envision the following usage pattern.
from fastapi import FastAPI
from dataclasses_avroschema.avrodantic import AvroBaseModel

from faststream.kafka.fastapi import KafkaRouter
from faststream.kafka.schema_registry import SchemaRegistryConfig

router = KafkaRouter(
"localhost:9092",
schema_registry_config=SchemaRegistryConfig(
    format="avro",
    subject_naming_strategy="topic-key-value",
    schema_storage_strategy="from-pydantic-model",
    url="localhost:8081"
))

class Incoming(AvroBaseModel):
    m: dict

@router.subscriber("test")
@router.publisher("response")
async def hello(m: Incoming):
    return {"response": "Hello, world!"}

app = FastAPI(lifespan=router.lifespan_context)
app.include_router(router)

Describe alternatives you've considered
Each application may try to implement it's own integration with schema registry

I am not a big Kafka specialist, but your code example with SchemaRegistryConfig looks great. @sternakt what do you think about this feature?

We are currently creating a new Kafka broker based on the Confluent library. We'll look into this as soon as the new broker support is finished.

@devova can we close this issue? Is it working for you?

Opening a new issue https://github.com/airtai/faststream/issues/1297 and closing this one.


Update blade directives registration

Came across an issue where the directives are all rendered as strings. After searching in recent changes came to the conclusion that https://github.com/spatie/laravel-ray has caused this.
Seems like https://github.com/spatie/laravel-ray/blob/main/src/RayServiceProvider.php#L226 uses a much cleaner method to check if the blade.compiler is resolved then my previous fix in PR.

WIP

They've reverted it to afterResolving.

My project has this same issue. I'm not using spatie's ray package. But unless i use @fridzema's branch, all the blade directives output as strings. @fridzema Did you ever get this figured out?

Hi, same here.
We are during a huge update (PHP & Laravel) and blade directives stopped working.

@fridzema's change makes our application work again.


Separate into Opal bootstrap and loader?

I've been wondering if this loader is really doing 2 things: presenting a "bootstrapped" version of Opal in Node and serving as webpack loader. What if this was split into 2 packages

Opal NPM - could also be called Opal bootstrap. This would include the compiler.rb patches, bundlerCheck/Execute, getRubyMetadata, and the getOpal*Filename.js files

Webpack loader - everything else

This would make it easier to implement #7 because a separate plugin could be developed that could modify webpack's "load path" but take advantage of the Opal NPM code above just like the loader does. No urgency here, but any thoughts @cj  ?

@cj - any opinion on who "owns" the new project (github and NPM) ?

Also, looks like https://github.com/anthonny/opal-npm-wrapper is out there, although there are some things that would need tweaking with that.

I might raise an issue and ask how willing they would be to cover the cases the code in the repo does.

any opinion on who "owns" the new project (github and NPM) ?
I don't mind :)

Interesting.  Let me know if you get a hold of him.  Should we just fork it for now?

No urgent need for me to do anything, I'm content to wait and see how it plays out personally.


weird distortion at corners - joystick application

Trying to design a joystick using the MLX90393.
Using the code in this library, I converted core code to Python/raspberry pi.
Communicating with the device using SPI, I can get a x/y angle of +/-30degrees from joystick tilt.
Problem is with the corners, the angle gets reported larger than 30degrees, which creates spikes outwards.
Does anyone know why this happens?

Image of the corner issue
https://drive.google.com/open?id=10TBIbMajsiFhMsbUNz5z_pK-BCS_CI4a

Hmm.  You'll have to provide a little more info.  What does your joystick design look like, and why do you think the resulting data should be square?

I wrote a quick simulation of your setup, and I'm pretty sure this distortion comes from the curvature of the field around the disk magnet.  You can see a video of the simulation here:
magnet joystick simulation video
Here's a single frame.  The red and blue dots represent a collection of magnetic dipoles to simulate the disk magnet.  The sensor is at the black dot on top.

You can see the curved field away from the axis of the magnet.  That seems to be the problem.  If I re-run the simulation with a uniform magnetic field instead, the distortion goes away:
joystick with uniform field video

Two things come to mind.  First, this distortion looks very similar to pincushion distortion caused by some lens systems.  See the wikipedia article for more info.  You may be able to adequately correct the response using a model for the radial distortion as explained in the article.  You could either choose coefficients by trial and error, or use your collected data to calibrate the joystick.
Also, if you are willing to change the design, and you really need a linear response, you could use a Helholtz (or Maxwell) coil to generate a uniform field.  In this case, I would probably mount the sensor on the joystick and keep the field stationary.  This solution would consume more power, since you'd need to energize the coil to create the field, but should eliminate most of the distortion.
I wrote the simulation in python.  I'll probably clean it up and release it as a more general package over time, but let me know if you want to play with it sooner.

Update: it's not because the field lines are curved, it's because of the way the extended geometry of the disk magnet curves the field lines.  If you use a single, tiny dipole as the magnet, the output is linear, even though the field lines are all circles.  You can show this on paper fairly easily for the field from a dipole.
I think the answer is that for any actual magnet, you'll need to calibrate the joystick to produce a linear output, if that's what you need.


Copy @douglm's information into developers block

@douglm any objections to just copying what you've got in the contributors section?

Nonee
On 10/15/15 22:33, Nicholas Blair wrote:

@douglm https://github.com/douglm any objections to just copying what
you've got in the contributors section?

    You can view, comment on, or merge this pull request online at:

https://github.com/Bedework/bw-ical4j-cl/pull/10
    Commit Summary

Copy @douglm's information into developers block
File Changes

M pom.xml
https://github.com/Bedework/bw-ical4j-cl/pull/10/files#diff-0 (9)
Patch Links:

https://github.com/Bedework/bw-ical4j-cl/pull/10.patch

https://github.com/Bedework/bw-ical4j-cl/pull/10.diff

—
Reply to this email directly or view it on GitHub
https://github.com/Bedework/bw-ical4j-cl/pull/10.

Cool, 1.1.1 release being built right now.


False positive reporting - messing up originating languages

So whilst trying out this project I quickly ran into a scenario where it is reporting false positives.
To reproduce:

Create a project in Python using Poetry
Add boto3 as a dependency
Generate the necessary poetry.lock

This poetry.lock file will have the following:
[[package]]
name = "boto3"
version = "1.28.15"
description = "The AWS SDK for Python"
optional = false
python-versions = ">= 3.7"
files = [
    {file = "boto3-1.28.15-py3-none-any.whl", hash = "sha256:84b7952858e9319968b0348d9894a91a6bb5f31e81a45c68044d040a12362abe"},
    {file = "boto3-1.28.15.tar.gz", hash = "sha256:a6e711e0b6960c3a5b789bd30c5a18eea7263f2a59fc07f85efa5e04804e49d2"},
]

[package.dependencies]
botocore = ">=1.31.15,<1.32.0"
jmespath = ">=0.7.1,<2.0.0"
s3transfer = ">=0.6.0,<0.7.0"

This will result in the following output from depscan:
║<EMAIL_ADDRESS>⬅ CVE-2022-32511    │ 🧾 Vendor Confirmed │ 1.6.1       │ CRITICAL │   9.8 ║

Which is unfortunately for the Ruby dependency, not the Python one. Seeing how the SBOM includes clear pkg:pypi PURLs and references, I wonder if this information can be used to separate the CVEs on originating languages. Otherwise this will be a major time waster in projects that have multiple languages in the same repository or when people are scanning an entire organisation.

@ashemedai Thank you for a reproducible test case. We will investigate and update here.

We have a similar issue with many vulns - here are some examples
<EMAIL_ADDRESS>-> all CVEs related to K8S
<EMAIL_ADDRESS>-> all CVEs related to ES
<EMAIL_ADDRESS>->
    * CVE-2022-28948
    * CVE-2021-4235 (Go)
    * CVE-2022-3064 (Go)
*<EMAIL_ADDRESS>->
    * CVE-2018-1002207 (Go version)
    * CVE-2019-10743
*<EMAIL_ADDRESS>-> CVE-2023-34617 (java)
*<EMAIL_ADDRESS>-> CVE-2021-20291 (Go version)
*<EMAIL_ADDRESS>-> CVE-2017-18589 (Rust)

Do you have an ETA for fixing this?
Thanks!

@bh86 @yossipik we might need a way to reproduce this. Could you join our discord and share a sample project or sbom?
https://discord.gg/pF4BYWEJcS

So the issue seems to only happen when the project type is not specified. I am working on a fix for this regardless given some projects have multiple dependency types.

So the issue seems to only happen when the project type is not specified. I am working on a fix for this regardless given some projects have multiple dependency types.

Thanks Caroline! Indeed, I don't specify any project type because usually my sbom file contains multiple package types.
By looking at the source code, it seems the project type argument is somehow mandatory to get correct results. My assumption was that you extract the project type from the purl.

@ashemedai @bh86 @yossipik I've put out a fix on the linked branch. Please test it out and let me know how it goes!

Please retest with the 4.2.1 release.

works! thanks for fixing it

Thank you for the confirmation! Please share your love on social media.


Stop button in Chart view doesn't stop async queries

When an async query (GLOBAL_ASYNC_QUERIES=True) is run in Chart view, clicking "Stop" button doesn't have any effect. Clicking Stop button, closing tab, refreshing page or going to other page leaves the query running in a database.
How to reproduce the bug

Go to 'Chart' view
Click on 'Update Chart' to execute query
Click on 'Stop' to stop query

Expected results
Running query is cancelled in a database.
Actual results
Nothing happens.
Environment

browser type and version:
superset version: 2.1.0
python version: 3.8.13
node.js version: 16.9.1
any feature flags active: GLOBAL_ASYNC_QUERIES=True

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[ ] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context
Clicking "Stop" button aborts fetch request, but that doesn't cancel running query in a database. With GLOBAL_ASYNC_QUERIES enabled that request is very fast, so in most cases there's no even request to abort.
To stop a running async query we need to terminate Celery task and cancel query in a database.

Yeah, I met the same issue. When the PR could got merged?

I'm hopeful that @ganczarek will come back to re-base and help merge https://github.com/apache/superset/pull/24905 - but if not, anyone else is welcome to adopt and credit their work, or open a PR with a different approach.
This was reported as a 2.x issue, but we're closing those out as we only support 3.x and newer. Please let us know if this is still happening in current versions.

This seems to have gone stale, as we no longer even support 3.x. Happy to revisit/reopen if this is still an issue in 4.x


Request: New discord status (watching and listening)

Add discord listening and watching status for bots.

That's implemented in 4.0. Currently there are no plans to release 3.3.

You aren't D#+ Developer.

I'm the most active maintainer, and the one that implemented this in 4.0. 3.x is on feature freeze unless we decide to release 3.3.

"You aren't D#+ Developer."

Boi

Ofc. Those were clearly faked.

That's implemented in 4.0. Currently there are no plans to release 3.3.

@Emzi0767 Collaborator != Project Owner.

@uwx i don't asked nothing for you, don't intrudes where it is not called.

I mean, emzi's a bit more than a "collaborator" at this point, and anyway, why would you need specifically the project owner?

This isn't a place for shitposting.


Improve Person links

At the moment a person can have 3 links:

website
twitter
github

That is ok, however, if we have some time to optimize this it would be a great addition if we could make this more dynamic: Something like a label and url would be a possible solution.
i.e. Some guests don't have a website, but only a linkedin site ...

looks reasonable

Or what about just reading it into a Map<String, String> with the key as
name that will be and always provide full URL as value
Artur Dryomov<EMAIL_ADDRESS>schrieb am Di., 31. Juli 2018,
08:19:

Yes, I’ve been thinking about this as well. I think we can change YAML
structure a bit.

id: artur.dryomov
name: Artur Dryomov
twitter: arturdryomov
github: ming13
site:
name: "blog"
url: "https://arturdryomov.online"

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/TheContext/podcast-ci/issues/16#issuecomment-409108038,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAjnrs6oIz32gPW9CAckjDRn2EX89HqPks5uL_b-gaJpZM4Va9Yo
.


Picker bindings don't work when id: \.self

As reported by our user @rbartolome, the binding in this code doesn't work:
struct PickerTest: View {
  @State private var selection: String = ""
  let items: [String] = ["Item 1",
                         "Item 2",
                         "Item 3",
                         "Item 4"]
  var body: some View {
    VStack {
      Picker("MyPicker:", selection: self.$selection) {
        ForEach(self.items, id: \.self) {
          Text($0)
        }
      }
      Text("You selected: \(self.selection)")
    }
  }
}

I've been able to narrow it down to our Picker implementation assuming that SelectionValue generic parameter is always an integer. The assumption is obviously broken in this case, where it is a string.

@rbartolome I'm sorry, but fixing this will require signifcant changes to our Picker implementation, and I'm working on it right now. I would advise you to use an Int binding to array indices in the meantime, as we do in our PickerDemo. Would that be a suitable workaround for you?

@MaxDesiatov The PickerDemo makes no pre-selection. Try set the selection property to 1. It should show the 2nd element in Picker but it shows the default text "Pick a text style..."

Here's another simple example that doesn't update the value correctly, which doesn't use ForEach at all:
import TokamakDOM

@main
struct TokamakApp: App {
  @State var selection: String = "None"

  var body: some Scene {
    WindowGroup("Tokamak App") {
      VStack {
        Text(selection)
        Picker(selection: $selection, label: Text("Picker")) {
          Text("A")
          Text("B")
          Text("C")
        }
      }
    }
  }
}


New Jupyter Notebooks Section: MPAS Plot with Datashader and Geoviews

This PR adds @clyne 's Jupyter notebook demonstration of interactive (pan, zoom, etc.) MPAS plotting with Datashader and Holoviews-Geoviews packages.
See here to explore how the output of this PR looks like.
@clyne @anissa111 @michaelavs and others, please let me know what you think, if you find anything missing/wrong with this draft PR, etc.
Due to the nature of this development, there are significant additions to GeoCAT-examples by this PR. Below list shows those as a checklist where the checked off items are currently implemented in this draft PR, but the PR should better be merged with all/most of them checked:

[x] Add "nbsphinx" dependency to make galleries directly out of Jupyter notebooks (for now, those that are pre-executed ) just like sphinx-gallery does with executing .py files

See conda_environment.yml and conf.py for details

[x] Add a new page "Notebook Gallery" to the same level with "Gallery" or "GeoCAT-comp examples" in GeoCAT-examples gallery
- See docs/index.rst and docs/gallery_nb.rst

[x] Add the first Jupyter notebook example "docs/gallery-notebooks/Datashader/MPAS_Datashader_Trimesh.ipynb"
- It reads in 30km DYAMOND MPAS data from geocat-datafiles

[ ] Modify installation.rst to describe how readers can execute Jupyter notebooks examples in their local

[ ] Explore Dask parallelization for this notebook on 3.75 km dyamond_1 dataset (@anissa111 I might ask your help on this as my first attempt didn't end up with better performance than the current implementation). This may be done after merging this PR though.

[ ] Figure out how we can read-in much bigger datasets. This is not the scope of this PR but just for future reference.

[ ] Apply dynamically resizing margins in the document pages (what @erinlincoln did for GeoCAT-viz or @hCraker did for GeoCAT-comp) so that we can show larger plots, and the entire content could dynamically adjust to the screen size. See GeoCAt-comp docs page for example. (I intentionally didn't implement this right away here to allow reviewers to easily compare new docs pages with the original docs, but it is already implemented in my local). I don't know if we intentionally didn't apply tht to GeoCA-examples yet?

As a minor implementation, this PR also adds the NSF logo to the doc homepage.

RE: dynamically resizing margins
Hi @erogluorhan. It was not intentional to leave GeoCAT-examples as is. @erinlincoln and I made the improvements for -viz and -comp as we were working on those repo, but we never got around to doing the same for -examples. I believe we can make this improvement in -examples without any issue.

What you have so far looks good! Please ignore my review as it will be stale soon I imagine.

Explore Dask parallelization for this notebook on 3.75 km dyamond_1 dataset (@anissa111 I might ask your help on this as my first attempt didn't end up with better performance than the current implementation). This may be done after merging this PR though.

I struggled with getting much benefit Dask either. However, if I monitored the process table while the notebooks was running there were times when multiple cores were clearly active. This might be an issue to follow up on with Jim Bednar. Jim's team were instrumental in helping me get the first version of this notebook together, and it might be a good idea to get their input.

This is awesome, @erogluorhan! Thanks for putting this together.
A few general comments:

It would be nice to add a download link for the notebook if possible
Maybe consider changing the image width height to something on the
order of 1200x600. There is a lot of detail in this data and it is
kind of hard to appreciate at 600x300.
There is a comment under "Load data and coordinates" about figuring
out how to provide access to larger data. I'm not sure that this
is entirely necessary. The example certainly demonstrates how to
plot an MPAS data set with the 30km data. Using higher resolution
data is simply a matter of changing the path.
For the future you might consider something like nbautoexport to facilitate
PR reviews of notebooks.

Typos:
"is note used" -> "is not used"
"splits a global" -> "splits a global mesh"
"optimized performance" -> "optimized machine code"
"resolution date" -> "resolution data"

This is great input; thanks a lot @clyne! Pleas find my responses below to the general comments:

Yes, this is something I attempted to have but couldn't get it done with nbsphinx yet. For now, I have put this into the checklist of this PR not to forget before merge.
Yes, I agree, and indeed I wanted to do that but the current narrow margin of the page hadn't allowed me to make plots bigger and look fine. Now, I pushed the dynamic resizing feature for the pages, so I also made plots 1200x600. Please check the newly generated RTD page.
Rephrased.
I'd need to explore nbautoexport a bit. Thanks a lot for the recommendation!

Typos are all fixed. Thanks for catching these!

Explore Dask parallelization for this notebook on 3.75 km dyamond_1 dataset (@anissa111 I might ask your help on this as my first attempt didn't end up with better performance than the current implementation). This may be done after merging this PR though.

I struggled with getting much benefit Dask either. However, if I monitored the process table while the notebooks was running there were times when multiple cores were clearly active. This might be an issue to follow up on with Jim Bednar. Jim's team were instrumental in helping me get the first version of this notebook together, and it might be a good idea to get their input.

Indeed! I am hoping to get some help from @anissa111 's Dask knowledge on this. (FYI: In my own experience, the execution of Example 1's rasterize block on 3.75 km resolution data took 1 minute in my local and 2:30 minutes with Dask LocalCluster I later commented out).
After discussing this with Anissa, I'd further discuss with Jim et al.

@anissa111 If you are planning to give this PR a review, please make sure to review the latest changes on the

INSTALLATION.md and instal.rst files to reflect the new installation instructions for Jupyter notebooks
The renaming of the "\Plots" folder to "\Gallery" (Ths refactoring added several py files to this PR as if they are changed, but nothing's changed in them actually). This renaming could have been left to a new PR, but the modification in INSTALLATION.md triggered me to address it here.

If/oce you approve this PR, I am planning to merge it as is and create new issue(s) for the items that were not checked off in the description (Binder, etc).


Add support for iterating over lists with a generator.

This requires a thorough review. I spotted two bugs in which the get_all url didn't match the other url. The bugs were that the one url didn't use the plural version.

Looking at this set of changes, it looks like a great addition. I see you did make some stylistic changes with how you handled using the _build_path() calls by making the call as a variable assignment when it has additional arguments outside of the attributes on the main class, which I believe would be very useful on all endpoint calls, and not just calls to the all() endpoint methods. I do see what appears to be a slight difference between the README.md and README.rst information in at least the Pagination section as well. Both of these changes could probably be additional expansions made later. Have any of these endpoint changes and updates been tested on an account that has at least 2 pages of data according to the default counts and offsets for the endpoints changed?

Ah, sorry about the stylistic changes. I did my best to stay consistent, but I had to make some changes to make it easier to implement the changes.
I did test all three variations with the lists API endpoint with my data. I'll be using this for one of my projects. I tested the pagination logic, but had to change the page_size as I don't have 1000 records for any of my data.


Refactoring to support net.PacketConn and dialling

This makes a few changes:

Adds a new ListenOn function which takes an existing net.PacketConn as an argument rather than a UDP endpoint address
Adds a new Dial function to the Endpoint which allows creating a new outbound connection on the existing net.PacketConn
Changes listener to an exported Endpoint type, which is now returned by Listen and ListenOn (which still conforms to net.Listener so the return type changing won't break existing code)
Adds a unit test for the above

The goal being that the package now can operate pretty much agnostically on anything that conforms to the net.PacketConn interface, like Yggdrasil and Pinecone overlay networks, QUIC connections etc.

Why is the new Endpoint type needed?  Why not have the client check whether a given net.Listener satisfies the Dialer interface?

type dialer interface {
    Dial ...
}

l, err := Listen(...)
d := l.(dialer)
d.Dial(...)

What happens if a client does

c, err := l.Dial(a)
d, err := l.Dial(a)
c.Close()
d.Close()

Double close, right?  Is that the expected behaviour?


Asseration error while reading TBranchElement

Hi,
I encounter the same error as in this earlier issue (https://github.com/scikit-hep/uproot/issues/198) but the proposed workaround does not work in this situation. And maybe this is caused by something entirely different. So I'm opening a new issue here.
Here is a minimal example of the issue.
> import uproot
> file = uproot.open("09D95BE7-23C8-5542-9FA4-102E3FEDB7A2.root")
> file["Events"].array("SimVertexs_g4SimHits__SIM.obj")
---------------------------------------------------------------------------

AssertionError                            Traceback (most recent call last)
<ipython-input-190-4eb779d53508> in <module>()
----> 1 file["Events"].array("SimVertexs_g4SimHits__SIM.obj")

/home/thomaskeck/venc/local/lib/python2.7/site-packages/uproot/tree.pyc in array(self, branch, interpretation, entrystart, entrystop, flatten, flatname, awkwardlib, cache, basketcache, keycache, executor, blocking)
    440
    441     def array(self, branch, interpretation=None, entrystart=None, entrystop=None, flatten=False, flatname=None, awkwardlib=None, cache=None, basketcache=None, keycache=None, executor=None, blocking=True):
--> 442         return self.get(branch).array(interpretation=interpretation, entrystart=entrystart, entrystop=entrystop, flatten=flatten, flatname=flatname, awkwardlib=awkwardlib, cache=cache, basketcache=basketcache, keycache=keycache, executor=executor, blocking=blocking)
    443
    444     def arrays(self, branches=None, outputtype=dict, namedecode=None, entrystart=None, entrystop=None, flatten=False, flatname=None, awkwardlib=None, cache=None, basketcache=None, keycache=None, executor=None, blocking=True):

/home/thomaskeck/venc/local/lib/python2.7/site-packages/uproot/tree.pyc in array(self, interpretation, entrystart, entrystop, flatten, flatname, awkwardlib, cache, basketcache, keycache, executor, blocking)
   1292             keycache = {}
   1293
-> 1294         basket_itemoffset = self._basket_itemoffset(interpretation, basketstart, basketstop, keycache)
   1295         basket_entryoffset = self._basket_entryoffset(basketstart, basketstop)
   1296

/home/thomaskeck/venc/local/lib/python2.7/site-packages/uproot/tree.pyc in _basket_itemoffset(self, interpretation, basketstart, basketstop, keycache)
   1249         for j, key in enumerate(self._threadsafe_iterate_keys(keycache, True, basketstart, basketstop)):
   1250             i = basketstart + j
-> 1251             numitems = interpretation.numitems(key.border, self.basket_numentries(i))
   1252             basket_itemoffset.append(basket_itemoffset[-1] + numitems)
   1253         return basket_itemoffset

/home/thomaskeck/venc/local/lib/python2.7/site-packages/uproot/interp/jagged.pyc in numitems(self, numbytes, numentries)
     86
     87     def numitems(self, numbytes, numentries):
---> 88         return self.content.numitems(numbytes - numentries * self.skipbytes, numentries)
     89
     90     def source_numitems(self, source):

/home/thomaskeck/venc/local/lib/python2.7/site-packages/uproot/interp/objects.pyc in numitems(self, numbytes, numentries)
    140
    141     def numitems(self, numbytes, numentries):
--> 142         return self.content.numitems(numbytes, numentries)
    143
    144     def source_numitems(self, source):

/home/thomaskeck/venc/local/lib/python2.7/site-packages/uproot/interp/numerical.pyc in numitems(self, numbytes, numentries)
    170         dtype, shape = _dtypeshape(self.fromdtype)
    171         quotient, remainder = divmod(numbytes, dtype.itemsize)
--> 172         assert remainder == 0
    173         return quotient
    174

AssertionError:

What makes this different from the previous issue is that the interpretation is a jaggedarray
> file["Events"]["SimVertexs_g4SimHits__SIM.obj"].interpretation
asjagged(astable(asdtype("[('id_', '>u4'), ('fX', '>f8'), ('fY', '>f8'), ('fZ', '>f8'), ('fT', '>f8'), ('itrack', '>i4'), ('vtxId', '>u4'), ('procType', '>u4')]")), 10)

I'm not familiar with the ROOT format, but after some investigation it seems like the basket contains an extra 49 bytes (maybe a class header?) in front of the entire table (not in front of each element, hence the previous workaround does not work).
I'm using uproot version 3.4.5, uproot-methods version 0.4.2 and python 2.7.
The file can be found here (2.9GB!):
https://eospublichttp.cern.ch/eos/opendata/cms/MonteCarloUpgrade/RunIIAutumn18DR/TTToHadronic_TuneCP5_13TeV-powheg-pythia8/GEN-SIM-RAW/PUAvg50IdealConditions_102X_upgrade2018_design_v9-v1/110000/09D95BE7-23C8-5542-9FA4-102E3FEDB7A2.root

Yes, the workaround in #198 is peculiar to the structure of that file. My failure there was not finding a way to get uproot to detect the layout from the metadata it provides; your file presumably provides different metadata for this different layout. I'll take a look as soon as I can.

I've taken a look at the data and I'm thoroughly perplexed. If the basket data just had an excess 49 bytes at the beginning, we'd be able to take care of that easily (assuming that we aren't demanding uproot to have discovered this—49 bytes is a lot of extra header!):
b = file["Events"]["SimVertexs_g4SimHits__SIM.obj"]
reinterpret = uproot.asjagged(b.interpretation.content, 49)   # or 49 + 10
b.array(reinterpret)

The asjagged interpretation has a skipbytes parameter (uproot believes it should be 10 in your case, as it is in all the examples of STL vectors I've been able to generate), so I just unwrapped the original asjagged and created a new one with the bytes skipped. That doesn't help.
But just looking at the raw data, it doesn't look like there's anything there. asdebug is a special interpretation that I use to diagnose these things: it's asjagged(asdtype("uint8")), using knowledge of the entry/event boundaries from ROOT (asjagged) but displaying only the raw bytes in each entry/event. The first 5550 bytes are extremely regular:
>>> b.array(uproot.asdebug)[0][:500]
array([ 64,   0, 160,  44,  64,   9,   0,  12,   0,   0,   2,  42,  64,
         0,   0,   6,   0,  10,   0,   0,   0,   0,  64,   0,   0,   6,
         0,  10,   0,   0,   0,   0,  64,   0,   0,   6,   0,  10,   0,
         0,   0,   0,  64,   0,   0,   6,   0,  10,   0,   0,   0,   0,
        64,   0,   0,   6,   0,  10,   0,   0,   0,   0,  64,   0,   0,
         6,   0,  10,   0,   0,   0,   0,  64,   0,   0,   6,   0,  10,
         0,   0,   0,   0,  64,   0,   0,   6,   0,  10,   0,   0,   0,
         0,  64,   0,   0,   6,   0,  10,   0,   0,   0,   0,  64,   0,
         0,   6,   0,  10,   0,   0,   0,   0,  64,   0,   0,   6,   0,
        10,   0,   0,   0,   0,  64,   0,   0,   6,   0,  10,   0,   0,
         0,   0,  64,   0,   0,   6,   0,  10,   0,   0,   0,   0,  64,
         0,   0,   6,   0,  10,   0,   0,   0,   0,  64,   0,   0,   6,
         0,  10,   0,   0,   0,   0,  64,   0,   0,   6,   0,  10,   0,
         0,   0,   0,  64,   0,   0,   6,   0,  10,   0,   0,   0,   0,
        64,   0,   0,   6,   0,  10,   0,   0,   0,   0,  64,   0,   0,
         6,   0,  10,   0,   0,   0,   0,  64,   0,   0,   6,   0,  10,
         0,   0,   0,   0,  64,   0,   0,   6,   0,  10,   0,   0,   0,
         0,  64,   0,   0,   6,   0,  10,   0,   0,   0,   0,  64,   0,
         0,   6,   0,  10,   0,   0,   0,   0,  64,   0,   0,   6,   0,
        10,   0,   0,   0,   0,  64,   0,   0,   6,   0,  10,   0,   0,
         0,   0,  64,   0,   0,   6,   0,  10,   0,   0,   0,   0,  64,
         0,   0,   6,   0,  10,   0,   0,   0,   0,  64,   0,   0,   6,
         0,  10,   0,   0,   0,   0,  64,   0,   0,   6,   0,  10,   0,
         0,   0,   0,  64,   0,   0,   6,   0,  10,   0,   0,   0,   0,
        64,   0,   0,   6,   0,  10,   0,   0,   0,   0,  64,   0,   0,
         6,   0,  10,   0,   0,   0,   0,  64,   0,   0,   6,   0,  10,
         0,   0,   0,   0,  64,   0,   0,   6,   0,  10,   0,   0,   0,
         0,  64,   0,   0,   6,   0,  10,   0,   0,   0,   0,  64,   0,
         0,   6,   0,  10,   0,   0,   0,   0,  64,   0,   0,   6,   0,
        10,   0,   0,   0,   0,  64,   0,   0,   6,   0,  10,   0,   0,
         0,   0,  64,   0,   0,   6,   0,  10,   0,   0,   0,   0,  64,
         0,   0,   6,   0,  10,   0,   0,   0,   0,  64,   0,   0,   6,
         0,  10,   0,   0,   0,   0,  64,   0,   0,   6,   0,  10,   0,
         0,   0,   0,  64,   0,   0,   6,   0,  10,   0,   0,   0,   0,
        64,   0,   0,   6,   0,  10,   0,   0,   0,   0,  64,   0,   0,
         6,   0,  10,   0,   0,   0,   0,  64,   0,   0,   6,   0,  10,
         0,   0,   0,   0,  64,   0,   0,   6,   0,  10,   0,   0,   0,
         0,  64,   0,   0,   6,   0,  10,   0,   0,   0,   0,  64,   0,
         0,   6,   0,  10,   0,   0], dtype=uint8)
>>> b.array(uproot.asdebug)[0][:500].view(">i4")
array([1073782828,<PHONE_NUMBER>,        554,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360,
            16384,     393226,          0,<PHONE_NUMBER>,     655360],
      dtype=int32)

There's no way these fit into a structure like
>>> b.interpretation
asjagged(astable(asdtype("[('id_', '>u4'), ('fX', '>f8'), ('fY', '>f8'), ('fZ', '>f8'), ('fT', '>f8'), ('itrack', '>i4'), ('vtxId', '>u4'), ('procType', '>u4')]")), 10)

None of those first (very repetitive) 5550 bytes looks anything like a big-endian double-precision floating point number, for instance. Also, the data we're expecting has a data type that repeats every 48 bytes, but the data we see repeats every 20 bytes. There's no way they can line up.
So I'm baffled. Can you read these data in ROOT?

@jpivarski thanks for your analysis of the issue!
This looks strange indeed! I did not create the files myself, and currently have not setup root.
I will install ROOT, check the file and report back here.

I cannot read the data in ROOT because I'm missing the dictionaries.
Explicitly defining the class SimVertex also does not help.

tkeck@localhost:~$ root -l 09D95BE7-23C8-5542-9FA4-102E3FEDB7A2.root
root [0]
Attaching file 09D95BE7-23C8-5542-9FA4-102E3FEDB7A2.root as file0...
Warning in TClass::Init: no dictionary for class edm::EventAuxiliary is available
Warning in TClass::Init: no dictionary for class edm::Hash<2> is available
Warning in TClass::Init: no dictionary for class edm::EventID is available
Warning in TClass::Init: no dictionary for class edm::Timestamp is available
[...]
root [3] float test;
root [4] Events->SetBranchAddress("SimVertexs_g4SimHits__SIM.obj", &test)
Error in TTree::SetBranchAddress: The pointer type given "Float_t" (5) does not correspond to the type needed "vector" by the branch: SimVertexs_g4SimHits__SIM.obj
(int) -2
root [12] class SimVertex {
root (cont'ed, cancel with .@) [13]public:
root (cont'ed, cancel with .@) [14]unsigned int id;
root (cont'ed, cancel with .@) [15]double fX;
root (cont'ed, cancel with .@) [16]double fY;
root (cont'ed, cancel with .@) [17]double fZ;
root (cont'ed, cancel with .@) [18]double fT;
root (cont'ed, cancel with .@) [19]int itrack;
root (cont'ed, cancel with .@) [20]unsigned int vtxId;
root (cont'ed, cancel with .@) [21]unsigned int procType;
root (cont'ed, cancel with .@) [22]}
root [24] std::vector test3;
root [25] Events->SetBranchAddress("SimVertexs_g4SimHits__SIM.obj", &test3)
Error in TTree::SetBranchAddress: The class requested (vector) for the branch "SimVertexs_g4SimHits__SIM.obj" is an instance of an stl collection and does not have a compiled CollectionProxy. Please generate the dictionary for this collection (vector) to avoid to write corrupted data.
(int) -3

So I guess this is an issue with the ROOT files themselves, rather than with uproot.
I close the issue for now and check back with the people who provided the files.
Thanks for your help (in particular the uproot.asdebug is very handy)!


Fixed typo, added namespace doc & Cake upgrade

Found some minor typos in docs / literals
Added Cake non standard namespace docs
Upgraded and built against lates released Cake (0.6.3)

Merged :+1:


Add changes to runtime-api.yaml

Codecov Report
Patch coverage has no change and project coverage change: +3.10 :tada:

Comparison is base (7c17831) 32.41% compared to head (b91a69d) 35.52%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1216      +/-   ##
==========================================
+ Coverage   32.41%   35.52%   +3.10%
==========================================
  Files         376       82     -294
  Lines       43800     7792   -36008
  Branches    14200     4366    -9834
==========================================
- Hits        14198     2768   -11430
+ Misses      29352     5024   -24328
+ Partials      250        0     -250

Flag
Coverage Δ

adapter
?

admin-domain-service
?

backoffice-domain-service
?

devportal-domain-service
?

idp-domain-service
80.10% <ø> (ø)

runtime-domain-service
32.15% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 294 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Improve code sharing between index.js in 4 repos. import js file to set site variables?

took site and siteLower constants and moved them from index.js to siteConstants.js in this commit: https://github.com/rrelyea/evusheld/commit/f5fa0c67573ed36728ba64faa6f26c0ba3fa4f61


Added cities of Nepal

Added cities of Nepal
👨💻 Changes proposed

✔ Check List (Check all the applicable boxes)

[x] My code follows the code style of this project.
[x] This PR does not contain plagiarized content.
[x] The title of my pull request is a short description of the requested changes.

📄 Note to reviewers

Good one @iamrahul8
Congrats 🎉, your PR got merged. To be eligible for the swags from ClueLess. Follow these steps.

Star this repository. ⭐
Follow Clueless on GitHub
SignIn / SignUp in [ClueLess Website](https://www.clueless.tech/) .
Go to the profile section from the top right-hand corner.

Comment your profile URL under this comment tagging the person who merged your PR.

That’s it. Follow the leaderboard!
And don’t forget to share your achievement with your friends.

@nikhil25803  Here is my profile - https://www.clueless.tech/profile/admin/107087746951592717989


Remove downgrade of the python version in GHA configuration

Use the default system version of python instead.
Need to remove this step from the workflow configuration.

Python 3.6 is the default Python on CentOS 8 as well, so although it is a carryover from Qpid Dispatch, it is still reasonable for skupper-router to test with Python 3.6 as much as possible, because there is a good chance it will be deployed on Python 3.6.
Having said that, CentOS/RHEL 8 offers multiple versions of some packages, and Python is one of those.
$ podman run --rm -it ubi8/ubi dnf search 'python3*-devel'
python36-devel.x86_64 : Libraries and header files needed for Python development
python38-devel.x86_64 : Libraries and header files needed for Python development
python39-devel.x86_64 : Libraries and header files needed for Python development

Therefore, it would be possible to pick one of the newer Pythons as the required minimum, if there was a good reason. I can think of, in order of importance

simply restricting the set of possibilities where there is a little to no benefit of keeping all options open (the logic behind my https://github.com/skupperproject/skupper-router/issues/48)
https://www.python.org/dev/peps/pep-0563/ useful feature for tidier type annotations
asyncio had few api changes since 3.6, so if that is ever used in testing, it is better to support only recent pythons

Good point re: Centos8, closing.


Move aws_tags to EC2Instance model, add utils_aws to import EC2 instance data.

This change is

Coverage decreased (-0.5%) to 51.761% when pulling 56a10afeabeaf242b2410fe51b691e3941ef1006 on ropable:master into a6e7710a6705c07d760ed8d45b199abc460fe0f2 on parksandwildlife:master.

Coverage decreased (-0.6%) to 51.651% when pulling ce09a6d15536f1177a386854fe56fa2583b81a40 on ropable:master into a6e7710a6705c07d760ed8d45b199abc460fe0f2 on parksandwildlife:master.


Created a small implementation of a BST

Description
A short summary of what is included in your Pull Request.
Fixes #(issue_no)

[ ] Issue_no: <insert your issue no here>
[X] Not applicable

Type of change
Choosing one or more options from the following as per the nature of your Pull request.

[X] New project insert
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Documentation Update

Project

Binary Search Tree:

Short Description: Created new add and remove python functions that represent a BST's properties

Checklist:
Please tick all the boxes that are fulfilled by your Pull Request.

[X] I have named my files and folder, according to this project's guidelines.
[X] My code follows the style guidelines of this project.
[X] I have commented on my code, particularly in hard-to-understand areas.
[X] I have created a helpful and easy to understand README.md, according to the given README_TEMPLATE.
[X] My changes do not produce any warnings.

Thank you so much for taking the time to review!


Problem wit nodemon

I have a problem with nodemon. I think it's very important to use only webpack for rebuilding components and style for them. I decided to turn off stylelinter and take control of writing my styles by myself. Hovewer, when I wrote code like that :
@import '../../theme/variables' body {   background-color: $color-dark-gray;   font-family: 'Helvetica-Light', Helvetica, Arial, sans-serif; }, where it lacks semicolons after import of file, I got follow message in terminal:
[nodemon] app crashed - waiting for file changes before starting....
The only way to repair it - to execute "rs" for reload server but it's too long.
I suspect that it happens because of WebpackIsomorphicTools, maybe it'll be better to use a classic webpack way as mentioned here? Or don't use it in development mode.
Thanks.

@maximusnikulin I think it's not the problem of nodemon, if you use node index.js to instead of 'nodemon index.js' through the npm start script, you will see the server stopped due to the error which you have demonstrated. In that situation, even hot reloading can't work, so you must fix the error and restart the server :)

Thanks for your answer, I'll try to find the reason of it.

@wellyshen is there some solution from it? Because it's quite tough when write your sass code and make mistake you should wait for a long time (till 10s) to reload server. I thought that style lint maight be nice solution but when I do failsOnError:true , and then do mistake in sass file my app doesn't rebuild. I would use your boilerplate in one of my project, but it last problem that I can't solve.
Thanks

Hi @maximusnikulin
I'm still looking for how to solve the hot reloading of this issue, but I can recommend you to use inline style library for react.js. One of the library call radium.
I personal use the inline style with react.js, you might ask me why don't use the library in the starter kit, it's because I need to consider to the general use case for most ppl, but I need to say inline style works great with react and hot reloading :)

@maximusnikulin
It is a bug of webpack-isomorphic-tools, now it has been fixed (detail).
If you don't want to stop the hot reloading process of .scss error, please turn the stylelint failOnError from webpack config file, like below :
 new StyleLintPlugin({
      syntax: 'scss',
      failOnError: false,      // Disable style lint error terminating here
    }),


Can we store all our downloadable files in git?

From What is my disk quota?:

We recommend repositories be kept under 1GB each. This limit is easy to stay within if large files are kept out of the repository. If your repository exceeds 1GB, you might receive a polite email from GitHub Support requesting that you reduce the size of the repository to bring it back down.

I suspect that the repository currently weighs in at under 1 GB, but I'm not sure how long that will last, as they continue to add more downloads.
Copying all these large files might also slow down Jekyll, but I'm not sure.
If this were a normal repository, I'd consider using Git LFS or just storing the large files in an S3 bucket, but I'm not sure how equipped we are to use those solutions with respect to ATOs, training the PCLOB folks to leverage them, etc.

As of right now, I believe we've got all the files we need in the repository, and my .git directory clocks in at 192 MB, which doesn't seem too bad.

I'm gonna close this, seems like we're good to go for the forseeable future.


Need to add mutex

Multiple parallel Ecode, Decode, Count and Insert hangs due to race conditions.
We should make formal mutex locks.

mutex lock can be done from the outside...  right?


Question regarding latest results

There seems to be a drastic regression in the latest results for languages that were previously on top.
Do you know why that might be?

Go and D did suffer perf degradation after updating their compilers: https://github.com/jinyus/related_post_gen/pull/414#issuecomment-1808287956
I downgraded the golang compiler which fixed the issue: https://github.com/jinyus/related_post_gen/pull/421#issue-1989611170
Is there any other language where you noticed regressions?

Go and D did suffer perf degradation after updating their compilers: #414 (comment)
I downgraded the golang compiler which fixed the issue for go: #421 (comment)
@cyrusmsk said he'd do the same for D. I will look into it today if he doesn't get a chance to do it.
Is there any other language where you noticed regressions?

Didn't C# also loose some points? I don't remember, but wasn't it higher up? 🤔

Honestly I'm not sure that the issue in the updated version of compiler. But will see if manual installation of 1.34 LDC will help

@cyrusmsk @Imperatorn Just confirmed https://github.com/jinyus/related_post_gen/pull/429#issuecomment-1811459004. Performance is back to normal with 1.34.0 beta 1.

@cyrusmsk @Imperatorn Just confirmed #429 (comment). Performance is back to normal with 1.34.0 beta 1.

Thanks for investigating!

https://github.com/jinyus/related_post_gen/blob/2d76d834bd758dfd4465952fe9884fdce6ef1805/readme.md?plain=1#L70
I will close this now.


Firebase Messaging crash during app loading if Messaging was a added via SPM

[REQUIRED] Step 1: Describe your environment

Xcode version: 12.5
Firebase SDK version: 8.7.0
Installation method: Swift Package Manager
Firebase Component: Messaging

[REQUIRED] Step 2: Describe the problem
Steps to reproduce:
During app loading I get crash cause createTableWithName method. You can see stack on attached screenshot.

It looks like the sqlite3 database creation is failing. Does the app or any of its other dependencies do anything unusual that could change the state of the system sqlite3 library that FirebaseMessaging is depending on?

I think no, moreover everything is good with Messaging when it was installed via CocoaPod. This crash is specific to SPM

The 8.9.0 release will improve the logging when this crash occurs, but the crash occurs after a table create failure, so something has already gone wrong.

@DenisVerkhovod Can you try deleting the app and install again?


chore(scripts/technical-debt-metrics): no longer report missing module docstrings

Because there are no missing module docstrings anymore!

maintainer merge
🐙

Thanks!
bors merge


support phyloseq objects as inputs

add phyloseq → AbundanceData/ MbioDataset/ Collection methods so that people can run our computes on their own data. we should also look into explicit support for nephele and czid data as inputs, and possibly take .biom files as inputs as well.

also maybe SummarizedExperiment

high priority nice to have

I think the strategy will be to support TreeSummarizedExperiment as inputs and outputs , and rely on miaverse for support making these objects


[chip_earlgrey,spi_device,sival] Add new metadata to chip testplan for SiVal

Not all 'features' are covered by the current tests, this may be further fixed by either adding new tests or changing the granularity of the features identified.

Did a quick run-through of the grammar in the last push. Thanks @rswarbrick

CHANGE AUTHORIZED: hw/ip/spi_device/data/spi_device.hjson

CHANGE AUTHORIZED: hw/ip/spi_device/data/spi_device.hjson


Create routes to shell mappings using JSON

Hi, wondering if there's a way to feed in the route & commands via JSON file or JSON on an HTTP endpoint?  I would like to use values stored in a db to create the mappings if possible.

Hi @carlskii,
json is not the best format for configuring routes because there will be difficulties with escaping quotes of different types, single and double, something like:
{"/path": "sh -c \"echo \\"hello\\"\""}

there i'm not sure that correct quotes or not
but just shell-file can be as a way to configure routes/commands, like:
run_server.sh:
#!/bin/sh
shell2http -port 8000 \
  /date date \
  /route1 "sh -c 'echo 123'" \
  /route2 "some_shell_script.sh"

this could almost be a replacement for json


matcher: add prefix matching support for matching API

Signed-off-by: Xie Zhihao<EMAIL_ADDRESS>Commit Message: matcher: add prefix matching support for matching API
Additional Description: Envoy has the matching API support for exact map and prefix map while the prefix map has not been implemented yet. This patch realizes a general prefix map matcher based on trie lookup table for prefix matching. The prefix map matcher is not been optimized for IP (like the trie matcher based on LC trie) or path (like the path-splitting one).
Risk Level: Low or medium
Testing: Unit and integration
Docs Changes: N/A
Release Notes: WIP
Platform Specific Features: N/A
Fixes #20198

/retest

/assign @snowp

@yanavlasov for second set of eyes


syntax highlight with prism

Thanks!

All looks good but one important thing is missing, that line number(s) no longer clickable/selectable. :flushed:

Single line: https://try.gogs.io/unknwon/gogs/src/master/gogs.go#L14
Multiple lines: https://try.gogs.io/unknwon/gogs/src/master/gogs.go#L18-L20

In view file, white padding around code could be problem, too.

A plugin might be helpful: http://prismjs.com/plugins/line-highlight/ I'm still looking.

Style can be easily fix, the only thing left is the select line number, I looked over line-number plugin, I guess there is a way to add hook(either inside line-number plugin or another independent hook)

More problems found:
Plain file does not have line numbers anymore... https://try.gogs.io/unknwon/gogs/src/master/Dockerfile#L3

Just for reference: #1440

@Unknwon That's exactly what I was not sure about

I'm not sure about public/ng/js/gogs.js line 137- Gogs.renderCodeView. Looks its purpose is to deal with diff code views and that is still working properly but if this is the case I would suggest changing it and applying it only to .code-diff class and not .code-view which is more generic.

Never realized there was that option on file view. I understand that is a useful feature to link to a line or to a range. I already included the line highlight plugin just in case something like that were possible. I will try to think how to implement it.

In view file, white padding around code could be problem, too.

About the padding it is included on purpose on every view but it can be changed. But I wouldn't do it that way. First we have to agree about the html structure then change the CSS.
My idea is that .panel-content is a wrapper and must be there always. If we don't want padding on that wrapper we can just remove it.
Then if it is a markdown it will have .markdown wrapper div inside the .panel-content where we can put the extra padding.
If on the other hand it is a binary file it has .view-raw and we can add the padding there too.
And if its a source code file then it will have as wrapper .code-view and we can remove any padding there.
After having an agreement on that the CSS tweak shouldn't be a problem. Removing padding from .panel-content. Increasing a little the padding on '.markdownwrapper. Adding padding to.view-rawif neccesary. And removing any padding if any from .code-view`.

Plain file does not have line numbers anymore...

I think that is reported on prism as an issue so I hope it will be solved soon.

By the way now that we have the language-xxx class in code tag using one or other library would be just very simple. Any of prettyprint, prism, or highlightjs can be used. It is just a matter of adding the library and in the case of prettyprint and highlightjs just add the line of code that triggers the highlight. It the case of prettyprint being sure not to ask to highlight everything but only the codes that has a language-xxx class.
This is just as info so we can have it in consideration if in some moment we decide is better to replace the highlight library.
I like prism library anyway so I'm going to look for the solution to the lines highlight. Maybe there is a way by using what is already available on that library or otherwise tweaking those plugins to have a selection API for the lines.

It should be possible already to highlight pieces of code dinamically if I understand correctly this issue in prism.
Prism.highlightElement()

and data-line attribute should be what we are looking for. I'm going to play on the console how it works.

Something like:
$("pre").attr("data-line", 16);
Prism.highlightAll();

works fine. Calling Prism.highlightElement(our_pre_element) which would be a better choice triggers an error. Maybe a bug in the library. But highlightAll is just fine. Added that to an onclick event to the line numbers shouldn't be a problem, I hope.
About links I would say that instead of an anchor link as it is now ...#L16 we should pass the highlight as a queryString so using your previous examples they would be turned into:
Single line: https://try.gogs.io/unknwon/gogs/src/master/gogs.go?lines="14"
Multiple lines: https://try.gogs.io/unknwon/gogs/src/master/gogs.go?lines="18-20"

Reading that queryString, maybe validating it with a regexp, and using it on the pre tag data-line attribute is just piece of cake.
The library allows even multiple ranges:
Multiple ranges: https://try.gogs.io/unknwon/gogs/src/master/gogs.go?lines="14, 18-20"

The dynamic was not so simple that it seem at first glance. I had to edit the line numbers plugin to add that feature. It is just very little code but must be on that place. Here is the full code of the line numbers prism plugin. My added code is just at the end, the on click event binding.
Prism.hooks.add('after-highlight', function (env) {
	// works only for  wrapped inside
 (not inline)
	var pre = env.element.parentNode;
	var clsReg = /\s*\bline-numbers\b\s*/;
	if (
		!pre || !/pre/i.test(pre.nodeName) ||
		// Abort only if nor the
 nor the  have the class
		(!clsReg.test(pre.className) && !clsReg.test(env.element.className))
	) {
		return;
	}

	if (clsReg.test(env.element.className)) {
		// Remove the class "line-numbers" from the
		env.element.className = env.element.className.replace(clsReg, '');
	}
	if (!clsReg.test(pre.className)) {
		// Add the class "line-numbers" to the

		pre.className += ' line-numbers';
	}

	var linesNum = (1 + env.code.split('\n').length);
	var lineNumbersWrapper;

	var lines = new Array(linesNum);
	lines = lines.join('');

	lineNumbersWrapper = document.createElement('span');
	lineNumbersWrapper.className = 'line-numbers-rows';
	lineNumbersWrapper.innerHTML = lines;

	if (pre.hasAttribute('data-start')) {
		pre.style.counterReset = 'linenumber ' + (parseInt(pre.getAttribute('data-start'), 10) - 1);
	}

	env.element.appendChild(lineNumbersWrapper);

	$(".line-numbers-rows > span").on("click", function() {
		var line = $(this).index() + 1;
		$(pre).attr("data-line", line);
		Prism.highlightAll();
	});

});;

First for that on click to work I had to relax in the CSS the rule that disables events on those elements in gogs-prism.css. Changing pointer-events to auto instead of none.
So there is no problem to add that feature.
We should agree about the HTML structure that I explained before. In my opinion it should be as explained.
For Markdown

    the markdown output here

For Raw files

    the file here. if an image just the image.

For Source Code Files

    the highlighted source code here.

And that's why the template view_file.tmpl looked as follows in my pull request.

    	{{if .ReadmeExist}}

    	  {{if .FileContent}}{{.FileContent | Str2html}}{{end}}

      {{else if not .IsFileText}}

            {{if .IsImageFile}}

            {{else}}
            {{.i18n.Tr "repo.file_view_raw"}}
            {{end}}

    	{{else if .FileSize}}

{{.FileContent}}

    	{{end}}

I forgot to mention that line numbers are not selectable and that's done on purpose on prism. And I agree. Almost 100% of the times people wants to copy just the code.
I never had the need to copy the line numbers. In fact it has been many times in the past that I had to remove line numbers manually because a website code snippets had numbers and there were no way to select just the code.
So only the code is selectable and it is on purpose.
About the dynamic highlight I don't see it very usefull either but anyway adding the snippet to line numbers plugin will provide that feature.
What is very very very useful for sure is the possibility to link to the code with highlighted parts. And that can be coded very very easily with a queryString.
I can start implementing all that but I would prefer to first agree on the html structure to change that either again and edit CSS to show source code without padding.

Thanks for your explanation!
About the highlight thing I think we are on the same page.
But about line number, I insist that line numbers should be selectable, it is very useful to point other people to a specific position in a large file.

But I wouldn't call it selectable but linkable. And yes I agree that linkable is a very very very useful feature and should be coded and it is very easy to implement using prism.
I'm going to start coding it all and will do a pull request againts this branch so you can test it all.

Sure... linkable! :grinning:

I have all implemented but for some reason the line numbers are out of place. I'm not able to figure it out just now, as much I looked into it I don't see anything strange and the line numbers are placed in a really strange place. I will look into it tomorrow. I will start again from the previous commit applying the changes to see if I figure out the problem.
Just a question that maybe you can help. As explained before the lines to highlight are going to come from a queryString on the url, like
Single line: https://try.gogs.io/unknwon/gogs/src/master/gogs.go?lines=14
Multiple lines: https://try.gogs.io/unknwon/gogs/src/master/gogs.go?lines=18-20
Multiple ranges: https://try.gogs.io/unknwon/gogs/src/master/gogs.go?lines=14,18-20,23-24

The code I have just now to read the query parameter looks like this
					lines := ctx.Query("lines")
					//TODO: sanitization, remove spaces and validation if needed
					ctx.Data["HighlightLines"] = lines

And the question is what we have to do in that TODO. Because I don't know if Query already sanitizes the input. And what you think about removing spaces and validating that input with a regexp.

Sounds great!
For your question:
No, Query only returns value of the key.
For this specific case, I think only need to check if those are valid int numbers, if not then simply ignore them. Correct me if I understand your question wrong.

Ok then we validate the string with a regexp that ensures is valid: it only can contain values separated by commas, and each of those values must be a valid range, being a valid range just a number or two numbers separated by an hyphen. I can allow spaces which don't have meaning too.
Tomorrow I will finish it.

No problem, take your time!


formatting - datetime and number format for Desktop

https://github.com/MicrosoftDocs/powerbi-docs/blob/live/powerbi-docs/supported-languages-countries-regions.md
Can we add a note about how Desktop gets default number and date format options? We have had a few customer complaints about this.
Windows Settings -> Region -> "Additional date, time, and regional settings" link

This launches the old Control Panel to Clock and Region. Click the "Change date, time, or number formats" option here.

Ensure it says "Match Windows display language"

Or change it if necessary.
This setting controls default date format (separator, month before day or opposite, etc) and number thousands/decimal separator.

Thanks for the feedback, @MatthLee. I'm assigning to the author to consider your update request.

resolving in https://github.com/MicrosoftDocs/powerbi-docs-pr/pull/3726


Changelog and version numbers for 6.1.6 preview

Codecov Report

Merging #237 into dev will increase coverage by <.01%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                dev     #237      +/-   ##
============================================
+ Coverage     33.45%   33.45%   +<.01%
  Complexity     1487     1487
============================================
  Files            97       97
  Lines         23390    23390
  Branches       3840     3840
============================================
+ Hits           7824     7825       +1
+ Misses        14002    14000       -2
- Partials       1564     1565       +1

Flag
Coverage Δ
Complexity Δ

#JDBC41
33.33% <ø> (+0.02%)
1478 <ø> (+3)
:arrow_up:

#JDBC42
33.35% <ø> (-0.03%)
1483 <ø> (-2)

Impacted Files
Coverage Δ
Complexity Δ

...a/com/microsoft/sqlserver/jdbc/SQLJdbcVersion.java
0% <ø> (ø)
0 <0> (ø)
:arrow_down:

src/main/java/microsoft/sql/DateTimeOffset.java
37.14% <0%> (-2.86%)
8% <0%> (-2%)

...in/java/com/microsoft/sqlserver/jdbc/IOBuffer.java
36.84% <0%> (-0.11%)
0% <0%> (ø)

.../microsoft/sqlserver/jdbc/SQLServerConnection.java
41.1% <0%> (-0.09%)
227% <0%> (-2%)

...m/microsoft/sqlserver/jdbc/SQLServerResultSet.java
23.96% <0%> (-0.05%)
169% <0%> (-1%)

...om/microsoft/sqlserver/jdbc/SQLServerBulkCopy.java
45.05% <0%> (+0.14%)
183% <0%> (+1%)
:arrow_up:

...rc/main/java/com/microsoft/sqlserver/jdbc/DDC.java
25.66% <0%> (+1.11%)
42% <0%> (+3%)
:arrow_up:

...om/microsoft/sqlserver/jdbc/SimpleInputStream.java
41.91% <0%> (+1.47%)
10% <0%> (+1%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a0f330...8bf3387. Read the comment docs.


ListView stutters when scrolling (noticeable and consistent stuttering).

Is there an existing issue for this?

[X] I have searched the existing issues
[X] I have read the guide to filing a bug

Steps to reproduce

Using the latest Master Flutter channel. and iOS 16.6 (even iOS 16.6.1)
Created a basic Flutter app with a simple ListView.builder widget
Running the App in debug or even release mode, it can be observed that the ListView is stuttering (noticeable and consistent stuttering) when scrolling. This used to happen before when Promotion devices just came out but was also fixed quite some time ago. But apparently this issue has resurfaced.

Expected results
This is a simple ListView. Scrolling should be smooth and consistent at 120Hz.
Actual results
For some reason, display stays at 120Hz when not scrolling but as soon as I start scrolling the refresh rate is all over the place as seen in the video. This causes stuttering which is quite annoying and noticeable. Also this stuttering is very consistent and continuous. This has started recently. I am not sure if it is due to iOS 16.6. I am running iOS 16.6.1.
Code sample
Simple ListView scroll stutter example
import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return const MaterialApp(
      home: ListApp(),
    );
  }
}

class ListApp extends StatelessWidget {
  const ListApp({super.key});

  @override
  Widget build(BuildContext context) {
    final width = MediaQuery.of(context).size.width;
    final height = MediaQuery.of(context).size.height;
    return Scaffold(
      body: Center(
        child: ListView.builder(
          shrinkWrap: true,
          primary: false,
          itemBuilder: (context, index) {
            return Padding(
              padding:
                  const EdgeInsets.symmetric(horizontal: 32.0, vertical: 32),
              child: Container(
                width: width * 0.8,
                height: height * 0.3,
                decoration: BoxDecoration(
                    color: Colors.purple,
                    borderRadius: BorderRadius.circular(15)),
              ),
            );
          },
        ),
      ),
    );
  }
}

Screenshots or Video

Screenshots / Video demonstration
https://github.com/flutter/flutter/assets/49350919/8b1f5919-5e6e-446a-9efb-cb0ef98bfd89

Logs
Logs

Flutter Doctor output
Doctor output
[✓] Flutter (Channel master, 3.14.0-14.0.pre.211, on macOS 13.5 22G74
    darwin-arm64, locale en-IN)
    • Flutter version 3.14.0-14.0.pre.211 on channel master at
      /Users/pankajkumar/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 69f6e7d15d (40 minutes ago), 2023-09-11 00:50:38 -0400
    • Engine revision 7e41b064eb
    • Dart version 3.2.0 (build 3.2.0-134.0.dev)
    • DevTools version 2.27.0

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/pankajkumar/Library/Android/sdk
    • Platform android-34, build-tools 34.0.0
    • Java binary at: /Users/pankajkumar/Applications/Android
      Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build
      17.0.6+0-17.0.6b829.9-10027231)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.3.1)
    • Xcode at /Users/pankajkumar/Library/Mobile
      Documents/com~apple~CloudDocs/Desktop/Xcode.app/Contents/Developer
    • Build 14E300c
    • CocoaPods version 1.12.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[!] Android Studio (version unknown)
    • Android Studio at /Users/pankajkumar/Applications/Android Studio Iguana
      2023.2.1 Canary 3.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    ✗ Unable to determine Android Studio version.
    • Java version OpenJDK Runtime Environment (build
      17.0.8+0-17.0.8b1000.8-10699129)

[✓] Android Studio (version 2022.3)
    • Android Studio at /Users/pankajkumar/Applications/Android
      Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build
      17.0.6+0-17.0.6b829.9-10027231)

[!] Android Studio (version unknown)
    • Android Studio at /Users/pankajkumar/Applications/Android Studio Hedgehog
      2023.1.1 Beta 3.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    ✗ Unable to determine Android Studio version.
    • Java version OpenJDK Runtime Environment (build
      17.0.7+0-17.0.7b1000.6-10550314)

[✓] IntelliJ IDEA Community Edition (version 2023.2.1)
    • IntelliJ at /Users/pankajkumar/Applications/IntelliJ IDEA Community
      Edition.app
    • Flutter plugin version 75.1.4
    • Dart plugin version 232.9559.10

[✓] VS Code (version 1.70.1)
    • VS Code at /Users/pankajkumar/Downloads/Visual Studio Code.app/Contents
    • Flutter extension version 3.64.0

[✓] VS Code (version 1.78.2)
    • VS Code at /Applications/Visual Studio Code 2.app/Contents
    • Flutter extension version 3.64.0

[✓] Connected device (3 available)
    • iPhone 13 Pro Max (mobile)  • ios
      • iOS 16.6.1 20G81
    • macOS (desktop)                     • macos                     •
      darwin-arm64   • macOS 13.5 22G74 darwin-arm64
    • Chrome (web)                        • chrome                    •
      web-javascript • Google Chrome 116.0.5845.179

[✓] Network resources
    • All expected network resources are available.

Hi @pkumar8888. Thanks for filing this. This seems to be similar to https://github.com/flutter/flutter/issues/131319 whose fix was cherry-picked in 3.10.7 version as mentioned in the last comments of that issue. Kindly check out this comment and this for more information on this.
Closing this as this seems to be a known issue which has been fixed, if you disagree, kindly comment and I'll reopen this.
Thank you!

Hi..I just checked the case #131319. But I would like to be on the latest flutter version and not on the 3.10 branch. This issue was non existent on 3.14 earlier versions (not sure exactly which version brought about this stuttering).
So I would like to know what are the plans and priority for this issue on version 3.14 and higher, Because this is more or less a show stopper for my startup at this point.

Hi @pkumar8888. Sorry about that, per this https://github.com/flutter/flutter/issues/131319#issuecomment-1683070516, I guess for this to work well if you would like to remain on a stable channel, you'd need to be on the 3.10.7.
You might also want to read more about how bug fix releases work here: https://github.com/flutter/flutter/wiki/Flutter-build-release-channels#will-a-particular-bug-fix-be-provided-in-a-hotfix-release (and whole that page for more info)

I am still having this issue on the newly released 3.14.  People are saying its fixed in 3.13 but im on the latest and the stutter is still there.

I am also wondering why is the fix not yet included in the latest flutter version ( let it be stable or the master channel).
Also surprised why this is not on the top priority as this issue is definitely affecting all the iOS devices on iOS 16.6 .

After doing some more research I found these open issues.
https://github.com/flutter/flutter/issues/134309
https://github.com/flutter/flutter/issues/134242
It seems that the fix 3.10.7 was causing some crashes so it was reverted back to the jank issue so that apps wouldn't crash. At this time they are still looking for a proper fix to address both the stuttering and the crashes caused by the first fix.
It is a little frustrating how far I had to dig to get a simple answer to why the first fix was reverted in all of the newer releases of Flutter.  Also, this issue has been around for over a month now so I hope a fix comes soon.  My work app on every screen with a listview or CustomScrollView stutters so much that it is very distracting to the end user making our app look very unprofessional.  I just spent a few days trying to re-engineer some of the screens as small as possible to try to mitigate some of the stutter but that only helped a tiny bit.  I have one screen which is just a listview of 10 rows and that stutters. I really hope they find a fix to this issue soon as most apps have listviews.

I am still having this issue on the newly released 3.14. People are saying its fixed in 3.13 but im on the latest and the stutter is still there. I am looking at the engine change made in 3.10.7 and that change is not in the 3.13 candidates. Wouldn't a fix go into the latest and greatest release.

The fix has been reverted by the flutter team in 3.13.3 (see https://github.com/flutter/flutter/issues/134309#issuecomment-1712513231). We had to downgrade to 3.13.2. But please be aware that picking a video may crash in 3.13.2 (see https://github.com/flutter/flutter/issues/133147)


Command to list societies covered for given variables

To easily assess D-PLACE coverage for particular topics, it would be nice to be able to list societies (including glottocode, top-level language family, region and macroarea) given a set of variables (or datasets) and maybe ways to filter NA or missing data values.

Yes this would be useful :)

Isn't this what extract does?


Convert this to Template repository?

Just a suggestion...

Can you elaborate? What does that mean?

Sorry, I was a bit laconic.
https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-repository-on-github/creating-a-template-repository

I think this repo has some illustrative value, but I don't think it would make a good template repo.
Perhaps we can add a note in the README of this repo that links people to PkgTemplates.jl instead?

I had forgotten about PkgTemplates.jl!
Feel free to close this.


Consumer not persisting Cursor

We have 4 consumers for the same topic, all are supposedly up to date (0 backlog).
But when restarting brokers, thus loading cursor state from zookeeper and bookkeeper, we're consistently getting one consumer reset a long time backwards.
Looking at zookeeper metadata that consumer is the only one to have an outdated markDeleteLedgerId and markDeleteEntryId, this happens for all 10 partitions of the consumer.
Not only has this consumer an outdated cursor, it seems to no be updating it as it consumes all the backlog generated when a broker is restarted and thus consumer is reset.

@merlimat @rdhabalia we really need help with this it's causing issues to us in production.

@sschepens Few questions to try identify the issue:

Is that consumer receiving at the same rate as the other consumer?
Is it keeping some messages in the consumer receiving queue?
Does it change if using ack timeout on the consumer ?
Can you try to remove the throttling of persisting the mark-delete position? There are 2 ways for doing that :

Change the broker-wide setting: managedLedgerDefaultMarkDeleteRateLimit=0. That mens to persist each and every update
Change the setting on a particular namespace :
bin/pulsar-admin namespaces set-persistent --bookkeeper-ensemble 2  --bookkeeper-write-quorum 2  --bookkeeper-ack-quorum 2 --ml-mark-delete-max-rate 0

managedLedgerDefaultMarkDeleteRateLimit controls how frequently the position is being persisted, to avoid too many writes on BookKeeper when consuming messages.
The default is 0.1 (updates per seconds), which means 1 update every 10 seconds.
Given the current implementation (using a guava rate limiter), subsequent acks are tracked in memory. The problem might be related to the fact that if there are no more writes on the topic, the persisted position for the cursor doesn't get a chance to get updated.

@merlimat

Is that consumer receiving at the same rate as the other consumer?

Yes it is, we also tried consuming harder.

Is it keeping some messages in the consumer receiving queue?

It wouldn't appear so, as backlog reported by broker before restarting is 0, which would confirm that it considers all messages were acked.

Does it change if using ack timeout on the consumer ?

We're already using ack timeout for all consumers.

Can you try to remove the throttling of persisting the mark-delete position?

I'll try, we're now using managedLedgerDefaultMarkDeleteRateLimit=10 to theoretically save state 10 times per second.

Given the current implementation (using a guava rate limiter), subsequent acks are tracked in memory. The problem might be related to the fact that if there are no more writes on the topic, the persisted position for the cursor doesn't get a chance to get updated.

Hmm shouldn't this have at least a way of checking when state has not been saved for some time and force a save?

Right now all the checking is done when processing an acknowledgment. There was the plan of fixing that, by going through all the topics every 1min to flush all the pending cursor updates, thought it's still like this.

@merlimat does changing managedLedgerMaxMarkDeleteRate via pulsar-admin require restarting brokers to take changes?

it requires re-loading the topics, pulsar-admin namespaces unload ...

Right now all the checking is done when processing an acknowledgment. There was the plan of fixing that, by going through all the topics every 1min to flush all the pending cursor updates, thought it's still like this.

The strage thing is, all consumers consume at the same rate, and should have acked messages and saved state at about the same time, still this is the only consumer experiencing this issue.
Here are metrics showing what happened:
Thursday 17:20 Buenos Aires Time a bunch of messages were produced:

About the same time, all 4 consumers, consumed all the messages at about the same rate:

Friday 19:40 Buenos Aires Time following a bundle unload one consumer gets reset and starts consuming messages again:

Are these consumers on the same subscription or in 4 different subscriptions?

4 different subscriptions, all are shared subscriptions and are hosted on the same 2 instances

Ok, anything in the logs that could signal there was any failure in recovering the cursor after the unload?
In case of failure to recover the cursor position, the behavior is to rollback to the oldest available message, to ensure no data is skipped.

I couln't find any log specifying a failure recovering the cursor, on the contrary, the cursor appeared to be recovered correctly, but on a really old position.

If you're able to reproduce it, can you past the stats-internal for the topic (after consuming and before restarting)?

I wasn't able to reproduce this again, but we had already set markDeleteRatio=0, and also reset the consumer.
I will inform if we experience this behavior again

@merlimat this has happened again for different consumers when restarting a broker. We have the following persistence for the namespace settings:
{
  "bookkeeperEnsemble" : 3,
  "bookkeeperWriteQuorum" : 3,
  "bookkeeperAckQuorum" : 3,
  "managedLedgerMaxMarkDeleteRate" : 0.0
}

This is absolutely not tolerable, we must find a way to fix this

By the way, this are all partitioned topics with 10 partitions

@sschepens as @merlimat mentioned: is it possible to get stats-internal for a given topic before restarting the server.
bin/pulsar-admin persistent stats-internal persistent://property/cluster/ns/topic
it provides current state of the cursor and importantly  markDeletePosition and individuallyDeletedMessages. if consumer missed to ack one of the message then markDeletePosition position will not move. So, if possible can we try to capture if it happens next time.
Sample output:
"cursors" : {
    "subscription1" : {
      "markDeletePosition" : "7029:1506",
      "individuallyDeletedMessages" : "[]",

@rdhabalia i'm checking this, we restarted another broker and the backlog bumped up yet again, and the consumer had no individuallyDeletedMessages pending...
We're going to try one more time capturing stats-internal before and after restart, but, anyways, this doesn't seem to be the case that one messages was not acked.

This is the behavior we're experiencing with backlog, the bumps are in the order of millions of messages.
And as I said, there are no, or maybe just a few unacked messages at a given time.

@sschepens In the stats internal you can see also the real cursor position, apart from the individuallyDeletedMessages. That could give few hints on where to look. Also can you attach the logs for that particular topic, before/after restart? That might be useful as well.

@sschepens also, having both internal and normal stats, might be useful in setting up a quick test to try to reproduce the issue.

@merlimat here I upload topic stats-internal for each partition before and after restart.
The consumers that got bumped are:
f6f49168781f402c99ddfa871bc0e90c-fury-orderfeed-test2.loyal-api (blue)
f6f49168781f402c99ddfa871bc0e90c-fury-orders.purchases-api (violet)
As the name specifies, the first one is a test consumer, i'm more interested in the second one. I can see that the first did have several unacked messages before restarting, even though, with each restart it seems to get more and more lagged.
This is the screenshot of the last restart:

topic-stats.zip

I'm attaching the logs of the instance before and after restart.
logs_instance_i-04cd666923f24ceaf.zip

we're collectiong logs mentioning the topic from all other instances after the restart

@sschepens Do you have message retention on time enabled, correct? (I'm seeing older ledgers are not being deleted, just wanted to make sure)

And these are the logs of all our brokers before and after restarting that instance.
all_instances_logs.zip

@sschepens about the violet consumer, in the logs you pasted, does it gets backlog on all the partitions? I don't see any problem in the "after" stats for that subscription.

These are the logs of all of our brokers before and after restart.
logs_all_instances.zip

@sschepens about the violet consumer, in the logs you pasted, does it gets backlog on all the partitions?  I don't see any problem in the "after" stats for that subscription.

I cannot now that right now, we don't have our metrics split by partition, and we didn't get the normal stats before and after.
Maybe the issue is not related to what is shown on the stats-internal, but the issue is certainly present.

@sschepens Do you have message retention on time enabled, correct? (I'm seeing older ledgers are not being deleted, just wanted to make sure)

Yes, this is our retention policy:
{
  "retentionTimeInMinutes" : 10080,
  "retentionSizeInMB" :<PHONE_NUMBER>
}

@sschepens about the violet consumer, in the logs you pasted, does it gets backlog on all the partitions? I don't see any problem in the "after" stats for that subscription.

@merlimat I would expect that the increased backlog should come from partitions that got unloaded from the restarted broker only.

@sschepens My point is that, looking that  "after" internal stats, the cursor for f6f49168781f402c99ddfa871bc0e90c-fury-orders.purchases-api doens't show backlog :
{
  "entriesAddedCounter": 36299,
  "numberOfEntries": 6101124,
  "totalSize": 784103746,
  "currentLedgerEntries": 36299,
  "currentLedgerSize": 4663005,
  "lastLedgerCreatedTimestamp": "2017-02-03 17:47:24.912",
  "lastLedgerCreationFailureTimestamp": null,
  "waitingCursorsCount": 5,
  "pendingAddEntriesCount": 0,
  "lastConfirmedEntry": "69312:36298",
  "state": "LedgerOpened",
  "ledgers": [
....

    "f6f49168781f402c99ddfa871bc0e90c-fury-orders.purchases-api": {
      "markDeletePosition": "69312:36297",
      "readPosition": "69312:36299",
      "waitingReadOp": true,
      "pendingReadOps": 0,
      "messagesConsumedCounter": 36298,
      "cursorLedger": 69336,
      "cursorLedgerLastEntry": 31461,
      "individuallyDeletedMessages": "[]",
      "lastLedgerSwitchTimestamp": "2017-02-03 17:47:24.972",
      "state": "Open"
    },
....

If you subtract entriesAddedCounter - messagesConsumedCounter you get the backlog that it gets reported by brokers.  In this case, 1 single in-flight message.

.. Ok, found the same thing on partition-7 :
    "f6f49168781f402c99ddfa871bc0e90c-fury-orders.purchases-api": {
      "markDeletePosition": "59439:44524",
      "readPosition": "61542:30970",
      "waitingReadOp": false,
      "pendingReadOps": 0,
      "messagesConsumedCounter": -986146,
      "cursorLedger": -1,
      "cursorLedgerLastEntry": -1,
      "individuallyDeletedMessages": "[(59439:44525‥60577:45163], (60577:45164‥60886:818], (60886:819‥60886:21168], (60886:21169‥60886:27590], (60886:27591‥60886:41528], (60886:41529‥60886:49252], (60886:49253‥61145:10375], (61145:10376‥61145:36353], (61145:36354‥61542:7158], (61542:7159‥61542:14589], (61542:14590‥61542:26054], (61542:26055‥61542:29862], (61542:29863‥61542:30509], (61542:30510‥61542:30515], (61542:30516‥61542:30517], (61542:30518‥61542:30534], (61542:30535‥61542:30537], (61542:30538‥61542:30556], (61542:30557‥61542:30559], (61542:30560‥61542:30563], (61542:30565‥61542:30566], (61542:30567‥61542:30568], (61542:30569‥61542:30573], (61542:30577‥61542:30581], (61542:30582‥61542:30583], (61542:30585‥61542:30587], (61542:30589‥61542:30591], (61542:30610‥61542:30613], (61542:30619‥61542:30637], (61542:30638‥61542:30640], (61542:30641‥61542:30643], (61542:30647‥61542:30649], (61542:30672‥61542:30674], (61542:30675‥61542:30677], (61542:30678‥61542:30679]]",
      "lastLedgerSwitchTimestamp": "2017-02-03 18:33:43.431",
      "state": "NoLedger"
    },

So, that cursors had 1 single unacked message (59439:44525) before restart :
    "f6f49168781f402c99ddfa871bc0e90c-fury-orders.purchases-api": {
      "markDeletePosition": "59439:44524",
      "readPosition": "70009:12925",
      "waitingReadOp": true,
      "pendingReadOps": 0,
      "messagesConsumedCounter": 12922,
      "cursorLedger": -1,
      "cursorLedgerLastEntry": -1,
      "individuallyDeletedMessages": "[(59439:44525‥70009:12922]]",
      "lastLedgerSwitchTimestamp": "2017-02-03 18:16:48.247",
      "state": "NoLedger"
    },

The same message still appears as "unacked" even after restart. Is there the possibility that the consumer is getting some unexpected data and fails to consume that particular message?
You can dump the content of the message by doing :
$ pulsar-admin persistent peek-messages $MY_TOPIC -s $MY_SUBSCRIPTION

The contents of the message seems to be fine. For some reason is still unacked.
message_id_59439_44525.txt

The message is from yesterday for some reason it was never consumed.

So, if you have set the ack-timeout, this message should be keeping getting resent to the consumer. You can enable debug logs (either on broker or consumer) to check that.
Another option is to get a tcpdump capture of the traffic to the consumer (for port 6650).

@merlimat this messages is effectively being redelivered constantly.
This brings up the question, should pulsar support dead-letter queues?
I don't know what we can use to detect this realtime to prevent this issues.

Just to clarify, is the consumer acknowledging the message?
About how to detect that, one way is to monitor the storage size constantly increasing (though in your case, with retention that is not feasible). Otherwise, monitor the re-delivery rate for the consumers (in the topic stats).
About dead-letter queue. It's an interesting topic and I'd would support that, just not enabled by default. Potentially it could be an option on the namespace itself to define a dead-letter topic where to dump messages after N delivery attempts.
When placing messages there, the broker could attach few properties to the messages themselves.

Just to clarify, is the consumer acknowledging the message?

No, because it's failing to process it.

About how to detect that, one way is to monitor the storage size constantly increasing (though in your case, with retention that is not feasible). Otherwise, monitor the re-delivery rate for the consumers (in the topic stats).

I don't think message re-delivery rate is going to work, messages are potentially constantly being redelivered because they could spend ackTimeout in the internal queue of the consumers.
Plus, we would like to identify which precise messages are being constantly redelivered.

About dead-letter queue. It's an interesting topic and I'd would support that, just not enabled by default. Potentially it could be an option on the namespace itself to define a dead-letter topic where to dump messages after N delivery attempts.
When placing messages there, the broker could attach few properties to the messages themselves.

Of course, dead-letter queues should be optional, but there are also things to consider, should there be a dead-letter topic for each consumer? if not, how would we handle this?

Another thought @merlimat
Would it be possible for brokers to store the value of individuallyDeletedMessages every given time or something, and have new brokers read that value to restore the actual status of that consumer?
The only issue I can think of is that the value of individuallyDeletedMessages could grow forever, maybe we could allow for a given number of unacked messages and then print warnings or errors for that consumer?

@merlimat I'm thinking, what would happen if a consumer like the one we had issues with has a single messages that it cannot process and that message get expired via ttl or retention policies, would broker immediately discard it from individuallyDeletedMessages list.
My concern is, when this message gets expired the broker could advance the markDeletePosition, if it would not do this, when bundle gets unloaded it would reset the consumer back to the beginning, wouldn't it?

Would it be possible for brokers to store the value of individuallyDeletedMessages every given time or something, and have new brokers read that value to restore the actual state of that consumer?

Yes, we thought about that many times. The thing is storing that information, avoids the big backlog on restart but doesn't address the root cause of why the message wasn't acked.
At the end, having a hole in the ack-sequence is always the symptom of a separate problem. And for the system, the data cannot be removed from disk because the storage is not a traditional key-value store where you can point-delete data.
About the size of the individuallyDeletedMessages data itself, the good thing is that we have
introduced a max amount of "unacked" messages that a consumer can have. After that, the broker will stop pushing messages to the consumer. I think default is 50K.
To summarize my point of view

It would be nice to address the ack-hole problem, even if only to a limited degree, in a way that has no performance impact when it's not needed:

eg: only do that if mark-delete falls behind more than X thousand messages compared to read-positition

Give an intuitive and easy to use way to reject messages
Better report in stats when an "ack-hole" is detected, for quicker identification or alerting

My concern is, when this message gets expired the broker could advance the markDeletePosition, if it would not do this, when bundle gets unloaded it would reset the consumer back to the beginning, wouldn't it?

When the broker applies TTL, it will acknowledge the messages and thus close the gap and move the markDeletePosition forward.

Closing this one since the changes are already implemented and merged in #276


Dev

attribute default namespace bug fixed.

Codecov Report

Merging #7 (aeacca8) into master (28c5bf7) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master       #7      +/-   ##
==========================================
+ Coverage   94.47%   94.49%   +0.02%
==========================================
  Files           7        7
  Lines         543      545       +2
==========================================
+ Hits          513      515       +2
  Misses         30       30

Flag
Coverage Δ

unittests
94.49% <100.00%> (+0.02%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pydantic_xml/serializers.py
93.66% <100.00%> (ø)

pydantic_xml/utils.py
93.54% <100.00%> (+0.44%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


[v8]RC1 File Manager Download Error

Select a file in file manager
Select 'Download' from dropdown
Call to a member function isError() on a non-object in
Download fails silently in the UI.
PHP error log shows:
C:\xampp\htdocs\cswea.org\concrete\tools\files\download.php on line 54
This only happens when attempting to download from the File manager search list.
If you attempt to download from the image view it works ok.

I should add that this is using the Download option at the top of the file list - right clicking the image and selecting 'download' from that menu also works ok.


[Fix] update_port_status on OFPPR_DELETE to deactivate an interface

Fixes #20, this addressed the first requirement, the other two ones were broken down to be addressed on issue 39 on topology:
Description of the change
[1.7.0] - 2022-01-04
Changed

Changed update_port_status on OFPPR_DELETE to deactivate an interface instead of deleting it to keep the same object reference that topology uses when managing the status of a link.

Local exploratory test

Before shutting down mininet, all interfaces are active from the topology perspective:

❯ http http://<IP_ADDRESS>:8181/api/kytos/topology/v3/interfaces | jq -r '.interfaces[].active'
true
true
true
true
true
true
true
true
true
true

After shutting down mininet, all interfaces are active from the topology perspective (the only ones UP are OFPP_LOCAL):

❯ http http://<IP_ADDRESS>:8181/api/kytos/topology/v3/interfaces | jq -r '.interfaces[].active'
false
false
true
false
false
false
true
false
false
true

After all switches reconnect again:

❯ http http://<IP_ADDRESS>:8181/api/kytos/topology/v3/interfaces | jq -r '.interfaces[].active'
true
true
true
true
true
true
true
true
true
true

Appreciated your review, Antonio


formula_installer: copy hidden files into bottles.

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example. - I'm getting a Ruby segfault when I run brew tests - sorry!

[ ] Have you successfully run brew tests with your changes locally? - I'm getting a Ruby segfault when I run brew tests - sorry!

Ref: Homebrew/homebrew-core#46252, #5783, Homebrew/homebrew-core#36801, #2792, Homebrew/homebrew-core#14662, Homebrew/legacy-homebrew#31760
When copying etc and var files into a bottle, hidden files and directories are ignored. In particular, this has caused this line of the MariaDB formulae to not have any affect on bottles.
If merged, a rebuild should be performed on all of the MariaDB formulae to fully fix  Homebrew/homebrew-core#46252.

Apologies for having to rely on the CI for brew tests, for the reaons stated above. I did test it locally it on a MariaDB bottle build. I'll try look into what is happening with the test failure.

I should note that, although I said:

If merged, a rebuild should be performed on all of the MariaDB formulae to fully fix Homebrew/homebrew-core#46252.

Rebuilds are not currently possible since the source packages have since been pulled from their site (Homebrew/homebrew-core#46453) but there will be a natural rebuild once the next version of MariaDB is released, due very soon. So please ignore my advice on performing a rebuild for now.

Looks great, nice work here @Bo98, you rock!