Training Time estimation on single GPU A100 80G

Hi
I am pre training the BERT model for FP32,BF16 ,my estimated training time for FP32, 128 sequence length , 256 batch size, is 160 hrs, with mosic bert , is this expected , i don't see much reduction when it compared with hugging face BERT.

Hi @tiru1930, can you please indicate what algorithms you are using when training BERT? We have our BERT example with recipes our examples repo. CC: @jacobfulano and @alextrott16

Hi @tiru1930, can you provide more details on the hardware you are running and the throughput measurements? Also is there a specific reason you are using fp32 instead of bf16? Feel free to reach out over the community slack

Hi
It's nividia A100  GPU 80G , I need to have fb32,bf16 and quantized model
for our own hardware validation and comparisons.
Regards
Tiru
On Thu, May 18, 2023, 10:57 PM jacobfulano @.***> wrote:

Hi @tiru1930 https://github.com/tiru1930, can you provide more details
on the hardware you are running and the throughput measurements? Also is
there a specific reason you are using fp32 instead of bf16? Feel free to
reach out over the community slack
https://join.slack.com/t/mosaicml-community/shared_invite/zt-1btms90mc-GipE2ufuPkKY0QBrmF3LSA
—
Reply to this email directly, view it on GitHub
https://github.com/mosaicml/examples/issues/331#issuecomment-1553381311,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AC5FIVY7FDFMQPFFTUNLTXDXGZLZNANCNFSM6AAAAAAYGENWI4
.
You are receiving this because you were mentioned.Message ID:
@.***>

@tiru1930 Unfortunately, we will need more detail to proceed. I'm not sure:

what you mean by quantized model given we're discussing training
can you please provide concrete throughput measurements that we can compare against our numbers to verify?

@mvpatel2000

We are building new GPU, so for comparison purpose we want to train BERT model on NVIDIA GPU with different configurations as i mentioned FP32,FB16,BF16 and INT8 etc.
I don't understand what is throughput here, my end goal is to Train the full BERT model with Mosaic settings on Single GPU. Since it was mentioned on 8GPUS it took 4.5 hrs, I am expecting it to be completed in 36 hrs in single GPU

@tbaggu
Throughput is defined as how many samples a model processes per second. If you are using a custom accelerator (I'm not sure what you mean by building a new GPU), it is hard for us to help you as it could be the accelerator is slower.

Is This MFU calculated  using the same throughput? do we have any API to
capture this number ?
I can get number of tokens processed per sec ,  can give this thing to MFU
calculation
In below formula
MFU = 6 * observed throughput  /  therictial
On Tue, May 30, 2023, 11:30 PM Mihir Patel @.***> wrote:

@tbaggu https://github.com/tbaggu
Throughput is defined as how many samples a model processes per second. If
you are using a custom accelerator (I'm not sure what you mean by building
a new GPU), it is hard for us to help you as it could be the accelerator is
slower.
—
Reply to this email directly, view it on GitHub
https://github.com/mosaicml/examples/issues/331#issuecomment-1568849268,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AC5FIVYKNH34R4FY26DMMZ3XIYYULANCNFSM6AAAAAAYGENWI4
.
You are receiving this because you were mentioned.Message ID:
@.***>

It should be a reasonable approximation and is correct in limit of 0 sequence length. As the sequence length grows, the approximation becomes worse.
We have the exact formula here assuming you only have linear layers and attention

Ok thank you, will check it out.
On Fri, Jun 30, 2023, 10:52 PM Mihir Patel @.***> wrote:

It should be a reasonable approximation and is correct in limit of 0
sequence length. As the sequence length grows, the approximation becomes
worse.
We have the exact formula here
https://github.com/mosaicml/llm-foundry/tree/main/scripts/train/benchmarking#mfu
assuming you only have linear layers and attention
—
Reply to this email directly, view it on GitHub
https://github.com/mosaicml/examples/issues/331#issuecomment-1614952526,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AC5FIV7MINX3QQ6VISO2LCLXN4DMLANCNFSM6AAAAAAYGENWI4
.
You are receiving this because you were mentioned.Message ID:
@.***>


How to remove utc at the end

How to remove utc at the end

I'm sorry for the late response.
Sorry, I did not understand the intent of this ticket.
Could you please explain in more detail?


Misc

Find out why functional tests are failing for widget tests

Codecov Report

Merging #5617 (fff7e86) into main (f4b5352) will increase coverage by 0%.
The diff coverage is n/a.

:exclamation: Current head fff7e86 differs from pull request most recent head 41b29a0. Consider uploading reports for the commit 41b29a0 to get more accurate results

@@          Coverage Diff          @@
##            main   #5617   +/-   ##
=====================================
  Coverage     68%     69%
=====================================
  Files        401     401
  Lines      26708   26708
  Branches    3895    3895
=====================================
+ Hits       18395   18443   +48
+ Misses      6798    6734   -64
- Partials    1515    1531   +16

Impacted Files
Coverage Δ

...lient/datascience/kernel-launcher/kernelProcess.ts
66% <0%> (-5%)
:arrow_down:

src/client/api/pythonApi.ts
57% <0%> (-4%)
:arrow_down:

...nt/datascience/editor-integration/hoverProvider.ts
84% <0%> (-3%)
:arrow_down:

src/client/common/utils/decorators.ts
76% <0%> (-2%)
:arrow_down:

...datascience/editor-integration/codelensprovider.ts
85% <0%> (-2%)
:arrow_down:

...tascience/jupyter/liveshare/hostJupyterNotebook.ts
44% <0%> (-2%)
:arrow_down:

src/client/datascience/telemetry/telemetry.ts
77% <0%> (-2%)
:arrow_down:

src/client/datascience/jupyter/kernels/helpers.ts
73% <0%> (-2%)
:arrow_down:

...ient/datascience/editor-integration/codewatcher.ts
69% <0%> (-1%)
:arrow_down:

src/client/datascience/jupyter/jupyterNotebook.ts
74% <0%> (-1%)
:arrow_down:

... and 7 more


Unique contraints cause duplicate fields when using postgres

If you have a column that is a primary key and also has a unique constraint that column will be exported twice in the output. We don't need to worry about the unique constraint so it will need to be ignored

I have traced the issue and have a fix for it (I can make a pull request). The issue is caused by this left join:
        LEFT JOIN pg_index i ON
            (pgc.oid = i.indrelid AND i.indkey[0] = a.attnum)

This causes two effects:

Compound primary keys don't work
If the field is repeated in another index you will get a row for both primary_key: true and primary_key: false (which now generates the field twice).

The solution I have is just to query the primary key(s) in another query otherwise the SQL gets too complicated.


create tables

Start using sqlalchemy
note this pr will fail until we update the terraform to make the db publicly available

I am merging this now, and will update this in the refactor.


[CT-3228] [feature] dbt docs generate --defer should work on catalog.json

Is this a new bug in dbt-core?

[X] I believe this is a new bug in dbt-core
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

In development environment, I run models in my own schema in the database.
The models not present in my schema are deferred from a common schema public. We do this using dbt run -s model --defer --state public-artifacts (Assuming public-artifacts contain the public-schema's docs)

Now, running dbt docs generate will only add those nodes in catalog.json which are present in my schema. This is expected.
But I want the nodes not present in my schema to be deferred from public schema while generating docs as well.
To do this, I tried dbt docs generate --defer --state public-artifacts, but the catalog generated was exactly the same as generated without supplying --defer
Expected Behavior
dbt docs generate --defer --state DEFERRED_STATE should work for catalog.json as well.
Steps To Reproduce

Run dbt docs generate, and store the catalog.json output.
Run dbt docs genreate --defer --state SOME_DEFERRED_STATE, and store the catalog.json output.
Compare both, they will be similar.

Relevant log output
No response
Environment
- OS MAC OS:
- Python 3.11:
- dbt 1.6.6:

Which database adapter are you using with dbt?
snowflake
Additional Context
No response

@anshjain18 Thanks for opening!
With the latest changes in v1.7, this is the behavior of dbt docs generate + --defer + --select/--exclude:

If a model is selected, it is not deferred. (This is how --defer has always worked.)
If a model is excluded, it is not catalogued. (This is new behavior in v1.7)

I don't think we're going to support the use case as outlined. Instead, I'd recommend:

[happy path] Running dbt clone with deferral/state to create a full "dev" schema first, and then running dbt docs generate
[possible] Write some code to merge together catalog.json from different environments :)


C++ Header files missing

Solution to issue cannot be found in the documentation.

[X] I checked the documentation.

Issue
Sorry if I've missed something obvious here...
I have an old project that follows the instructions here: https://awkward-array.org/doc/stable/user-guide/how-to-use-header-only-layoutbuilder.html
This used to work fine with older conda builds of awkward but I've just recently tried to get it going again and I'm not having any luck. Not only does python -m awkward.config --cflags fail (no awkward.config) there doesn't seem to be any builder headers installed at all anymore.
I understand the code has changed quite a lot recently, so maybe building awkward objects in C++ no longer supported?
I wasn't sure if this issue belonged in awkward-feedstock or awkward-cpp-feedstock but since this is a C++ issue I went with this one.
Installed packages
# packages in environment at /home/sam/miniconda3/envs/awktest:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
awkward                   2.0.8              pyhd8ed1ab_0    conda-forge
awkward-cpp               9               py311ha3edf6b_0    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.12.7            ha878542_0    conda-forge
importlib_resources       5.12.0             pyhd8ed1ab_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libblas                   3.9.0           16_linux64_openblas    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgfortran-ng            12.2.0              h69a702a_19    conda-forge
libgfortran5              12.2.0              h337968e_19    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
liblapack                 3.9.0           16_linux64_openblas    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
libsqlite                 3.40.0               h753d276_0    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
numpy                     1.24.2          py311h8e6699e_0    conda-forge
openssl                   3.0.8                h0b41bf4_0    conda-forge
packaging                 23.0               pyhd8ed1ab_0    conda-forge
pip                       23.0.1             pyhd8ed1ab_0    conda-forge
python                    3.11.0          he550d4f_1_cpython    conda-forge
python_abi                3.11                    3_cp311    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
setuptools                67.4.0             pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
typing_extensions         4.4.0              pyha770c72_0    conda-forge
tzdata                    2022g                h191b570_0    conda-forge
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zipp                      3.15.0             pyhd8ed1ab_0    conda-forge

Environment info
active environment : awktest
    active env location : /home/sam/miniconda3/envs/awktest
            shell level : 3
       user config file : /home/sam/.condarc
 populated config files : /home/sam/.condarc
          conda version : 4.11.0
    conda-build version : not installed
         python version : 3.9.7.final.0
       virtual packages : __cuda=11.6=0
                          __linux=5.19.0=0
                          __glibc=2.35=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/sam/miniconda3  (writable)
      conda av data dir : /home/sam/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/sam/miniconda3/pkgs
                          /home/sam/.conda/pkgs
       envs directories : /home/sam/miniconda3/envs
                          /home/sam/.conda/envs
               platform : linux-64
             user-agent : conda/4.11.0 requests/2.27.1 CPython/3.9.7 Linux/5.19.0-32-generic ubuntu/22.04.2 glibc/2.35
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

@gillins thanks for the report! This is a mistake on our part; at one point we were going to bundle these header files with the Awkward package, but we soon decided that they should be taken directly from our repo, as they are version independent of the awkward packages.
The recommended way to get these headers now is to copy them from the repo: https://github.com/scikit-hep/awkward/tree/main/header-only
Let me know how this goes!

Thanks @agoose77 ! I'll have a go at updating my code and see if I can get it running.

Thanks @gillins. I discussed this @jpivarski today, and we plan to publish these headers as a zip archive alongside a CMakeLists.txt such that the ExternalProject or add_submodule approaches work out of the box, hopefully making this less painful for CMake users. I do not know whether this applies to you, but in any case, we can better publish these.

Yes this applies to me! Sounds very useful. One other thing i've just noticed - the link to more examples is broken here: https://awkward-array.org/doc/main/user-guide/how-to-use-header-only-layoutbuilder.html#more-examples

Excellent, one fewer bugs in the wild! I'll make a PR, thank you! :)

One more question @agoose77 - is there some more documentation or an example on returning an awkward array to Python? Previously, I called builder.snapshot() to return a awkward::ContentPtr and this was all that was required. The documentation mentions something about allocating memory and using py::buffer_protocol, but I must admit you lost me here. Surely there is a relatively easy way where I don't have to learn too much about the details? Thanks!

N.b. that we're exploring a rework of how third-party users include our header-only libraries in this branch.
The problem of allocating memory is to ensure that Python owns the buffers. You can return a view over the memory allocated by to_buffers using the buffer protocol or a memory view. The former is advantageous in that the Python user doesn't need to know details about the array type, etc.
However, these arrays are still owned by C++, so their lifetimes are tied to the scope of the C++ program. It's perhaps better in this case to move ownership of the C++ buffers to Python, e.g. https://github.com/pybind/pybind11/issues/2271#issuecomment-650569067
This usually involves some kind of capsule object to free the allocated memory when it goes out of Python scope. I'll do some more digging.

Here's a demo using scikit-build-core. If you are not familiar with this Python build backend, don't worry; all the important information is in demo.cpp.

One more question @agoose77 - is there some more documentation or an example on returning an awkward array to Python? Previously, I called builder.snapshot() to return a awkward::ContentPtr and this was all that was required. The documentation mentions something about allocating memory and using py::buffer_protocol, but I must admit you lost me here. Surely there is a relatively easy way where I don't have to learn too much about the details? Thanks!

ArrayBuilder is glued to Python with pybind11, but RDataFrame's use of it goes through ROOT's cppyy, and some of the potential users of LayoutBuilder wanted to use Cython. (And if fastjet, which preceeded this, is ever updated to take advantage of it, that one is glued to Python with SWIG.) So we wanted LayoutBuilder to be independent of which Python-C++ glue you use.
For now, I have some instructions on how to do it with pybind11 in https://github.com/impy-project/chromo/issues/65#issue-1374710755. It's the 4-step process starting at

Moreover, we want NumPy to own the array buffers

More complete instructions for at least two backends would be better, I know. I'll put that on my mental to-do list.

@jpivarski how would you feel about creating an examples directory and placing a near-copy of demo.zip there? It needs to be modified for FetchContent, but that can happen once 2.1.0 is out.
Then we could also add a Cython backend

Could it be nested under docs? If that means that this 2 kB ZIP file gets pushed to every version on the documentation server, that's fine.

It could be! To be clear, though, I'd unzip it and make it a subdir so people can browse the code. Do you have other reasons to keep it a zipfile, or is it just from my wording?

By your wording. I thought it was a zipfile for convenience of users downloading it and trying it out.

My thought was that we create a subdir of the Git repo that can be linked to from the documentation. This would not be easily versioned alongside the documentation (unless we use the {eval} role from MyST and pass in the headers version to a define for a C++ kernel), but it would provide an example file-system layout. A benefit of this would be that we could also run it in the test suite to ensure that it builds.
We could equally just list the files and their contents in-line.

Thanks @agoose77 that demo was very useful! Any chance of an example going in the other direction (awkward array created in Python, being passed to C++)?

@gillins could you elaborate on what you'd like to do there? The only C++ interfaces that we provide for manipulating Awkward Arrays are the RDataFrame headers, which are intended to be jitted. I suppose that if you know the form ahead of time, you could pre-generate the C++ interfaces intended for RDataFrame, though that's not how we originally intended them to be used.
I think our policy for manipulating Awkward Arrays from C++ is either

Use Python from C++
Operate on the flat buffers according to your needs

Option (2) is the fastest, but most restrictive. You'd most likely define what kind of array your C++ can work with, e.g. a ragged array of integers, and accept that from the user. If you want to make the user-facing API very high level, you could use the Python API to validate the array form, and then convert it to buffers for use in your application (or, instead of to_buffers, flatten it and use ak.num to compute the sublist lengths).
@jpivarski might have more to add here depending upon your use case.

I guess I was after something like Numpy provides where I could reach into the awkward array and get values out. Obviously in C++ I'd have to cope with each data type and/or layout explicitly or bail if not what expected but this is common with Numpy.
Next best would be to_buffers - (if you have an example of this that'd be great) but this would involved copying (?) so potentially slow...

We provide an interface to Numba for fast iteration over Awkward Arrays. Unfortunately we do not offer the same for C++. Perhaps @ianna can weigh in with how feasible it would be to use the RDataFrame generator to build a C++ module to compile. It's certainly possible at a technical level, though I don't know how tied to ROOT it is right now. But, as I say, being able to do something is distinct from whether one should do it.

Next best would be to_buffers - (if you have an example of this that'd be great) but this would involved copying (?) so potentially slow...

to_buffers is "fast" - It should never involve a copy (unless you request a different byteorder). This would produce a series of flat arrays, and you'd use the metadata arrays (indices, offsets, etc) to figure-out how to pull out the values of interest.
It might be helpful to explain the use cases that we might better be able to offer you advice, though :)

Which to_buffers? If you mean the Python function,
https://awkward-array.org/doc/main/reference/generated/ak.to_buffers.html
there are no copies. The buffers this function returns are references to the data in the Awkward Array itself. (Modify them and the Awkward Array will be modified...)
If you mean the C++ method of LayoutBuilder,
https://github.com/scikit-hep/awkward/blob/7fc01d81398dd3ba7bef653768fd6d54d42a38a3/header-only/layout-builder/awkward/LayoutBuilder.h#L135-L144
it copies. Several versions ago, the builder → Python method was a "snapshot" that shared buffers, so that "snapshot" would be a lightweight operation and the builder could continue to build (append only) while an array is in use. However, (a) that required the builder to work through std::shared_ptr, rather than std::unique_ptr, which was a significant slowdown, (b) shareable buffers must be contiguous, so they had to reallocate-and-copy every time they reached an (exponentially larger) limit; replacing the no-copy snapshot with a copy allowed us to build an array non-contiguously and do the concatenation during the copy-to-arrays. So the fact that it does copy was motivated by speed, in two different ways.
The ak.from_rdataframe, ak.to_rdataframe functions are complete and ready for users. The latter is strictly zero-copy: you can iterate over an Awkward Array in C++ by creating an RDataFrame out of it. The Awkward data types have unguessable type names, but they appear in places where you can use auto.
I didn't see a tutorial on it in our documentation website, but these tests cover every data type and @ianna presented an interactive tutorial on it at PyHEP (video).

Yes, https://awkward-array.org/doc/main/reference/generated/ak.to_buffers.html - I didn't realise that there was no copy involved - that's perfect. I'll have a go at this - thanks.
Use case is Satellite Image Segmentation which results in data being put into a Raster Attribute Table for the segments. We normally access this in a format independent way using GDAL's functions which have Python/Numpy wrappers. However, the KEA format has some other functionality that GDAL doesn't support that I'd like to make available to Python users. What I'm using awkward for is the "neighbours" functionality - each segment can have a list of neighbours that can vary greatly in length. So I'm creating functions to read and write neighbours from/to a KEA file for Python users. People end up writing code to crawl around a landscape working out how the segments (and their other attributes) relate to one another.
I've been really impressed with how well awkward arrays work for this use case so I'm keen to push through with this - I really appreciate your help and the hard work getting awkward to this point. The Numba support is really important for us so thanks for putting that in there!

Actually, I'd like to follow your project and explicitly help out. One of our major goals is to expand Awkward Array beyond particle physics, and this looks like a great match. (Most of the non-physics applications so far have been geographical. That's probably a hint we should pay attention to!)
If you want to ping me on Awkward issues in pyshepseg as they come up, I may be able to help. Incidentally, I worked on a terrain-classification project with hyperspectral images and GDAL about 10 years ago, though we were trying to glue GDAL to Java for Hadoop; Python is easier.

Thanks @jpivarski will ping you guys when I have something ready for review. Yes Python is easier!


feat: add single deposition info panel

#926

fix/refactor various components
add info panel
add deposition overview metadata table
add annotations summary metadata table
add annotations methods summary metadata table

https://github.com/user-attachments/assets/e78dd5b3-4555-4b34-895b-7f93beb74ff7

live link: https://dev-kira-1.cryoet.dev.si.czi.technology/depositions/10083?metadata=deposition&tab=metadata

noticed the table goes into a loading state when clicking "View All Info", maybe a shouldRevalidate() missing somewhere?

Note: if feasible, updating other panels for consistency is something I can brink up in upcoming updates to the other panels.

does this mean ALL info panels will have this column sizing update? asking bc it would be easier to change all of them at once than just this one

@kne42 AH, in that case I would like to change them all if it is feasible to make that change.

@Janeece just checked this and it looks like in the code the first column is set to a static 170px and the second is fill remainder with a min width of 170px as well. i think the reason has to do with the info panel's size, would you be able to give me the constraints of the info panel sizing behaviour?

ok it was a bug, things should be fixed now, can you check the link again?


Implement customizable serializer

Pre-requirement for https://github.com/taskiq-python/taskiq-pipelines/issues/13
The main idea is:

JSONFormatter exists for the backward compatibility.
ProxyFormatter is used by default, which operates over broker.serializer API
The default serializer is JSONSerializer which is a thin wrapper around plain json.dumps() and json.loads()
TaskiqSerializer base class works with bytes, not str. The broker message type is a bytes anyway, plus formats like msgpack and cbor work with bytes anyway.

I can implement cbor and msgpack serializers as well if you are ok with optional dependencies (say, cbor2 is required only if a user explicitly imports taskiq.serializers.cbor_serializer:CBORSerializer.

To fix mypy please consider running poetry update.
To fix failing tests, make them async and mark with pytest.mark.anyio.  This problem happens because initialize semaphore in InmemoryBroker. This works in py3.10 and above, but doesn't work for older versions. We will fix it.

tests should be green now, at least locally I don't see any problems

Codecov Report

Merging #214 (c198624) into develop (e66f3aa) will increase coverage by 5.65%.
Report is 131 commits behind head on develop.
The diff coverage is 73.28%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
@@             Coverage Diff             @@
##           develop     #214      +/-   ##
===========================================
+ Coverage    67.62%   73.27%   +5.65%
===========================================
  Files           37       51      +14
  Lines          942     1553     +611
===========================================
+ Hits           637     1138     +501
- Misses         305      415     +110

Files
Coverage Δ

taskiq/__init__.py
100.00% <100.00%> (ø)

taskiq/abc/formatter.py
100.00% <100.00%> (ø)

taskiq/abc/middleware.py
100.00% <ø> (ø)

taskiq/abc/result_backend.py
100.00% <ø> (ø)

taskiq/abc/schedule_source.py
100.00% <100.00%> (ø)

taskiq/abc/serializer.py
100.00% <100.00%> (ø)

taskiq/acks.py
100.00% <100.00%> (ø)

taskiq/api/__init__.py
100.00% <100.00%> (ø)

taskiq/cli/common_args.py
100.00% <100.00%> (ø)

taskiq/cli/watcher.py
0.00% <ø> (ø)

... and 37 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Since these mypy problems are totally minor, we can merge it now. I'll fix it later.

Thank you very much for your contribution.

 The next step is PR for taskiq-pipelines

Could you please consider publishing a new release?
It is a pre-requirement for conversion taskiq-pipelines to use new serializers.

Sure.

Changes available in https://github.com/taskiq-python/taskiq/releases/tag/0.9.3.


Run Benchmark against Ajuna Infrastructure

@masapr @brenzi How should we continue with this? Is it still relevant for the current sprint or should we postpone it?

I moved the issue to the product backlog. It will not be adressed before the infrasstructure is setup


TypeConverter doesn't work properly in View

Hi,
I have two tables DriftMachineTable, DriftMachineInfoTable. I use TypeConverters for Main and Details classes in DriftMachineInfoTable. When I create a view that includes those tables, it doesn't  use TypeConverter properly.
@UseRowClass(DriftMachineInfoEntity)
class DriftMachineInfoTable extends Table {
  @override
  String get tableName => 'machine_info';

  IntColumn get machineId => integer().named("machineId")();
  TextColumn get main => text().named("main").map(const DriftMainConverter()).nullable()();
  TextColumn get details => text().named("details").map(const DriftDetailsConverter()).nullable()();

  @override
  Set<Column>? get primaryKey => {machineId};
}

@UseRowClass(DriftMachineEntity)
class DriftMachineTable extends Table {
  @override
  String get tableName => 'machine';

  IntColumn get id => integer()();
  IntColumn get typeLutId => integer().named("typeLutId").nullable()();
  IntColumn get actorId => integer().named("actorId").nullable()();
  IntColumn get modelId => integer().named("modelId").nullable()();
  IntColumn get profileLutId => integer().named("profileLutId").nullable()();
  TextColumn get name => text().named("name").nullable()();
  TextColumn get labels => text().named("labels").map(const DriftStringListConverter()).nullable()();

  @override
  Set<Column>? get primaryKey => {id};
}

DriftMachineView
abstract class DriftMachineView extends View {

  DriftMachineTable get machineTable;
  DriftMachineInfoTable get machineInfoTable;

  Expression<int?> get id => machineTable.id;
  Expression<int?> get typeLutId => machineTable.typeLutId;
  Expression<int?> get actorId => machineTable.actorId;
  Expression<int?> get modelId => machineTable.modelId;
  Expression<int?> get profileLutId => machineTable.profileLutId;
  Expression<String?> get name => machineTable.name;
  Expression<String?> get labelsTestConverter => machineTable.labels;

  // machine info
  Expression<String?> get main => machineInfoTable.main;
  Expression<String?> get details => machineInfoTable.details;

  @override
  Query as() => select([
        id,
        typeLutId,
        actorId,
        modelId,
        profileLutId,
        name,
        machineInfoTable.main,
        machineInfoTable.details,
      ]).from(machineTable).join([
        leftOuterJoin(machineInfoTable, machineInfoTable.machineId.equalsExp(machineTable.id)),
      ]);
}

Generated DriftMachineViewData from DriftMachineView.
Main and Details are String types, but it supposed to be:
Main? main;
Details? details

class DriftMachineViewData extends DataClass {
  final int? id;
  final int? typeLutId;
  final int? actorId;
  final int? modelId;
  final int? profileLutId;
  final String? name;
  final String? main;
  final String? details;
  DriftMachineViewData(
      {this.id,
      this.typeLutId,
      this.actorId,
      this.modelId,
      this.profileLutId,
      this.name,
      this.main,
      this.details});
  factory DriftMachineViewData.fromData(Map<String, dynamic> data,
      {String? prefix}) {
    final effectivePrefix = prefix ?? '';
    return DriftMachineViewData(
      id: const IntType().mapFromDatabaseResponse(data['${effectivePrefix}id']),
      typeLutId: const IntType()
          .mapFromDatabaseResponse(data['${effectivePrefix}type_lut_id']),
      actorId: const IntType()
          .mapFromDatabaseResponse(data['${effectivePrefix}actor_id']),
      modelId: const IntType()
          .mapFromDatabaseResponse(data['${effectivePrefix}model_id']),
      profileLutId: const IntType()
          .mapFromDatabaseResponse(data['${effectivePrefix}profile_lut_id']),
      name: const StringType()
          .mapFromDatabaseResponse(data['${effectivePrefix}name']),
      main: const StringType()
          .mapFromDatabaseResponse(data['${effectivePrefix}main']),
      details: const StringType()
          .mapFromDatabaseResponse(data['${effectivePrefix}details']),
    );
  }

I couldn't reproduce this with an example view defined in drift's sources. For that it generated the column with the type converter if I select the column with a table reference (as opposed to a column defined explicitly with a getter on the view class). But it looks like you're doing the same thing. It could be that this isn't working in all cases, or that I accidentally fixed this in the current development version that's not released yet :D
For me to take another look at your exact example, could you post a small but complete repro maybe (something including the type converter definitions)? Something I could ideally just copy and run the generator on to see where exactly it's going wrong in your case.

Thanks for replying back,
I created repository with example above :)
https://github.com/OlegShNayax/drift_view_converter_example
I use dependency_overrides in this repository in order to use the last fix of view class (https://github.com/simolus3/drift/commit/6105be75a3d4b5c795b7648c8a90f724ea102cb7)

dependency_overrides:
  drift:
    git:
      url: https://github.com/simolus3/drift.git
      ref: 6105be75a3d4b5c795b7648c8a90f724ea102cb7
      path: drift
  drift_dev:
    git:
      url: https://github.com/simolus3/drift.git
      ref: 6105be75a3d4b5c795b7648c8a90f724ea102cb7
      path: drift_dev
  sqlparser:
    git:
      url: https://github.com/simolus3/drift.git
      ref: 6105be75a3d4b5c795b7648c8a90f724ea102cb7
      path: sqlparser


pull request for e2e test at 11/28/2016 06:26:04

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Update README: urvliew must be installed already.

Clarify that this only works if you already have urlview, which was not clear to me at first!

Thanks for the effort! However, I feel this is already covered by the dependencies section.

oh, huh, I completely missed that section -- maybe because it's right under the video, I just glossed over it. Thanks.


Export engaged Citizens list

As City Staff,
I want the ability to export a list of engaged Citizens and their contact information,
so that I can contact these users in the future regardless of the platform I use.
From CETF Report, p 15:
"Meanwhile, those who manage or implement engagement efforts express
concerns about “reinventing the wheel” each time they launch an effort, and that there is no citywide database of people who want to engage on civic issues and what they care about. The Community Registry could be a useful tool but it is not being managed effectively or kept current."

we need to discuss privacy issues here


runtime, cmd/compile: consider write barrier short-circuit when *dst == src

I did a quick and dirty instrumentation of writebarrierptr and found that when executing cmd/compile to build std, about 25% of calls on average had *dst == src. In that case, there's no need to do the actual assignment or gray any objects (I think). I don't know how (a)typical that 25% number is.
We should investigate whether checking for this and short-circuiting is a net performance gain, in general. There are multiple places in the stack this could occur:
(1) writebarrierptr (and friends)
Add
if *dst == src {
  return
}

to the beginning of the method.
(2) Wrapper routines
Add checks like:
if dst[1] != src[1] {
	writebarrierptr(&dst[1], src[1])
}

to the runtime routines that call writebarrierptr, like the wbfat.go routines, and writebarrierstring and friends. This is different than (1) insofar as it skips the writebarrierptr function call instead of having it return immediately.
(3) Generated code
We currently generate wb-calling code like:
if writeBarrier.enabled {
  writebarrierptr(dst, src)
} else {
  *dst = src
}

We could instead generate code like:
if *dst != src {
  if writeBarrier.enabled {
    writebarrierptr(dst, src)
  } else {
    *dst = src
  }
}

cc @aclements @randall77
I don't plan to work on this soon, as I need to undistract myself.

Just for the kicks, I tried that, showed some mixed results for go test -v -bench=. -run=NONE runtime:
BenchmarkGrowSliceBytes-8        33.3          32.4          -2.70%
BenchmarkGrowSliceInts-8         58.0          59.4          +2.41%
BenchmarkAppend-8                18.6          17.4          -6.45%
BenchmarkAppendGrowByte-8        1903243       1920778       +0.92%
BenchmarkAppendGrowString-8      278920100     143081182     -48.70% # << this is really interesting
BenchmarkAppend1Byte-8           3.13          3.11          -0.64%
BenchmarkAppend4Bytes-8          3.08          3.18          +3.25%
BenchmarkAppend7Bytes-8          3.12          3.19          +2.24%
BenchmarkAppend8Bytes-8          2.79          3.04          +8.96%
BenchmarkAppend15Bytes-8         4.03          4.27          +5.96%
BenchmarkAppend16Bytes-8         4.28          4.18          -2.34%
BenchmarkAppend32Bytes-8         3.16          3.38          +6.96%
BenchmarkAppendStr1Byte-8        3.10          3.62          +16.77%
BenchmarkAppendStr4Bytes-8       3.05          3.23          +5.90%
BenchmarkAppendStr8Bytes-8       3.06          2.83          -7.52%
BenchmarkAppendStr16Bytes-8      4.18          4.00          -4.31%
BenchmarkAppendStr32Bytes-8      3.45          4.38          +26.96%
BenchmarkAppendSpecialCase-8     18.3          19.3          +5.46%
BenchmarkCopy1Byte-8             3.86          4.47          +15.80%
BenchmarkCopy2Byte-8             3.93          4.20          +6.87%

Want me to prep a CL?

By "that", do you mean (1), (2), or (3)?

We have to be careful not to run afoul of the issue in #14855 .  I don't think this optimization would cause that problem, but it probably deserves a closer look.

I tried 2, modified writebarrierptr itself.
I'll mess with it later and test against the case in #14855.

I posted a CL21027 with the benchstat.

Thinking out loud, one interesting aspect of modifying the calling code (3 above) is that it might allow other optimization passes to remove the code entirely by proving that *dst == src. I guess that depends on whether *dst is actually a load or is an ssa.Value.

The benchmark results from CL 21027 suggest that append may be the main source of idempotent pointer updates. Josh, do you happen to know what was one level up from the *dst==src write barriers? If it's mostly append, I suspect we can optimize just append to avoid the unnecessary pointer updates altogether.
One potential downside of modifying writebarrierptr this way is that it can cause extra coherence traffic by first pulling the pointer's cache line into shared and then upgrading it to modified. Though I doubt that's what causing the slowdowns shown in the CL commit message since those benchmarks are sequential.
I don't think these changes run afoul of the races in #14855. In that case we were doing the pointer write, but not doing the write barrier. It should be safe if we don't do either.
If we go the route of modifying the calling code, it would probably be better to make it:
if writeBarrier.enabled {
  if *dst != src {
    writebarrierptr(dst, src)
  }
} else {
  *dst = src
}

so we don't penalize the code with an extra (and potentially poorly predictable) branch when write barriers are off.

@aclements I'm trying something like that, I'll let you know what happens after the benchmarks finish.

I did a bit of experimenting. All of these results are for 4c9a470 (two days ago; forgot to sync).
First, I reproduced @josharian's result using the following patch:
diff --git a/src/runtime/mbarrier.go b/src/runtime/mbarrier.go
index 523d890..15244f8 100644
--- a/src/runtime/mbarrier.go
+++ b/src/runtime/mbarrier.go
@@ -14,6 +14,7 @@
 package runtime

 import (
+       "runtime/internal/atomic"
        "runtime/internal/sys"
        "unsafe"
 )
@@ -123,10 +124,19 @@ func writebarrierptr_nostore1(dst *uintptr, src uintptr) {
        releasem(mp)
 }

+var wb, wbe, wbz uint64
+
 // NOTE: Really dst *unsafe.Pointer, src unsafe.Pointer,
 // but if we do that, Go inserts a write barrier on *dst = src.
 //go:nosplit
 func writebarrierptr(dst *uintptr, src uintptr) {
+       if src == 0 {
+               atomic.Xadd64(&wbz, +1)
+       } else if *dst == src {
+               atomic.Xadd64(&wbe, +1)
+       }
+       atomic.Xadd64(&wb, +1)
+
        *dst = src
        if writeBarrier.cgo {
                cgoCheckWriteBarrier(dst, src)
diff --git a/src/runtime/proc.go b/src/runtime/proc.go
index d386797..b146fa4 100644
--- a/src/runtime/proc.go
+++ b/src/runtime/proc.go
@@ -208,6 +208,7 @@ func main() {
 // os_beforeExit is called from os.Exit(0).
 //go:linkname os_beforeExit os.runtime_beforeExit
 func os_beforeExit() {
+       println("WB", wb, wbe, wbz)
        if raceenabled {
                racefini()
        }

I get this result:
$ go build -a |& awk '/^WB/{n+=$2; eq+=$3; z+=$4} END{print n " " eq " " z}'
14525087 3412554 1422002

That is, of 14M write barriers, 10% are writing nil and 23.5% are *dst == src (but not writing nil). I separately checked that almost all of the 10% nil writes also have *dst == src.
We should probably skip the write barrier earlier if src is nil. Currently we go through quite a bit of work before checking for a nil pointer.
Focusing on the ones where *dst == src, but src != nil, I profiled where they're coming from using this patch:
diff --git a/src/runtime/mbarrier.go b/src/runtime/mbarrier.go
index 523d890..e8ce821 100644
--- a/src/runtime/mbarrier.go
+++ b/src/runtime/mbarrier.go
@@ -127,6 +127,13 @@ func writebarrierptr_nostore1(dst *uintptr, src uintptr) {
 // but if we do that, Go inserts a write barrier on *dst = src.
 //go:nosplit
 func writebarrierptr(dst *uintptr, src uintptr) {
+       if src != 0 && *dst == src {
+               cpc := getcallerpc((unsafe.Pointer)(&dst)) - 1
+               f := findfunc(cpc)
+               file, line := funcline(f, cpc)
+               print("WBE ", file, ":", line, " ", funcname(f), "\n")
+       }
+
        *dst = src
        if writeBarrier.cgo {
                cgoCheckWriteBarrier(dst, src)

The result is
$ go build -a std |& awk '/^WBE/{print $2}' logx | sort | uniq -c | sort -nr | head -n 10
  16519 /home/austin/go.dev/src/cmd/compile/internal/gc/typecheck.go:176
  15369 /home/austin/go.dev/src/cmd/compile/internal/gc/dcl.go:58
  15155 /home/austin/go.dev/src/cmd/compile/internal/ssa/sparseset.go:45
  12066 /home/austin/go.dev/src/cmd/compile/internal/gc/dcl.go:753
   8865 /home/austin/go.dev/src/cmd/compile/internal/ssa/value.go:167
   8493 /home/austin/go.dev/src/cmd/compile/internal/ssa/func.go:94
   8206 /home/austin/go.dev/src/cmd/compile/internal/ssa/stackalloc.go:266
   7828 /home/austin/go.dev/src/runtime/mheap.go:603
   7288 /home/austin/go.dev/src/runtime/mheap.go:196
   6871 /home/austin/go.dev/src/runtime/hashmap.go:793

Of these, all but dcl.go:58, dcl.go:753, mheap.go:603, and hashmap.go:793 are appends. @randall77, can we optimize the generated append code so that the path that grows the slice in place doesn't write back the pointer if we're writing to the same slice we're growing? I imagine this would be something like the slicing optimization.
We can safely skip the write barrier on mheap.go:603. I'm a little surprised that does equal pointer writes so often. There may be also be an algorithmic improvement here.
We could put a conditional around the one in hashmap.go. I think that happens when we're iterating over a map and the iterator stays in the same bucket. I don't totally understand this code; there may be better ways to improve it.

I accidentally uploaded a 2nd CL, https://go-review.googlesource.com/#/c/21138/
➜ benchcmp /tmp/master.txt /tmp/wbfat.txt | grep -v 0\. # just show changes > +-0.x%
benchmark                        old ns/op     new ns/op     delta
BenchmarkGrowSliceBytes-8        35.4          32.1          -9.32%
BenchmarkAppend1Byte-8           3.16          3.11          -1.58%
BenchmarkAppend8Bytes-8          2.79          2.69          -3.58%
BenchmarkAppendStr8Bytes-8       3.38          3.23          -4.44%
BenchmarkAppendStr32Bytes-8      3.23          3.18          -1.55%
BenchmarkAppendSpecialCase-8     17.6          18.5          +5.11%
BenchmarkCopy4Byte-8             3.89          3.78          -2.83%
BenchmarkCopy128Byte-8           5.47          5.62          +2.74%
BenchmarkCopy2String-8           3.52          3.62          +2.84%
BenchmarkCopy12String-8          3.68          3.52          -4.35%
BenchmarkCopy32String-8          3.47          3.35          -3.46%
BenchmarkCopy128String-8         5.47          5.33          -2.56%
BenchmarkCopyFat32-8             1.36          1.31          -3.68%
BenchmarkStackGrowthDeep-8       12598         13692         +8.68%

benchmark                     old MB/s     new MB/s     speedup

benchmark                        old allocs     new allocs     delta
BenchmarkAppendGrowByte-8        51             52             +1.96%
BenchmarkAppendGrowString-8      184            188            +2.17%

@randall77, this is a regression in the SSA back end, similar to the reslicing one. The old back end avoided updating the base pointer in the append fast path, and therefore avoided the write barrier.
$ cat /tmp/x.go
package p

func f(x *[]int) []int {
	*x = append(*x, 1)
	return *x
}
$ go tool compile -ssa=0 -S /tmp/x.go | grep :4
	// fast path - no write barrier
	0x0028 00040 (/tmp/x.go:4)	MOVQ	"".x+88(FP), BX
	0x002d 00045 (/tmp/x.go:4)	NOP
	0x002d 00045 (/tmp/x.go:4)	MOVQ	(BX), DX
	0x0030 00048 (/tmp/x.go:4)	MOVQ	8(BX), AX
	0x0034 00052 (/tmp/x.go:4)	MOVQ	16(BX), CX
	0x0038 00056 (/tmp/x.go:4)	MOVQ	AX, BP
	0x003b 00059 (/tmp/x.go:4)	INCQ	BP
	0x003e 00062 (/tmp/x.go:4)	CMPQ	BP, CX
	0x0041 00065 (/tmp/x.go:4)	JHI	$1, 128
	0x0043 00067 (/tmp/x.go:4)	MOVQ	BP, 8(BX)
	0x0047 00071 (/tmp/x.go:4)	LEAQ	(DX)(AX*8), BX
	0x004b 00075 (/tmp/x.go:4)	MOVQ	$1, (BX)

	// slow path code at bottom of function
	0x0080 00128 (/tmp/x.go:4)	LEAQ	type.[]int(SB), BX
	0x0087 00135 (/tmp/x.go:4)	MOVQ	BX, (SP)
	0x008b 00139 (/tmp/x.go:4)	MOVQ	DX, 8(SP)
	0x0090 00144 (/tmp/x.go:4)	MOVQ	AX, 16(SP)
	0x0095 00149 (/tmp/x.go:4)	MOVQ	CX, 24(SP)
	0x009a 00154 (/tmp/x.go:4)	MOVQ	BP, 32(SP)
	0x009f 00159 (/tmp/x.go:4)	PCDATA	$0, $0
	0x009f 00159 (/tmp/x.go:4)	CALL	runtime.growslice(SB)
	0x00a4 00164 (/tmp/x.go:4)	MOVQ	40(SP), DX
	0x00a9 00169 (/tmp/x.go:4)	MOVQ	48(SP), AX
	0x00ae 00174 (/tmp/x.go:4)	MOVQ	56(SP), CX
	0x00b3 00179 (/tmp/x.go:4)	MOVQ	"".x+88(FP), BX
	0x00b8 00184 (/tmp/x.go:4)	CMPQ	BX, $0
	0x00bc 00188 (/tmp/x.go:4)	JEQ	$1, 260
	0x00be 00190 (/tmp/x.go:4)	MOVQ	AX, BP
	0x00c1 00193 (/tmp/x.go:4)	MOVQ	AX, "".autotmp_0001+64(SP)
	0x00c6 00198 (/tmp/x.go:4)	INCQ	BP
	0x00c9 00201 (/tmp/x.go:4)	MOVQ	BP, 8(BX)
	0x00cd 00205 (/tmp/x.go:4)	MOVQ	CX, 16(BX)
	0x00d1 00209 (/tmp/x.go:4)	MOVQ	DX, "".autotmp_0000+72(SP)
	0x00d6 00214 (/tmp/x.go:4)	CMPB	runtime.writeBarrier(SB), $0
	0x00dd 00221 (/tmp/x.go:4)	JNE	$0, 231
	0x00df 00223 (/tmp/x.go:4)	MOVQ	DX, (BX)
	0x00e2 00226 (/tmp/x.go:4)	JMP	71
	0x00e7 00231 (/tmp/x.go:4)	MOVQ	BX, (SP)
	0x00eb 00235 (/tmp/x.go:4)	MOVQ	DX, 8(SP)
	0x00f0 00240 (/tmp/x.go:4)	PCDATA	$0, $1
	0x00f0 00240 (/tmp/x.go:4)	CALL	runtime.writebarrierptr(SB)
	0x00f5 00245 (/tmp/x.go:4)	MOVQ	"".autotmp_0000+72(SP), DX
	0x00fa 00250 (/tmp/x.go:4)	MOVQ	"".autotmp_0001+64(SP), AX
	0x00ff 00255 (/tmp/x.go:4)	JMP	71
	0x0104 00260 (/tmp/x.go:4)	MOVL	AX, (BX)
	0x0106 00262 (/tmp/x.go:4)	JMP	190
	0x0108 00264 (/tmp/x.go:4)	NOP
$

$ go tool compile -ssa=1 -S /tmp/x.go | grep :4
	// fast path - unnecessary update of base, including test of write barrier flag + possible call
	0x0017 00023 (/tmp/x.go:4)	MOVQ	"".x+88(FP), CX
	0x001c 00028 (/tmp/x.go:4)	MOVQ	16(CX), DX
	0x0020 00032 (/tmp/x.go:4)	MOVQ	8(CX), BX
	0x0024 00036 (/tmp/x.go:4)	MOVQ	BX, "".autotmp_0000+72(SP)
	0x0029 00041 (/tmp/x.go:4)	MOVQ	(CX), BP
	0x002c 00044 (/tmp/x.go:4)	LEAQ	1(BX), SI
	0x0030 00048 (/tmp/x.go:4)	MOVQ	SI, "".autotmp_0001+64(SP)
	0x0035 00053 (/tmp/x.go:4)	CMPQ	SI, DX
	0x0038 00056 (/tmp/x.go:4)	JGT	$0, 140
	0x003a 00058 (/tmp/x.go:4)	MOVQ	$1, (BP)(BX*8)
	0x0043 00067 (/tmp/x.go:4)	MOVQ	SI, 8(CX)
	0x0047 00071 (/tmp/x.go:4)	MOVQ	DX, 16(CX)
	0x004b 00075 (/tmp/x.go:4)	MOVL	runtime.writeBarrier(SB), AX
	0x0051 00081 (/tmp/x.go:4)	TESTB	AL, AL
	0x0053 00083 (/tmp/x.go:4)	JNE	$0, 119
	0x0055 00085 (/tmp/x.go:4)	MOVQ	BP, (CX)

	// slow path for write barrier call in append "fast" path
	0x0077 00119 (/tmp/x.go:4)	MOVQ	CX, (SP)
	0x007b 00123 (/tmp/x.go:4)	MOVQ	BP, 8(SP)
	0x0080 00128 (/tmp/x.go:4)	PCDATA	$0, $0
	0x0080 00128 (/tmp/x.go:4)	CALL	runtime.writebarrierptr(SB)

	// slow path for append calling growslice
	0x008c 00140 (/tmp/x.go:4)	LEAQ	type.[]int(SB), AX
	0x0093 00147 (/tmp/x.go:4)	MOVQ	AX, (SP)
	0x0097 00151 (/tmp/x.go:4)	MOVQ	BP, 8(SP)
	0x009c 00156 (/tmp/x.go:4)	MOVQ	BX, 16(SP)
	0x00a1 00161 (/tmp/x.go:4)	MOVQ	DX, 24(SP)
	0x00a6 00166 (/tmp/x.go:4)	MOVQ	SI, 32(SP)
	0x00ab 00171 (/tmp/x.go:4)	PCDATA	$0, $0
	0x00ab 00171 (/tmp/x.go:4)	CALL	runtime.growslice(SB)
	0x00b0 00176 (/tmp/x.go:4)	MOVQ	40(SP), BP
	0x00b5 00181 (/tmp/x.go:4)	MOVQ	56(SP), DX
	0x00bf 00191 (/tmp/x.go:4)	MOVQ	"".autotmp_0000+72(SP), BX
	0x00c4 00196 (/tmp/x.go:4)	MOVQ	"".autotmp_0001+64(SP), SI
	0x00c9 00201 (/tmp/x.go:4)	JMP	58
	0x00ce 00206 (/tmp/x.go:4)	NOP
$

For the hashtable iterator case, we can just do the check to see if we're writing something new back explicitly.
I'll open a separate bug for the append writeback.

I remeasured this after CLs 21812, 21813, and 21814, which address the append and the map iteration cases.
New results, using Austin's approach above:
29354 1173 2934
So 6% of writes have *dst == src, and 10% have src == nil.
The new top lines for *dst == src (line numbers at commit de7ee57c7ead59899d5b412a839c995de0e813b5) are:
 37.57%   420 runtime/proc.go:259 runtime.gopark
 14.13%   158 runtime/mheap.go:603 runtime.(*mheap).allocSpanLocked
  5.28%    59 runtime/mheap.go:588 runtime.(*mheap).allocSpanLocked
  4.92%    55 runtime/mgc.go:1725 runtime.gcCopySpans
  4.92%    55 runtime/mgc.go:1726 runtime.gcCopySpans

The gopark calls are unnecessary, since all the strings that reach it are constants, but there's no way for the compiler to know that.
I've filed #15226 for allocSpanLocked. gcCopySpans might deserve a look as well, but we're close to scraping bottom (yay!).
I will investigate whether it is better to exit writebarrierptr early if src == nil or whether it is better to check it in the generated code and send a CL for one of those to close this issue.

Mailed CLs 21820 and 21821 that implement the two "do less work if src==nil" strategies described above. The numbers for adding a check to the wrapper code are better, although it does increase binary size by ~0.1%. It's not obvious to me which of the two (if either) should go in. Feedback or better benchmark numbers welcomed.

So besides src==nil and *dst==src, there's another condition that would be interesting to get stats on: dst points to the stack.  It might illuminate code (I'm looking at you, convT2I and friends) where the writes are always within the stack.  With might still need the write barrier because of stack barriers, but maybe some rearrangement/optimization would be possible.

I had a sudden suspicious that I should look again at this. With the move of the write barrier to assembly, it has gotten a bit harder to instrument. Leaving in the vestigial wbe variable so I could re-use Austin's awk command, here is my instrumentation for amd64:
diff --git a/src/runtime/asm_amd64.s b/src/runtime/asm_amd64.s
index 2376fe0aae..f07c0bcbe1 100644
--- a/src/runtime/asm_amd64.s
+++ b/src/runtime/asm_amd64.s
@@ -2384,6 +2384,15 @@ TEXT runtime·gcWriteBarrier(SB),NOSPLIT,$120
        // faster than having the caller spill these.
        MOVQ    R14, 104(SP)
        MOVQ    R13, 112(SP)
+
+       MOVQ    $1, R13
+       CMPQ    AX, (DI)
+       JNE     3(PC)
+       LOCK; XADDQ R13, runtime·wbe(SB)
+
+       MOVQ    $1, R13
+       LOCK; XADDQ R13, runtime·wb(SB)
+
        // TODO: Consider passing g.m.p in as an argument so they can be shared
        // across a sequence of write barriers.
        get_tls(R13)
diff --git a/src/runtime/proc.go b/src/runtime/proc.go
index f20e77eee5..0e0c1d446c 100644
--- a/src/runtime/proc.go
+++ b/src/runtime/proc.go
@@ -224,9 +224,12 @@ func main() {
        }
 }

+var wb, wbe, wbz uint64
+
 // os_beforeExit is called from os.Exit(0).
 //go:linkname os_beforeExit os.runtime_beforeExit
 func os_beforeExit() {
+       println("WB", wb, wbe, wbz)
        if raceenabled {
                racefini()
        }

The result for make.bash piped through Austin's awk command is: 70074409 16189273 0.
That's >23% of write barriers with *dst == src, which is suspiciously similar to the amount prior to the append optimization. I suspect that that optimization got lost when the write barrier insertion got moved in to the SSA backend. I've filed #24209 to look at that again.

I suspect that that optimization got lost when the write barrier insertion got moved in to the SSA backend.

No, it is not lost. Using @rsc's test case:
$ cat x.go
package p

func f(x *[]int) []int {
    *x = append(*x, 1)
    return *x
}
$ go tool compile -S x.go | grep :4
	// fast path
	0x0026 00038 (x.go:4)	MOVQ	8(AX), CX
	0x002a 00042 (x.go:4)	MOVQ	16(AX), DX
	0x002e 00046 (x.go:4)	MOVQ	(AX), BX
	0x0031 00049 (x.go:4)	LEAQ	1(CX), SI
	0x0035 00053 (x.go:4)	CMPQ	SI, DX
	0x0038 00056 (x.go:4)	JGT	110
	0x003a 00058 (x.go:4)	LEAQ	1(CX), DX
	0x003e 00062 (x.go:4)	MOVQ	DX, 8(AX)
	0x0042 00066 (x.go:4)	MOVQ	$1, (BX)(CX*8)
	// slow path
	0x006e 00110 (x.go:4)	LEAQ	type.int(SB), AX
	0x0075 00117 (x.go:4)	MOVQ	AX, (SP)
	0x0079 00121 (x.go:4)	MOVQ	BX, 8(SP)
	0x007e 00126 (x.go:4)	MOVQ	CX, 16(SP)
	0x0083 00131 (x.go:4)	MOVQ	DX, 24(SP)
	0x0088 00136 (x.go:4)	MOVQ	SI, 32(SP)
	0x008d 00141 (x.go:4)	PCDATA	$0, $0
	0x008d 00141 (x.go:4)	CALL	runtime.growslice(SB)
	0x0092 00146 (x.go:4)	MOVQ	40(SP), AX
	0x0097 00151 (x.go:4)	MOVQ	48(SP), CX
	0x009c 00156 (x.go:4)	MOVQ	56(SP), DX
	0x00a1 00161 (x.go:4)	MOVQ	"".x+80(SP), DI
	0x00a6 00166 (x.go:4)	MOVQ	DX, 16(DI)
	0x00aa 00170 (x.go:4)	CMPL	runtime.writeBarrier(SB), $0
	0x00b1 00177 (x.go:4)	JNE	193
	0x00b3 00179 (x.go:4)	MOVQ	AX, (DI)
	0x00b6 00182 (x.go:4)	MOVQ	AX, BX
	0x00b9 00185 (x.go:4)	MOVQ	DI, AX
	0x00bc 00188 (x.go:4)	JMP	58
	0x00c1 00193 (x.go:4)	CALL	runtime.gcWriteBarrier(SB)
	0x00c6 00198 (x.go:4)	JMP	182
	0x00c8 00200 (x.go:4)	NOP

I think we also have a test that makes sure that we never write the pointer field in the non-growing case.

Hmm. Thanks. I wonder where all the *dst == src writebarrier calls are coming from now.

Maybe we need the check for zero...maybe all the *dst == src when both are nil?
Any which way, this seems worth digging into a bit.


add MPEG-4 video format

related to https://github.com/aler9/mediamtx/issues/1550

Codecov Report

Merging #236 (8003fd3) into main (b6727c0) will increase coverage by 0.03%.
The diff coverage is 88.23%.

:exclamation: Current head 8003fd3 differs from pull request most recent head 53691bb. Consider uploading reports for the commit 53691bb to get more accurate results

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##             main     #236      +/-   ##
==========================================
+ Coverage   85.61%   85.64%   +0.03%
==========================================
  Files          81       82       +1
  Lines        8726     8766      +40
==========================================
+ Hits         7471     7508      +37
- Misses        972      974       +2
- Partials      283      284       +1

Impacted Files
Coverage Δ

pkg/formats/g711.go
100.00% <ø> (ø)

pkg/formats/g722.go
100.00% <ø> (ø)

pkg/formats/generic.go
74.46% <ø> (ø)

pkg/formats/h265.go
81.01% <ø> (ø)

pkg/formats/lpcm.go
100.00% <ø> (ø)

pkg/formats/mjpeg.go
100.00% <ø> (ø)

pkg/formats/mpeg2_audio.go
100.00% <ø> (ø)

pkg/formats/mpeg2_video.go
100.00% <ø> (ø)

pkg/formats/mpeg4_audio.go
93.54% <ø> (ø)

pkg/formats/opus.go
87.71% <ø> (ø)

... and 6 more

... and 1 file with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

added in MediaMTX v0.22.1


simple.js mode is broken

Reproducing
Just open https://codemirror.net/demo/simplemode.html

Seems arising from the recent changes https://github.com/codemirror/CodeMirror/commit/db1908a6ef80c3a712dc9a20ab914a4fc97e4276 related with "[mode/simple addon] Allow token source to be a function."

Thanks for letting me know! I've released 5.27.2 with a fix.


Fix gpu_limit condition in VertexOrchestrator

Describe changes
In #935, I added the condition to check if it was needed to add the node selector constraint if gpu_limit was 0, but there was another bug when the value for gpu_limit was set, and the value could never be 0. As 0 is evaluated like a falsy value, the value of gpu_limit would always be the value of self.config.gpu_limit in the case resource_settings.gpu_count == 0.
I've also added unit tests for VertexOrchestrator._configure_container_resources method.
Pre-requisites
Please ensure you have done the following:

[x] I have read the CONTRIBUTING.md document.
[ ] If my change requires a change to docs, I have updated the documentation accordingly.
[ ] If I have added an integration, I have updated the integrations table and the corresponding website section.
[x] I have added tests to cover my changes.

Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Other (add details above)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Gabriel Martín Blázquez seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Can you keep the bookmark bar when it is hidden on common tabs?

In chrome, I used to hide the bookmark bar (Ctrl+Shift+B). When I do that, chrome will hide the bar on common tabs, but keep showing it on the default new tab, so I can view the bookmarks easily without losing visiable area. However, this extension will hide the bar. Is it possible to keep it?
————————————————
Chinese version:
我一般习惯设置隐藏书签栏，这个时候chrome会把其它页面的书签栏都隐藏了，但是默认新标签页的书签栏会保留。这样我就可以在保留最大可视面积的同时方便地访问书签栏。这个插件目前会把新标签页的书签栏也隐藏掉，请问有办法避免吗？

It's not possible to use default chrome new tab if you are using a new tab extension. Because chrome doesn't provide a way to modify its own default new tab, what the new tab extension can do is overriding the default new tab. If you want to view your bookmarks, you can visit this page chrome://bookmarks/.

报错了，启用后显示一片空白。m1处理器下的chorme
Unchecked runtime.lastError: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received
上下文
chrome://newtab/
堆叠追踪
index.html:0 (匿名函数)


Error while installing: "c++" did not execute successfully (status code exit status: 1)

Hello @Wilfred,
while installing using cargo install difftastic I get this error:
error occurred: Command "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "vendor/tree-sitter-haskell-src" "-Wall" "-Wextra" "--std=c++14" "-Wno-implicit-fallthrough" "-Wno-unused-parameter" "-Wno-ignored-qualifiers" "-o" "/tmp/cargo-installJaPRlT/release/build/difftastic-18c9fb06f0b5e91b/out/vendor/tree-sitter-haskell-src/scanner.o" "-c" "vendor/tree-sitter-haskell-src/scanner.cc" with args "c++" did not execute successfully (status code exit status: 1).
Versions:
14:20 $ rustc --version
rustc 1.57.0
14:20 $ cargo --version
cargo 1.57.0
difftastic version (from the failing install command):
v0.24.0
LLVM version:
13.0.0
System:
Linux 4.15.0-173-generic x86_64 GNU/Linux
I'm new to the rust environment. Any suggest on how to resolve this?
Cheers,
Peter

The Haskell parser is one of the more complex parsers (I think it's the only one that requires --std=c++14). I'm not sure why it isn't linking correctly for you here.
Did you get any additional build output before the error? Also, what does c++ --version say?

Same happens on my end:
c++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

I got the same error.  Also on Ubuntu 18.04.
g++ version 7.5
rustc version 1.59
I found some of the error output:
The following warnings were emitted during compilation:

warning: vendor/tree-sitter-haskell-src/scanner.cc: In instantiation of ‘operator*(C (&)(B), std::function<B(A)>)::<lambda(A)> [with A = State&; B = std::pair<bool, unsigned int>; C = bool]’:
warning: vendor/tree-sitter-haskell-src/scanner.cc:90:30:   required from ‘struct operator*(C (&)(B), std::function<B(A)>) [with A = State&; B = std::pair<bool, unsigned int>; C = bool]::<lambda(struct State&)>’
warning: vendor/tree-sitter-haskell-src/scanner.cc:90:37:   required from ‘std::function<C(A)> operator*(C (&)(B), std::function<B(A)>) [with A = State&; B = std::pair<bool, unsigned int>; C = bool]’
warning: vendor/tree-sitter-haskell-src/scanner.cc:368:73:   required from here
warning: vendor/tree-sitter-haskell-src/scanner.cc:90:28: error: variable ‘f’ has function type
warning:    return [=](A a) { return f(g(a)); };
warning:                             ^
warning: vendor/tree-sitter-haskell-src/scanner.cc:90:28: error: variable ‘f’ has function type
warning: vendor/tree-sitter-haskell-src/scanner.cc: In instantiation of ‘struct operator*(C (&)(B), std::function<B(A)>) [with A = State&; B = std::pair<bool, unsigned int>; C = bool]::<lambda(struct State&)>’:
warning: vendor/tree-sitter-haskell-src/scanner.cc:90:37:   required from ‘std::function<C(A)> operator*(C (&)(B), std::function<B(A)>) [with A = State&; B = std::pair<bool, unsigned int>; C = bool]’
warning: vendor/tree-sitter-haskell-src/scanner.cc:368:73:   required from here
warning: vendor/tree-sitter-haskell-src/scanner.cc:90:28: error: field ‘operator*(C (&)(B), std::function<B(A)>) [with A = State&; B = std::pair<bool, unsigned int>; C = bool]::<lambda(State&)>::<f capture>’ invalidly declared function type
warning: vendor/tree-sitter-haskell-src/scanner.cc: In instantiation of ‘std::function<C(A)> operator*(C (&)(B), std::function<B(A)>) [with A = State&; B = std::pair<bool, unsigned int>; C = bool]’:
warning: vendor/tree-sitter-haskell-src/scanner.cc:368:73:   required from here
warning: vendor/tree-sitter-haskell-src/scanner.cc:90:37: error: could not convert ‘<lambda closure object>operator*(C (&)(B), std::function<B(A)>) [with A = State&; B = std::pair<bool, unsigned int>; C = bool]::<lambda(State&)>{(& f), std::function<std::pair<bool, unsigned int>(State&)>(g)}’ from ‘operator*(C (&)(B), std::function<B(A)>) [with A = State&; B = std::pair<bool, unsigned int>; C = bool]::<lambda(State&)>’ to ‘std::function<bool(State&)>’
warning:    return [=](A a) { return f(g(a)); };
warning:                                      ^
warning: vendor/tree-sitter-haskell-src/scanner.cc: In instantiation of ‘parser::with(A (&)(State&))::<lambda(std::function<std::function<Result(State&)>(A)>)>::<lambda(State&)> [with A = symbolic::Symbolic]’:
warning: vendor/tree-sitter-haskell-src/scanner.cc:816:16:   required from ‘struct parser::with(A (&)(State&))::<lambda(std::function<std::function<Result(State&)>(A)>)> [with A = symbolic::Symbolic]::<lambda(struct State&)>’
warning: vendor/tree-sitter-haskell-src/scanner.cc:818:3:   required from ‘parser::with(A (&)(State&))::<lambda(std::function<std::function<Result(State&)>(A)>)> [with A = symbolic::Symbolic]’
warning: vendor/tree-sitter-haskell-src/scanner.cc:816:16:   required from ‘struct parser::with(A (&)(State&)) [with A = symbolic::Symbolic]::<lambda(class std::function<std::function<Result(State&)>(symbolic::Symbolic)>)>’
warning: vendor/tree-sitter-haskell-src/scanner.cc:818:3:   required from ‘std::function<std::function<Result(State&)>(std::function<std::function<Result(State&)>(A)>)> parser::with(A (&)(State&)) [with A = symbolic::Symbolic]’
warning: vendor/tree-sitter-haskell-src/scanner.cc:1501:29:   required from here
warning: vendor/tree-sitter-haskell-src/scanner.cc:816:16: error: variable ‘fa’ has function type
warning:        return f(fa(state))(state);
warning:                 ^~
warning: vendor/tree-sitter-haskell-src/scanner.cc: In instantiation of ‘struct parser::with(A (&)(State&))::<lambda(std::function<std::function<Result(State&)>(A)>)> [with A = symbolic::Symbolic]::<lambda(struct State&)>’:
warning: vendor/tree-sitter-haskell-src/scanner.cc:818:3:   required from ‘parser::with(A (&)(State&))::<lambda(std::function<std::function<Result(State&)>(A)>)> [with A = symbolic::Symbolic]’
warning: vendor/tree-sitter-haskell-src/scanner.cc:816:16:   required from ‘struct parser::with(A (&)(State&)) [with A = symbolic::Symbolic]::<lambda(class std::function<std::function<Result(State&)>(symbolic::Symbolic)>)>’
warning: vendor/tree-sitter-haskell-src/scanner.cc:818:3:   required from ‘std::function<std::function<Result(State&)>(std::function<std::function<Result(State&)>(A)>)> parser::with(A (&)(State&)) [with A = symbolic::Symbolic]’
warning: vendor/tree-sitter-haskell-src/scanner.cc:1501:29:   required from here
warning: vendor/tree-sitter-haskell-src/scanner.cc:816:16: error: field ‘parser::with(A (&)(State&))::<lambda(std::function<std::function<Result(State&)>(A)>)> [with A = symbolic::Symbolic]::<lambda(State&)>::<fa capture>’ invalidly declared function type
warning: vendor/tree-sitter-haskell-src/scanner.cc:816:18: error: field ‘parser::with(A (&)(State&))::<lambda(std::function<std::function<Result(State&)>(A)>)> [with A = symbolic::Symbolic]::<lambda(State&)>::<fa capture>’ invalidly declared function type
warning:        return f(fa(state))(state);
warning:                 ~~^~~~~~~
warning: vendor/tree-sitter-haskell-src/scanner.cc: In instantiation of ‘parser::with(A (&)(State&))::<lambda(std::function<std::function<Result(State&)>(A)>)> [with A = symbolic::Symbolic]’:
warning: vendor/tree-sitter-haskell-src/scanner.cc:816:16:   required from ‘struct parser::with(A (&)(State&)) [with A = symbolic::Symbolic]::<lambda(class std::function<std::function<Result(State&)>(symbolic::Symbolic)>)>’
warning: vendor/tree-sitter-haskell-src/scanner.cc:818:3:   required from ‘std::function<std::function<Result(State&)>(std::function<std::function<Result(State&)>(A)>)> parser::with(A (&)(State&)) [with A = symbolic::Symbolic]’
warning: vendor/tree-sitter-haskell-src/scanner.cc:1501:29:   required from here
warning: vendor/tree-sitter-haskell-src/scanner.cc:815:12: sorry, unimplemented: non-trivial designated initializers not supported
warning:      return [=](State & state) {
warning:             ^
warning: vendor/tree-sitter-haskell-src/scanner.cc: In instantiation of ‘std::function<std::function<Result(State&)>(std::function<std::function<Result(State&)>(A)>)> parser::with(A (&)(State&)) [with A = symbolic::Symbolic]’:
warning: vendor/tree-sitter-haskell-src/scanner.cc:1501:29:   required from here
warning: vendor/tree-sitter-haskell-src/scanner.cc:818:3: error: could not convert ‘<lambda closure object>parser::with(A (&)(State&)) [with A = symbolic::Symbolic]::<lambda(std::function<std::function<Result(State&)>(symbolic::Symbolic)>)>{(* & fa)}’ from ‘parser::with(A (&)(State&)) [with A = symbolic::Symbolic]::<lambda(std::function<std::function<Result(State&)>(symbolic::Symbolic)>)>’ to ‘std::function<std::function<Result(State&)>(std::function<std::function<Result(State&)>(symbolic::Symbolic)>)>’
warning:    };
warning:    ^

error: failed to compile `difftastic v0.24.0`, intermediate artifacts can be found at `/tmp/cargo-installluOyUe`

For me on Ubuntu 18.04, clang compiled difft successfully.
sudo update-alternatives --config c++

Changing c++ to use clang instead of g++ worked indeed.

This is what I did on my Ubuntu 18.04.6:
sudo apt install clang (clang wasn't installed on my machine)
sudo update-alternatives --config c++ and selected clang instead of g++.

Hello @Wilfred,
Thanks for your reply. My c++ version the same as reported before:
11:52 $ c++ --version
c++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
During the install command I get a number of warnings as already pasted by @joel-bd.
The work-around as described by @shameempk worked! Thank you!
As I'm not sure if this is the intended solution I'll leave the ticket open in case there might be another solution you'd like to reference back. Feel free to close it as you see fit.
Thanks everyone for the help!
Cheers,
Peter

updating to g++-8 also worked for me
g++-8 (Ubuntu 8.4.0-1ubuntu1~18.04) 8.4.0

In an 18.04 docker image, the update-alternatives workaround is not working for me. It still runs g++, not clang, during the build, despite c++ --version indicating clang. Any other way to force it?

I have another solution for outdated systems. On my Ubuntu 16.04 default c++ is very old (gcc 5.4). But I have gcc 9 and ccache installed there. So to build difftastic successfully I I had to execute this command:
CC=/usr/lib/ccache/gcc-9 CXX=/usr/lib/ccache/g++-9 cargo install difftastic

same issue in win10
λ  g++ --version
g++.exe (GCC) 11.2.0
λ  cargo --version
cargo 1.56.0 (4ed5d137b 2021-10-04)
λ  rustc --version
rustc 1.56.1 (59eed8a2a 2021-11-01)
error occurred: Command "g++.exe" "-O3" "-ffunction-sections" "-fdata-sections" "-m64" "-I" "vendor/tree-sitter-elixir-src" "-Wall" "-Wextra" "/std:c++14" "-Wno-implicit-fallthrough" "-Wno-unused-parameter" "-Wno-ignored-qualifiers" "-o" "C:\Users\xxx\AppData\Local\Temp\cargo-installDwCYOe\release\build\difftastic-1dfc0d39f512cf59\out\vendor/tree-sitter-elixir-src\scanner.o" "-c" "vendor/tree-sitter-elixir-src\scanner.cc" with args "g++.exe" did not execute successfully (status code exit code: 1).

0.26 fixed an issue where C++ files were built before the C libraries they needed, which some compilers complain about. 0.26.2 fixes compilation on Windows, and I now test all of Linux, macOS and Windows with GitHub actions.
There's also prebuilt binaries now available on GitHub releases: https://github.com/Wilfred/difftastic/releases
I'm going to close this, but feel free to open new issues if the latest release isn't building and you think it's a difftastic issue.


Bluenet BLE service not found on devices running Android 5.1

The custom Bluetooth Low Energy GATT service that Bluenet is advertising for Wifi provisioning is not showing up on multiple devices running Android 5.1. Instead two default services are shown.
This can be observed in both the React Native app and Nordic Semiconductor nRF Connect app, so it is probably not a problem in the app itself.
Bluetooth LE and 128 bit UUIDs in general can be eliminated as the possible issue, too, as these are working on at least one of these devices (Nuimos can be connected).
The output of logcat is identical to successfully connecting to a Nuimo.
Known problematic devices:

Huawei Mate S running Android 5.1.1
@grunskis phone? - Fairphone 2 with Android 5.1

Unsolved Stackoverflow question with the same problem: Here

Together with a user on IRC I finally found a solution!
The problem is that Android 5.1 devices try to connect with BR/EDR if it is advertised as supported and then aren't able to find the custom service anymore.
One solution would be not to advertise BR/EDR, but at the moment we cannot change this with BlueZ as far as I know.
The workaround is to disable BR/EDR for the Bluetooth adapter of the hub in general with the following commands:
btmgmt power on
btmgmt bredr off
btmgmt power on

Great find! Here's a way to explicitly require a BLE connection, in case this issue is related:
https://code.google.com/p/android/issues/detail?id=223104
The hack is implemented in our Android SDK:
https://github.com/getsenic/nuimo-android/blob/master/nuimo/src/main/kotlin/com/senic/nuimo/NuimoBluetoothController.kt#L105

@larsblumberg I think the problem with Android 7.0 is different (BR/EDR is not even advertised in that case) but the fix could be working anyway.
@tomster Until we can add and test the fix in the Android app, can we disable BR/EDR in Senic OS as it is not used at the moment? Either with the commands from above or with the main.conf settings file of BlueZ (see https://github.com/RadiusNetworks/bluez/blob/master/src/main.conf#L53) ?


Inconsistent Behavior

# Enable auto expand with disable auto collapse
## In 0.12.2
ethereum-types  = { version = "0.12.0", default-features = false, features = ["codec", "rlp"] }
## In 0.13.1
### And this is not right
ethereum-types  = { version = "0.12.0", default-features = false, features = [
    "codec",
    "rlp"
] }

# Disable auto expand with disable auto collapse
## In 0.12.2
crates-std    = [
	"codec/std",
	"ethbloom/std",
	"ethereum-types/std",
	"ethereum-types/serialize",
	"fixed-hash/std",
	"impl-codec/std",
	"impl-rlp/std",
	"keccak-hash/std",
	"primitive-types/std",
	"rlp/std",
	"serde",
]
darwinia-std  = [
	"ethash/std",
	"merkle-patricia-trie/std",
]
substrate-std = [
	"sp-io/std",
	"sp-runtime/std",
	"sp-std/std",
]

deserialize  = []
with-new-dag = []
## In 0.13.1
### Expanded entries not get align
crates-std = [
	"codec/std",
	"ethbloom/std",
	"ethereum-types/std",
	"ethereum-types/serialize",
	"fixed-hash/std",
	"impl-codec/std",
	"impl-rlp/std",
	"keccak-hash/std",
	"primitive-types/std",
	"rlp/std",
	"serde",
]
darwinia-std = [
	"ethash/std",
	"merkle-patricia-trie/std",
]
substrate-std = [
	"sp-io/std",
	"sp-runtime/std",
	"sp-std/std",
]
### Empty entries get align
deserialize  = []
with-new-dag = []

Hi, thanks for the report.
If I understand correctly, there are two concerns here:

Arrays being expanded in the new version, while they weren't in the previous versions. This is due to columnWidth being actually more closely respected, reducing columnWidth or adding more entries into the inline array would cause the same behaviour in the old version, so this is intentional.
Items spanning multiple lines are not aligned. Indeed this is the case, because I don't exactly know how they should be aligned, as aligning them doesn't make a lot of sense. I might align the first lines only, so that would mimic the behaviour of the old formatter.

Hi, thanks for the report.
If I understand correctly, there are two concerns here:
* Arrays being expanded in the new version, while they weren't in the previous versions. This is due to `columnWidth` being actually more closely respected, reducing `columnWidth` or adding more entries into the inline table would cause the same behaviour in the old version, so this is intentional.

* Items spanning multiple lines are not aligned. Indeed this is the case, because I don't exactly know how they should be aligned, e.g. what should happen with comments. I might align the first lines only, so that would mimic the behaviour of the old formatter.

Yes.
And I really need the second one, which made the format much concise and easy to review.

Sorry, implementing this properly is out of scope for me currently.


use version number instead of release in .travis.yml

release will change over time, but your REQUIRE file says this package supports julia 0.4 so it should continue to be tested

https://github.com/JuliaIO/CBOR.jl/pull/10


Test

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement
Learn more about why HashiCorp requires a CLA and what the CLA includes

0 out of 2 committers have signed the CLA.

[ ] samuellee-zy
[ ] samlee-hashicorp

Have you signed the CLA already but the status is still pending? Recheck it.


accessible domain selector

⚠⚠⚠ Since we do not accept all types of pull requests and do not want to waste your time. Please be sure that you have read pull request rules:
https://github.com/louislam/uptime-kuma/blob/master/CONTRIBUTING.md#can-i-create-a-pull-request-for-uptime-kuma
Tick the checkbox if you understand [x]:

[x] I have read and understand the pull request rules.

Description
I have tested #4122 and found that adding a button adds the styling for this too.
This PR adds the role of button to the element we use as such without adding the styling of the button
Resolves #4122
Type of change
Please delete any options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
User interface (UI)

Checklist

[x] My code follows the style guidelines of this project
[x] I ran ESLint and other linters for modified files
[x] I have performed a self-review of my own code and tested it
[x] I have commented my code, particularly in hard-to-understand areas (including JSDoc for methods)
[x] My changes generates no new warnings
[ ] My code needed automated testing. I have added them (this is optional task)

Screenshots (if any)

Thanks for working on this! <font-awesome-icon> sets aria-hidden="true", so it is ignored by a screen reader. Adding aria-hidden="false" makes it work, but it still isn't focusable with the keyboard. It is generally considered best practise to use native controls when ever possible and fall back to aria when there is no other option. See the first rule of ARIA
If it isn't possible to override the <button> styling to make it work, adding role="button" and aria-hidden="false" is usable (at least with the NVDA screen reader).

good catch! sorry

No need to apologise. It is perfect now as long as it is visually okay. 🙂


Credential in a proof request and is rendered with elements out of place

Looks fine in latest build (789)


The error is not clear when creating a second container app environment with the same location

OS: All
Build Version: 20220405.6
Regression: Not a regression
Repro Steps:

Create a container app environment with a location, such as "East US".
Create a new container app environment with the location "East US" again, an error occurs when creating the second container app environment with the location "East US".
Check whether the error is clear or not.

Expect:
The error is clear, such as "There is only one environment can be created in one location...".
Actual:
The error is not clear.

More Info:
The new container app environment with other locations can be created successfully.

The error code is 'MaxNumberOfEnvironmentsInSubExceeded' which we can check for, but based off the error message and code, I feel like the service is checking a different condition.  Like @v-ruizh mentioned, it seems like you can have more than 2 environments as long as you are creating them in different locations.
@anthonychu Know who the right person is to ask about this?

@v-ruizh Are you able to reproduce this still? I was able to create 3 environments in the same location.

@nturinski This issue does not reproduce on the latest build 20221008.1, close it.


[Suggestion] getPokemonsByFamily()

Can you please add a method "getPokemonsByFamily()" similar to "getPokemonsByType()" to collect all Pokemon from the same family ?
Thanks in advance

https://github.com/NicklasWallgren/PokemonGoAPI-PHP/commit/310b338f014616c1c53c2b9001aa335f8bfe58c3


Error: "Could not find module in path..." in Code Sandbox

Current Behavior
Code Sandbox examples from baseui are not able to import baseui module.

Tech
Version

Base UI
v6.19.2

React

browser

[x] I have searched the issues of this repository and believe that this is not a duplicate.

Looks like it's mispublished... top-level files live in dist/ now.

Repro:
https://codesandbox.io/s/upbeat-resonance-1iwub

fixed in 6.19.3 - thanks for the report!


Encountering Exception Error: expected shape()

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.
Thank you!

[X ] Check that you are up-to-date with the master branch of Keras. You can update with:
pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

[X ] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

[-] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

[X] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

Hi everyone,
I have been playing Faster RCNN in this depreciated repo: https://github.com/yhenon/keras-frcnn
Before my computer broke down on me GTX1080Ti (CUDA 7.0, cuDNN v3) in Ubuntu 14.04. I was able to run this repo straight-off and train/test my system without an issue.
I rebuild my software environment setup running on GTX1080Ti (CUDA 8.0, cuDNN v6) in Ubuntu 14.04.
The first thing I did was to download the stock repo and run the program to which I had the shock of my life:
"Exception: Error when checking target: expected rpn_out_regress to have shape (None, None, None, 36) but got array with shape (1, 38, 67, 72)"
I am wondering if anyone knows what I can do about it? I think the code is fine since I could run previously (exactly the same code)!
Sincerely,
Kenny

Good news! I managed to resolve my problem above which bugged me for 2 nights. After checking through shapes and all, I really can't figure out why.
Until... I tried uninstalling Keras v2.1.1 and installing Keras v2.0.6 (the repo I used when my computer haven't failed me).
Not sure why, but, thanks everyone! :)


Usability when saving positioned branch as new scene

Godot version
4.1.1
System information
Windows 10
Issue description
While prototyping it may happen, that a positioned node, e.g. an obstacle, should be saved as scene to reuse it. While doing so one still want's to preserve the position of the original obstacle in the level.
After saving the branch to a scene, the root node is offset in the new scene. For reusability most often the root node should be positioned at zero. When setting the position in the new scene, the position in the original level changes and one needs to reposition the obstacle.
Steps to reproduce

create scene A
add a Node2D
change the position of the node
save the node 2d as new scene
reset the position to zero in the new scene
the node is now positioned at zero in the original scene as well

Minimal reproduction project
N/A

Another problem with the current behavior, which often annoys me, is that when you create a scene with shifted origin and then decide to reset it to 0, your original instance will be moved, because it's using the default value of the position. Resetting the position automatically is a good idea.


Some small improvements to the tutorials.

Hello There!
Here's my first small attempt to work on the tutorials. In addition to my changes in the pull request I have several observation/recommendations in several of the tutorials listed below. I could work on some of these later but want to share them with you:

In "How to interactively inspect Target Pixel Files and Light Curves?" tutorial:
The first demo animation does not appear to be for HL Tau.

In Identifying rolling bands tutorial:
In the first video we are to see features at the “2 second” mark but the video does not have a second counter to reference where is 2 seconds. Might also be good to mention the rolling band is a roughly horizontal band in the video. It might be possible someone might confuse the asteroids with what you are referring to as rolling bands.
Also, emphasize this is ONLY a Kepler/K2 artifact.

In CorrectorClass tutorial:
Appending a constant to a design matrix is a standard thing to do. Consider doing that by default and allowing an option to NOT append to it.
When removing negative flux from the model why subtract off 5th percentile? Why not just subtract off min to get all values positive? Or, switch to a normalized basis, perform the systematic correction, then denormalize back, then adjust the secular term.
The example of Boyajian's Star might not be ideal for a TESS 2 min example because the PDC light curve has a lower CDPP than the RegressionCorrector residual (811 vs 836). In any event, we should probably provide a tutorial where we discuss how to diagnose when to use the PDC light curves versus generating your own.
A lot more can be done with the pca method. Two ideas: 1) consider adding method to use BIC or AIC or something to select the number of principle components to use. 2) Consider taking cuts on the design matrix before applying PCA, for example, to only keep highly correlated design matrix columns before finding the principle components.

In SFFCorrector tutorial:
Link to tutorial to tune parameters is not working.

in KeplerCBVCorrector tutorial:
So, search_ligthcurvefile also reads in the CBV FITS files?
Removing NaNs changes the length of the time series. Doing so means the CBVs no longer line up with the light curve. We should fill gaps instead. Looks like the fill_gaps method can use some improvements.
In plot_cbvs, consider reversing the order of the offset so that the curves are in the same vertical order as the legend.

Hey Jeff,
Thanks for the input, I'm closing this PR and opening another one with these changes. In response to your comments:

The first demo animation does not appear to be for HL Tau.

It is, the EPIC number is the K2 designation for HL Tau

Appending a constant to a design matrix is a standard thing to do. Consider doing that by default and allowing an option to NOT append to it.

This is a comment on the API, not the tutorial. If you feel strongly about this I would encourage you to open an issue. Same with your other comment on additional metrics for RegressionCorrector, I'd be very happy to discuss them on a separate issue.

When removing negative flux from the model why subtract off 5th percentile? Why not just subtract off min to get all values positive? Or, switch to a normalized basis, perform the systematic correction, then denormalize back, then adjust the secular term.

The minimum flux in some light curves could be zero, and so this is to be more robust to your average user. The tutorial should explain and illustrate why we are doing this.

The example of Boyajian's Star might not be ideal for a TESS 2 min example because the PDC light curve has a lower CDPP than the RegressionCorrector residual (811 vs 836). In any event, we should probably provide a tutorial where we discuss how to diagnose when to use the PDC light curves versus generating your own.

Boyajian's star was chosen because it is popular and an easy case to demonstrate why scattered light needs to be removed. A tutorial on PDC flux would be excellent.
I think your CBV corrector comments are also on the API, not the tutorial, and I think we're solving them in your PR #672.
I'm closing this and opening a new PR, thanks for playtesting our tutorials!! 👍


feat(Algebra/Lie): define derivations on Lie algebras

This defines derivations on Lie algebras. The current definition of Derivation is restricted to commutative associative algebras, and thus cannot be used to manipulate derivations on Lie algebras. As discussed in this Zulip thread, we thus give a parallel definition of a LieDerivation structure, which is specific to derivations on Lie algebras.
This PR is focused on the definition of this structure and the first associated properties. It adds a new file Algebra.Lie.Derivation.Basic which was mostly obtained by a large copy-paste of file RingTheory.Derivation.Basic. I report below the changes I made with respect to this other version.

Remove all #align directives
Adapt import statements
Add new definition of LieDerivation
Replace namespace Derivation with namespace LieDerivation
Remove open Algebra
Replace Algebra A with LieAlgebra L everywhere
Replace Derivation R A M with LieDerivation R L M everywhere
Delete field map_one_eq_zero' in instances, since we don't require it in the definition of Lie derivation
Adjust simp only tactics for leibniz' field in instances
Add lemma apply_lie_eq_add to restore usual Leibniz rule in Lie algebras (since our definition relies on left actions, so has a minus sign).
There is no unit 1 in a Lie algebra. We delete all related theorems

map_one_eq_zero (there is no 1 in a Lie algebra)
map_coe_nat (there is no "n : Nat" in a Lie algebra)
map_coe_int (there is no "n : Int" in a Lie algebra)
leibniz_pow (there is "power" in a Lie algebra)
leibniz_of_mul_eq_one (there is no [a,b]= 1 in a Lie algebra)
leibniz_inv and leibniz_invOf (no inverse in a Lie algebra)
map_algebraMap (no scalars in the Lie algebra)

Derivation has an mk' constructor to bypass the map_one_eq_zero' field in an AddCancelCommMonoid M. Since we don't require this property for Lie derivations, we delete everything related:  mk', coe_mk', coe_mk'_linearMap (the whole section Cancel)
Derivations have a theorem eqOn_adjoin to prove that if two derivations are equal on a set, they are equal on the subalgebra spanned by this set. We write it as eqOn_lieSpan. Since the proof of this result relied on an induction principle RingTheory.Adjoin.Basic.adjoin_induction, we needed a similar principle for Lie spans. We added it as the Algebra.Lie.Submodule.lieSpan_induction theorem in the Algebra.Lie.Submodule file.
Add a SMulBracketCommClass class to encode commutativity of scalar multiplication and left Lie action (as a substitute for SMulCommClass S L M), see section Scalar. Introduce instances for S = Nat and S = Int.
Unwrap useless section, useless variables
Delete compAlgebraMap theorem, related with a tower of algebras, since a Lie algebra cannot serve as a scalar field for another one.
Delete section Pushforward (and everything inside), as it was related to composing an A-linear map with a derivation. While this works when A is an associative algebra, one cannot use a Lie algebra in this role since it is not associative.
Could not figure out what the equivalent instance of IsCentralScalar S (Derivation R A M) for a Lie derivation should be, so I skipped this instance.

Thank you for this first review. I think that I have accepted and/or fixed each item.

bors merge


golang-nuts - banned?

My apologies for the off topic issue, but since I can't post to golang-nuts I was not sure where else to raise this.
I appear to have been banned from the golang-nuts group, but am not sure why. Is it possible to clarify why, or possibly reinstate my membership? It would have been through my google account<EMAIL_ADDRESS>with an additional email address<EMAIL_ADDRESS>Thanks.

I'm a moderator of that group but I see neither of those two email addresses listed as being either a member or banned.
Try joining the group first?

thanks for the quick response. i wonder if my membership is in some kind of broken state.

in response to an empty mail sent to<EMAIL_ADDRESS>I get

Hello<EMAIL_ADDRESS>The group you attempted to subscribe to is invite only.  You must be invited by the group owner.
If you have questions related to this or any other Google Group, visit the Help Center at https://groups.google.com/support/.
Thanks,
Google Groups
* there is no `join group` button when I visit https://groups.google.com/forum/#!forum/golang-nuts
* when I click `new topic` it takes me to the compose view, and when I try to change my display name (just to try), I see the following prompt and clicking on `join group to change` has no effect
<img width="512" alt="screenshot" src="https://cloud.githubusercontent.com/assets/51256/12103051/22f7271c-b2f4-11e5-9ca2-eb935bbb501e.png">
* when I try to post a test msg via the web UI, I get `You do not have permission to access this content. (#554)`

Well, I tried to invite you and I got:
No new invitations were sent
One person has been banned from the group
<EMAIL_ADDRESS>
So, I guess you are banned. Let me try to figure out this UI enough to see where.

Unfortunately
a) you did get banned somehow
b) the Google Groups interface makes fixing this close to impossible
We have 55,511 banned members at https://groups.google.com/forum/#!managemembers/golang-nuts/members/banned
But there's no search over them and I can only go through the list one page at a time. See https://productforums.google.com/forum/#!category-topic/Apps/LEt9s3tHecE for other people complaining about this.

It looks like maybe there's an API for Google Groups? Is it https://developers.google.com/admin-sdk/directory/v1/guides/manage-groups ? Maybe we can use that to write a tool to fix this.
/cc @ianlancetaylor @adg

The trick is to edit the search URL (s/active/banned) to search the banned user list:
<EMAIL_ADDRESS>I unbanned<EMAIL_ADDRESS>I'll close this issue assuming the problem is resolved.

thank you.
i sent an email with a new topic from<EMAIL_ADDRESS>and didn't see it show up in the web ui after ~30-40 minutes, so tried posting from the web ui, and got feedback that my topic would be posted after it was approved (wasn't able to grab a screenshot of the specific language used).
so, assuming the msg actually shows up at some point, it appears to be all good - hoping this first msg (which might now be a double post) is just subject to additional moderation.

I just approved your post (and deleted the duplicate). Your future posts
should go through without need for approval.
On 5 January 2016 at 19:46, Liam Staskawicz<EMAIL_ADDRESS>wrote:

thank you.
i sent an email with a new topic from<EMAIL_ADDRESS>and didn't see it
show up in the web ui after ~30-40 minutes, so tried posting from the web
ui, and got feedback that my topic would be posted after it was approved
(wasn't able to grab a screenshot of the specific language used).
so, assuming the msg actually shows up at some point, it appears to be all
good - hoping this first msg (which might now be a double post) is just
subject to additional moderation.
—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/13819#issuecomment-168935875.

thanks again.

i think there's still a problem (sorry) :/
follow up posts to the original msg submitted via email<EMAIL_ADDRESS>are not showing up in the web ui (though I am receiving other posts to the list on that address).
posting via the web ui results in an initial successful response, and then the message shows up as deleted in the discussion thread (my two attempts show as deleted there for me).
any suggestions? maybe<EMAIL_ADDRESS>also needs to be unbanned, since that address is also associated with my google account?

Your replies were marked as spam, and therefore were waiting for moderator approval.  I approved one of them.

gah, ok. thanks again, and sorry for the noise.


documentation: clarify python dependency requirement

Not being familiar with python, I had some trouble getting the extension to work as pip3 could not find the caseconverter package. Turns out I needed to run pip3 install case-converter (with a hyphen).
Maybe this is worth mentioning in the README ?
Very happy otherwise. Thanks for this extension!

Thanks for your comment!
I don't often work with python myself too.
So I just accidentally used the name of the library from the import and didn't check it...