Fix scrollbar issues in ScrollContainer

Fix scrollbars ignoring panel margins.
Fix scrollbars and children not updating immediately on theme/RTL changes.

This is more due to the bigger issue that clipping can only happen at the edge of Controls, and the borders can still be useful if only one scroll is active.
Besides, the current state is even worse:

The current state is that the panel margin appears nicely around the contents:
https://github.com/godotengine/godot/assets/2223172/a44a57e6-e405-4a71-abda-28c8c75ca26c
idk what happened in your screenshot.

Have one of the sides just expand instead of giving it a large minimum size and you will see.

I'm still getting the same results. Can you share the scene?

scroll-bug.zip

Ah, so it's because of the borders. If you remove the borders and instead give it a margin, it looks good.
Well, what I posted initially still looks weird, so hard to say which one is better.

It's only weird if both scrolls are active, and besides, using this buggy behavior the create separation between the scrolls and content is hacky, and should be substituted which proper theme constants.
Also, fixing this allows to create a separation between the scrolls and the borders, which will be handy for themes that have scrolls with shadows that get clipped by the container.

By "weird" I mean the space between scrollbar and border. It makes the scroll float above the content.

using this buggy behavior the create separation between the scrolls and content is hacky, and should be substituted which proper theme constants.

Not sure what do you mean here. The gap is caused by your changes. I just added a content margin to the StyleBox.
I checked how your scene looks with this PR and scrollbars look better with the border, but... eh. I think perfectly the contents should be clipped inside the middle part, so it doesn't go under scrollbars and margins. That's not really possible though.

By "weird" I mean the space between scrollbar and border. It makes the scroll float above the content.

That also happens without a border, so I don't know what you're getting at?

Not sure what do you mean here. The gap is caused by your changes. I just added a content margin to the StyleBox.

Talking about your previous statement:
"Panel margins are more useful for the ScrollContainer's contents, not the scrollbars. If you want margins around scrollbars, you can wrap it in a PanelContainer."

Here the scrollbar is at the border. The dark gray area is stylebox margin. It nicely wraps around the content.

With your changes, the margin appears behind the scrollbar, creating an ugly gap:

With border, the former case looks worse, while the latter is a bit better.
Also, looking at it again, it's not something you can achieve with a PanelContainer like I thought before.
Anyway, it's not like either case is perfect, so whatever. I'll just do a code review later.

Thanks!


Typescript type definitions

Similar to https://github.com/danakt/node-etherscan-api/blob/master/index.d.ts

wen type definitions?


Build test

Codecov Report
Base: 100.00% // Head: 100.00% // No change to project coverage :thumbsup:

Coverage data is based on head (c3fe09d) compared to base (d144f6f).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #29   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            2         2
  Lines           45        45
=========================================
  Hits            45        45

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Library name change request - AVR-IoT VEML3328

Library repository URL
https://github.com/microchip-pic-avr-solutions/veml3328_arduino_driver

Hi @M68426. I am confirming your request.
Unlike submissions, this type of request is still handled manually. The Library Manager index maintainer occasionally comes through to clear all the outstanding requests. They will close this issue when it is done. You will get a notification from GitHub at that time.
After that, you can monitor the indexing of the library via its dedicated logs web page.
Information about that here:
https://github.com/arduino/library-registry/blob/main/FAQ.md#can-i-check-on-library-releases-being-added-to-library-manager

Done in https://github.com/arduino/library-registry/pull/1619#issuecomment-1188819147


PMID:16797182

increased chromatin silencing at centromere outer repeat
increased protein localization to centromere inner repeat
abolished ribonuclease activity

increased chromatin silencing at centromere outer repeat FYPO:0006299
increased protein localization to chromatin at centromere inner repeat FYPO:0006300
abolished ribonuclease activity FYPO:0006301
edit file: f04954f96789e7ea0c9689d4797698e7288f9ea0
release: c736114245c09d058bbe81c0134f1b61b2e3e6b9


added the documentation for kusd transfer from acala to imbue

Added the initial documentation that shows how to transfer kusd from acala to our imbue network

We probably want a dedicated XCM config

added a separate section for xcm with a clickable badge. Will keep on adding additional stuff related to xcm there


How to migrate from v5.0.3 to v6.0.0?

I have full sync cardano-node and postgres.
I tried to migrate 5.0.3 to 6.00 but I got this error
postgres                | 2020-11-03 09:34:07.550 UTC [138] ERROR:  type "outsum" already exists
postgres                | 2020-11-03 09:34:07.550 UTC [138] CONTEXT:  SQL statement "CREATE DOMAIN outsum AS word128type;"
postgres                | 	PL/pgSQL function migrate() line 14 at EXECUTE
postgres                | 2020-11-03 09:34:07.550 UTC [138] STATEMENT:  SELECT migrate();

Is there any way to migrate?
Or Do I have to sync from the beginning?
I am using docker

As per release notes:
As this release has breaking changes, the database has to be dropped and recreated if upgrading.

@mmahut Thank you


Fixed passing labels to model in adapter training notebook

Quick fix for the adapter training notebook by explicitly avoiding the label columns to be removed by the HF Trainer. Diff is pretty ugly, the only relevant changes is the last arg in:
training_args = TrainingArguments(
    learning_rate=1e-4,
    num_train_epochs=6,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=32,
    logging_steps=200,
    output_dir="./training_output",
    overwrite_output_dir=True,
    # The next line is important to ensure the dataset labels are properly passed to the model
    remove_unused_columns=False,
)

Not sure if I understood you correctly. What should the classifier output instead of LABEL_1?

Thanks, great catch! Fixed it.


Add a visualizer

This PR adds a visualization helper (#29). It's in a pretty rough state currently, but it does work and produces useful visualizations for 2D buffers. Here are two of the schedules from the stencil_chain pipeline test:

https://codepen.io/Dillon-Sharlet/pen/abMEpdm
https://codepen.io/Dillon-Sharlet/pen/QWoaGeY

This works by printing the pipeline's body to a javascript implementation, and then using that to generate the visualization by drawing on canvas elements instead of actually running the pipeline. The artifact it produces is a self-contained HTML file that can be viewed in a browser (only tested in chrome).
I think someone that knows what they are doing with HTML and CSS could make the output look much more presentable with a small amount of work, but I've reached my limit of CSS struggles for now.

would be good to follow up with an example of usage in apps/


Skill Going to negative

We have lot of custom item that give a skill bonus. Sometimes, player equip them but don't need the skill so their ARROW pointing down

Since they have a bonus for the item, the skill going down until it reach 0. WHen they unequip the item, BANG!!!!!  skill go negative.

When skill is negative, Sometime, When server resync or save, the skill go back to (100 + negative value) With this example  100+ -20 = 80
It's normal? Core fix possible or I change on script?
I'll post this item for exemple:
[ITEMDEF i_pickaxe_sup]
ID=i_pickaxe

ON=@Create
HITPOINTS={31 60}
color=07a1
ON=@equip
src.mining=<src.mining>+200
ON=@unequip
src.mining=<src.mining>+-200

should not go to negative.
but also using wildly +20 -20 is not a good practice, without an if or some check

Yes can make a if but will not work for all. The important part is not go negative when it drop of 0.1 when arrow is down.

I think this is better safe way:
[ITEMDEF i_pickaxe_sup]
ID=i_pickaxe
BonusSkill1=Mining
BonusSkill1Amt=20.0

ON=@Create
HITPOINTS={31 60}
color=07a1

Yes for this exemple. Thx for info. Our custom item will not work. Possible to check on skill change if <0. Force to 0

if skill < bonus
skill= 0
endif

Made an easy fix in the script. Modif in core is useless? Maybe a feature.
ON=@Skillchange	//Utilisé pour éviter d'avoir un skill négatif
IF (<ARGN2> > 10000) // Go to negative -1 = 65526
	ARGN2=0
ENDIF

I'll add an additional check and warning in the source, but rather than considering this as a core bug, i see it more as a script error

Don't seem to work when I unequip a special item

This is not a Sphere issue, if you change skill value with script, you have to check while unequip if he still has enough skill or you have to block skill drop under skill requirement with @SkillChange or @SkillGain triggers.
Okay I agree when skills goes under 0, it bugged and get higher but, skills shouldn't be less than zero. So I suppose you to make sure noone can drop their skills under items bonus (maybe you can give a tag to save overrided values like TAG0.MINING += 200 and when he try to lower their skill you can block the skill goes under that value. Another fix could be check skillchange if the skill goes under 0 to set 0 and if goes more than skillclass maximum, you can set it skillclass' maximum.
But on the other hand, yeah I agree that 6k bug should be fixed, but as I say this is not a sphere bug totally, most of the bug caused by script's itself.

Thx I put This in my script:
ON=@Skillchange	//Use for avoiding to have negative skill
IF (<ARGN2> > 10000) // Go to negative -1 = 65526
	ARGN2=0
ENDIF


Tests for FragmentElementImpl

Wrong equals or hashCode implementations will result in silently breaking behaviours in DefaultFragmentStackPatcher .
e.g. fragments will be removed to early, since they cannot be found in the newStack

This also requires that the ElementImpl implements a correct hashCode and equals function!


Added support for the DRFDocs module to auto-generate HTTP API docume…

…ntation and expose it on /docs.

:+1: thank you!


Painless guidance for Mutliple Project Support

Basically files that need to work on server as well as client. Plus we need good TS debugging experience (more important once generators / async land).
For the server code we compile it using commonjs with source maps.

I don’t understand why this is being committed to this repository?

I don’t understand why this is being committed to this repository? This isn’t relevant to IDE codeintel…

Just like other examples we have for testing. This will be for testing multi target support i.e client code + server code + common code. All using tsconfig files.
When we start adding tests these will be the code samples that should continue to work. Or when working on feature PRs right now we can run their code against these examples ourselves.
I definitely need to add ignore on publish.

How are these examples used for testing?

Manually at the moment. I open the file in the ts folder. Change it. And make sure we get the right js at the right place.

refs https://github.com/TypeStrong/atom-typescript/issues/26

I am thinking the easiest solution would be to just symlink : http://stackoverflow.com/questions/954560/what-does-git-do-to-files-that-are-a-symbolic-link
This way when you are editing common files "which project you are working on" gets driven by the file system and not our code.
+ves:

Clear what active project is from the file path. Don't need to come up with hacky locating of relevant tsconfig.json files for common.
Don't have to deal with possible duplicates as in "errors from project client as well as errors from project server"

-ves:

You need to open and save the file twice if you want the change to take place for both client / server projects. This is a big negative as it kills "just start coding" philosophy.

@fdecampredon / @derekcicerone do you have a suggestion in this area?

We've tried both approaches actually: before the Eclipse plug-in supported multiple projects, we just used symlinks (like bower link) to make it work. Then we added explicit support though honestly we haven't done anything that makes it better than the symlinks so far.

You need to open and save the file twice if you want the change to take place for both client / server projects. This is a big negative as it kills "just start coding" philosophy.

Using the same module format (AMD or, maybe eventually, ES) on client and server doesn’t have this problem, nor does running a watcher or Node.js server for e.g. browserify if you build out to CJS. Why would you try to shove this into code intel plugin for Atom?

@csnover that is the answer that I was looking for. A single project for client + server + test, is that the recommendation? If so then the whole of milestone 3 https://github.com/TypeStrong/atom-typescript/milestones/ can go away.

A single project for client + server

I see no way of achieving this with outDir. Perhaps outDir is a bad idea.

Also source map line numbers are wrong. Everything is pointing to line 6 (the last line in the file).

My fault. I really didn't expect this to work.

A loader seems to be the only way to get a reliable sourcemap with correct line numbers

TODO: investigate error reporting on Build, Add an no emit js option under tsconfig:atom:emit:boolean

This issue is too old and thus closing. Some portions of the concerns here will be delt with #492 :rose:


Avoid count_nonzero calls

We should be able to purely load data with this trick:
import dask_awkward as dak

def load_columns(events, columns):
  for col in columns:
    _ = events[col]
  return events

dak.map_partitions(load_columns, events, [("Jet", "pt"), ("Jet", "eta"), ...])

This way we avoid using unnecessary ak.count_nonzero calls in materialize_branches.ipynb

This might also help keep memory requirements low, looks like a good setup to try out.


Fix missing logs for appserver

See https://github.com/puma/puma/commit/03ed48ca903879f3b9a649f5df1cf9d6577ef3d2

Isn't that something the app should do itself when needed? Otherwise this produces a lot of noise, when not needed. Especially if you have a lot of status check requests.

I don't see a point in feeding the output to logger if there's never any output. You can filter the app logs out by tag if necessary.

Also, that behavior has changed because of the puma update, not because we explicitly decided it.

Ah okay, so this suppresses all app log output? That's not what we want, of course.

Exactly.

🍄
(please add the change to CHANGELOG and squash-merge when tests are green)


add command for updating gomeasure to the latest version (for Linux and Windows)

Different proposed solutions

checking the latest versions from github release (fastest)

get https://api.github.com/repos/lordvidex/gomeasure/tags OR get https://api.github.com/repos/lordvidex/gomeasure/releases/latest
sort the list and pick the max tag payload {$0}.name and compare with --version output
then rerun install.sh script (to update) if there is a mismatch

downloading the checksums.txt file / downloading the latest binaries and confirming if checksums are the same (more accurate method, but slower)

@escalopa can you push to the new branch and create a PR, so we can track your progress


Is zuul2 preformance better than zuul1?

We built gateway with zuul1 and hystrix， now we want to upgrade  our zuul1 to zuul2, but the pressure test shows qps is much lower than zuul1(zuul1 1500+, zuul2 500+). Then I set netty worker to 20(cpu 4 core) and it reachs to 1100. Do you have some advice for best practices?

If you're doing a load test, try using the benchmark properties file. Generally, performance in Z2 should be better than Z1 (although roughly similar). Also, in Z2 you should generally have either 1 or 2 workers per CPU. Any more than that and you'll get diminishing returns.
Keep in mind that you can't block at all in Netty. You can't just port your code directly to Z2 unless you're positive that there's no blocking at any point.

Change the async executor in BaseZuulFilterRunner#filter(final ZuulFilter<I, O> filter, final I inMesg) to a custom fixed thread pool fixed this problem, cause we use hystrix with sync mode and it  needs plenty of thread to handle IO operations.

@AngryApe the qps base lines you are using for zuul1, is that the qps for a single instance of zuul1? or was it against a cluster. Also could you please share the server configuration?

@artgon if I've an origin server that can handle upto 4000 req/sec is it ok to assume that adding zuul2 in front would not cause considerable degradation?

@AngryApe Zuul 2 doesn't work that way. It's completely async, so you can't wait on requests in a blocking fashion. You shouldn't need to use Hystrix on the origin request.


Fixed two small typos in Canvas

Just a small PR fixing two typos in Canvas class.
What's the "correct way" to submit typo fixes? Do idependent PRs work, considering the fact that I'm not doing anything else in this part of the code?

Hooray Jenkins reported success with all tests good!

@vampcat separate PR is fine, although small fixes like this could also be done as "side effect" in a different PR... I'll just merge this now ;-)


Display a message about missing C# support on Android/iOS/Web platforms

When trying to export a C# project, this displays an error message after creating a export preset for an unsupported platform.
Support for these platforms is planned for a future release.

See https://github.com/godotengine/godot/issues/70796.

Preview

Thanks!

It should be possible to export non-C# games with a .NET-enabled version of Godot. But I agree that it's an improvement since many users were confused because there's no indication when exporting a C# game. So I'm OK with this :+1:.

Technically there's no export template provided, so users would need to copy export templates from the non-.NET build anyway, so it makes sense to recommend just using that editor build directly.


Wrapper around sql extension - more powerful tools

Write a wrapper around the sql extension that provides ease of use and a framework to work with this extension.

This should provide things like:

Standardise a way to interact with the database.
Provide the flexibility to not provide any data from feature files as a standard.
Ability to perform multiple inserts efficiently and at once.
Ability to easily fetch data from the keystore.
Handling of common data types such as DateTime and Money objects.
Provide easy configuration object.

Done https://packagist.org/packages/genesis/sql-api-wrapper


Only TXT records are stored in cache causing out-of-sync state

The issue I am seeing is that when external-dns creates new records with relevant TXT records it succeeds on the first sync, however, on the second sync it still thinks it needs to create new records. This then causes the sync to fail and due to https://github.com/kubernetes-incubator/external-dns/issues/421 it then doesn't sync any other records. I believe this is linked to https://github.com/kubernetes-incubator/external-dns/pull/589 and the way the cache is created/kept in sync.
To confirm this I ran the following:

Sync time of 10s and cache sync of 1 hour

{"level":"debug","msg":"Considering zone: /hostedzone/MYZONE (domain: my.domain.)","time":"2018-06-19T10:52:03Z"}
{"level":"debug","msg":"Adding test.my.domain. to zone my.domain. [Id: /hostedzone/MYZONE]","time":"2018-06-19T10:52:03Z"}
{"level":"debug","msg":"Adding external-dns_test.my.domain. to zone my.domain. [Id: /hostedzone/MYZONE]","time":"2018-06-19T10:52:03Z"}
{"level":"info","msg":"Desired change: CREATE test.my.domain A","time":"2018-06-19T10:52:03Z"}
{"level":"info","msg":"Desired change: CREATE external-dns_test.my.domain TXT","time":"2018-06-19T10:52:03Z"}
{"level":"info","msg":"Record in zone my.domain. were successfully updated","time":"2018-06-19T10:52:03Z"}
{"level":"debug","msg":"Using cached records.","time":"2018-06-19T10:52:22Z"}
{"level":"debug","msg":"Endpoints generated from ingress: kube-system/test: [test.my.domain 0 IN CNAME lb.elb.amazonaws.com]","time":"2018-06-19T10:52:22Z"}
{"level":"debug","msg":"Skipping endpoint external-dns_test.my.domain 0 IN TXT \"heritage=external-dns,external-dns/owner=external-dns,external-dns/resource=ingress/kube-system/test\" because owner id does not match, found: \"\", required: \"external-dns\"","time":"2018-06-19T10:52:22Z"}
{"level":"debug","msg":"Considering zone: /hostedzone/MYZONE (domain: my.domain.)","time":"2018-06-19T10:52:22Z"}
{"level":"debug","msg":"Adding test.my.domain. to zone my.domain. [Id: /hostedzone/MYZONE]","time":"2018-06-19T10:52:22Z"}
{"level":"debug","msg":"Adding external-dns_test.my.domain. to zone my.domain. [Id: /hostedzone/MYZONE]","time":"2018-06-19T10:52:22Z"}
{"level":"info","msg":"Desired change: CREATE test.my.domain A","time":"2018-06-19T10:52:22Z"}
{"level":"info","msg":"Desired change: CREATE external-dns_test.my.domain TXT","time":"2018-06-19T10:52:22Z"}
{"level":"error","msg":"InvalidChangeBatch: Tried to create resource record set [name='test.my.domain.', type='A'] but it already exists\n\tstatus code: 400, request id: my-aws-request-id","time":"2018-06-19T10:52:22Z"}
.
.
.

As you can see, it successfully creates the records required at the start, however, it doesn't seem to update the cache after this, causing subsequent runs to error talking to aws (until the cache is re-synced after 1 hour).

Sync time of 10s and cache sync of 1 second

This gets around the problem in a hacky way by disabling the cache. A quick fix would be to implement an option to disable the cache completely? With this setup I was able to get the records in place without errors so I guess it's a workaround.

Sync time of 10s and cache sync of 0 seconds

This is an interesting one, if you set this then you won't get any TXT records at all, but it does succeed to create the initial records.
My current setup is using the aws route53 provider with the txt registry with all default values (1 hour cache sync).
Update:
From having a quick look at the code, it seems that when calling
https://github.com/kubernetes-incubator/external-dns/blob/e34bf552d6c6be69b2c08c371a5e83014e50e06b/registry/txt.go#L62-L68 and using the cache, the function only returns the TXT records and not the records we want themselves. This then means that external-dns will keep on trying to create new records until it does a re-sync. To get around this you could add r as part of https://github.com/kubernetes-incubator/external-dns/blob/e34bf552d6c6be69b2c08c371a5e83014e50e06b/registry/txt.go#L118 to the cache, however, this would mean the cache is used for both TXT records and desired records.

We also saw issues deleting a record, which looks like it's also related to the new cache code:
time="2018-06-20T13:30:02Z" level=debug msg="Adding zalenium-k8s-app.example.org. to zone example.org. [Id: /hostedzone/<zone>]"
time="2018-06-20T13:30:02Z" level=debug msg="Adding zalenium-k8s-app.example.org. to zone example.org. [Id: /hostedzone/<zone>]"
time="2018-06-20T13:30:02Z" level=info msg="Desired change: DELETE zalenium-k8s-app.example.org A"
time="2018-06-20T13:30:02Z" level=info msg="Desired change: DELETE zalenium-k8s-app.example.org TXT"
time="2018-06-20T13:31:02Z" level=debug msg="Adding zalenium-k8s-app.example.org. to zone example.org. [Id: /hostedzone/<zone>]"
time="2018-06-20T13:31:02Z" level=debug msg="Adding zalenium-k8s-app.example.org. to zone example.org. [Id: /hostedzone/<zone>]"
time="2018-06-20T13:31:02Z" level=info msg="Desired change: DELETE zalenium-k8s-app.example.org A"
time="2018-06-20T13:31:02Z" level=info msg="Desired change: DELETE zalenium-k8s-app.example.org TXT"
time="2018-06-20T13:31:02Z" level=error msg="InvalidChangeBatch: Tried to delete resource record set [name=\'zalenium-k8s-app.example.org.', type='A'] but it was not found\n\tstatus code: 400, request id: 2cef7a54-748e-11e8-8c73-2f828130d984"
time="2018-06-20T13:32:02Z" level=debug msg="Adding zalenium-k8s-app.example.org. to zone example.org. [Id: /hostedzone/<zone>]"
time="2018-06-20T13:32:02Z" level=debug msg="Adding zalenium-k8s-app.example.org. to zone example.org. [Id: /hostedzone/<zone>]"
time="2018-06-20T13:32:02Z" level=info msg="Desired change: DELETE zalenium-k8s-app.example.org A"
time="2018-06-20T13:32:02Z" level=info msg="Desired change: DELETE zalenium-k8s-app.example.org TXT"
time="2018-06-20T13:32:03Z" level=error msg="InvalidChangeBatch: Tried to delete resource record set [name=\'zalenium-k8s-app.example.org.', type='A'] but it was not found\n\tstatus code: 400, request id: 511e2c03-748e-11e8-8c73-2f828130d984"
time="2018-06-20T13:33:02Z" level=debug msg="Adding zalenium-k8s-app.example.org. to zone example.org. [Id: /hostedzone/<zone>]"
time="2018-06-20T13:33:02Z" level=debug msg="Adding zalenium-k8s-app.example.org. to zone example.org. [Id: /hostedzone/<zone>]"
time="2018-06-20T13:33:02Z" level=info msg="Desired change: DELETE zalenium-k8s-app.example.org A"
time="2018-06-20T13:33:02Z" level=info msg="Desired change: DELETE zalenium-k8s-app.example.org TXT"

(Sorry, didn't look too close into this yet, just dumping this for later)

Seeing the same issue when deleting ( using r53 ) , It will try to delete again at the next pass ( and getting error)
setting cache at 1s makes it work again.

New version is out, caching is disabled by default and we fixed the caching issue. Feel free to try it out https://github.com/kubernetes-incubator/external-dns/releases/tag/v0.5.4


WIP - Jsonnet vendoring

As suggested by @brancz , move the vendoring to each environment.
Is this breaking downstream project?

We don't really need a vendor folder per environment. They build on each others based on the same dependencies.
How about simply having vendor/ and vendor-jsonnet/ or something here? :thinking:

Base environment should not have a lock file nor should it have the vendor checked in, only the operator environment should. The dev environment should have a local directory dependency pointing to the base environment.

We don't really need a vendor folder per environment. They build on each others based on the same dependencies.
How about simply having vendor/ and vendor-jsonnet/ or something here? thinking

@metalmatze I have tried this way, but Go mode vendor will scan also  vendor-jsonnet/
You can try and see:
jb install --jsonnetpkg-home=jsonnet-vendor
go mod vendor

Maybe we can put the go and operator code in a subdirectory?

Maybe we can put the go and operator code in a subdirectory?

OK, what do you think the structure will be? Where the jsonnetfile.json should be?
Don't Go modules implies to have the go.mod in the root directory?

Don't Go modules implies to have the go.mod in the root directory?

Is this really the case? If so I wasn't aware of it.

Where the jsonnetfile.json should be?

I'm thinking also a separate directory that we just call jsonnet.

A module is a collection of Go packages stored in a file tree with a go.mod file at its root.
from https://blog.golang.org/using-go-modules

The way I read this, github.com/observatorium/configuration/operator would be the "root" here of the mod, so github.com/observatorium/configuration/operator/go.mod should work just fine.

OK, I will try this way

@brancz PTAL
Do you want the jsonnetfile.json/jsonnetfile.lock.json under operator/jsonnet to be link to the one in root?
Still need to handle some changes in Makefile and circleci.

This still has WIP is this ready for review?

I still need to fix the Makefile.
Comments are welcome. :-)

Ready for review.
Do I need to remove the jsonnetfile.lock.json in the root directory and add it to .gitignore?

Do I need to remove the jsonnetfile.lock.json in the root directory and add it to .gitignore?

yes, we should only have the one from the operator environment as that's what produces the checked in vendoring

@brancz  PTAL

@brancz thanks for the guidance!


Return Observable from a Spring MVC Controller

See https://github.com/spencergibb/myfeed/commit/a301aba6139fce7a87a1aab1d1311e6ceb8adfb7 for an example implementation.
In short: create a ObservableReturnValueHandler that returns a DeferredResult that delegates to the Observable.

Can we do the same thing for a Stream (reactor)?


Use require.resolve to lookup contracts in deps

Change lookup in both Dependency and Contracts classes. Instead of looking for just node_modules in the current folder, it uses require.resolve (relative to the current workdir) to look for the dependency.
Fixes #1076

And here as well! :-)


[docs] detail on deviceready dependency

Update docs so that it's explicit that we need to wait for deviceready before PushNotification.init can be called. (See https://github.com/phonegap/phonegap-plugin-push/issues/408, among many others)

@fredgalvao Really? Oh well sure but every PhoneGap/Cordova method must be called after deviceready.

Me too, but the number of issues created related to this is yet more shocking. As is for another number of topics, but oh well.

@fredgalvao I added in a note but feel free to expand upon the explanation.


Error while deployment v20.12 : No matching distribution found for slip==0.4.0

Trying on 20.12
While deploying getting error:
ERROR: Could not find a version that satisfies the requirement slip==0.4.0
2020-12-21_06-46-33_ansible.log

Hi @AmanThakral02!! Did you install on a fresh OS? Remember that 20.12 release is based on CentOS 7.8 Minimal. CentOS 7.8 must be installed on all nodes (controller & workers)

@amr-mokhtar , I am using CentOS 7.8.2003 on both controller and worker

Hi @AmanThakral02,
We're investigating the issue, will get back to you as soon as possible.

@AmanThakral02  It looks that you can remove this dependency from roles/docker/files/requirements.txt
remove following lines:
slip==0.4.0
slip.dbus==0.4.0
Those dependencies will be removed in future releases.


error<EMAIL_ADDRESS>start: http-server -a localhost -p 8000 -c-1

when  npm install I met an error,
here is the log:
0 info it worked if it ends with ok
1 verbose cli [ 'C:\Program Files\nodejs\\node.exe',
1 verbose cli   'C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js',
1 verbose cli   'start' ]
2 info using<EMAIL_ADDRESS>3 info using<EMAIL_ADDRESS>4 verbose node symlink C:\Program Files\nodejs\node.exe
5 verbose run-script [ 'prestart', 'start', 'poststart' ]
6 info prestart<EMAIL_ADDRESS>7 verbose unsafe-perm in lifecycle true
8 info start<EMAIL_ADDRESS>9 verbose unsafe-perm in lifecycle true
10 info<EMAIL_ADDRESS>Failed to exec start script
11 error<EMAIL_ADDRESS>start: http-server -a localhost -p 8000 -c-1
11 error Exit status 8
12 error Failed at the<EMAIL_ADDRESS>start script.
12 error This is most likely a problem with the angular-seed package,
12 error not with npm itself.
12 error Tell the author that this fails on your system:
12 error     http-server -a localhost -p 8000 -c-1
12 error You can get their info via:
12 error     npm owner ls angular-seed
12 error There is likely additional logging output above.
13 error System Windows_NT 6.2.9200
14 error command "C:\Program Files\nodejs\\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "start"
15 error cwd D:\abc\angular-seed-master
16 error node -v v0.10.34
17 error npm -v 1.4.28
18 error code ELIFECYCLE
19 verbose exit [ 1, true ]
it seems the localhost 8000 is not avalilable, please can any body tell me how to fix it?

Do you have any update on it, I have the same issue, may you did solved it?

CyanChoi, have you tried running the app in multiple browsers?
When I tried running the angular seed app in Pale Moon, it threw the above error, but when I did the same in Chrome, it worked fine. I'm guessing it's a browser issue.

please can any body tell me how to fix it For me also have an same error

Have you check if the 8000 address is already in use by other server? (Mamp, wamp, etc)

I have a similar problem,  any one fixed this problem yet?

All dependencies and tools have been updated recently and this is hopefully not an issue any more.
If anyone is still having problems, please open a new issue.


503 UR when use websocket

hi：
I use envoy as the http 1.1 service proxy. When using envoy 1.7 with v1 configuration is ok. After upgrade to envoy 1.9 with  v2 configuration, I have been 503 UR error。
My service is deployed like this

The complete debug log

[2019-09-04 09:37:57.021][000012][debug][http] [source/common/http/conn_manager_impl.cc:200] [C161] new stream
[2019-09-04 09:37:57.021][000012][trace][http] [source/common/http/http1/codec_impl.cc:316] [C161] completed header: key=Host value=<IP_ADDRESS>
[2019-09-04 09:37:57.022][000012][trace][http] [source/common/http/http1/codec_impl.cc:316] [C161] completed header: key=Connection value=Upgrade
[2019-09-04 09:37:57.022][000012][trace][http] [source/common/http/http1/codec_impl.cc:316] [C161] completed header: key=Pragma value=no-cache
[2019-09-04 09:37:57.022][000012][trace][http] [source/common/http/http1/codec_impl.cc:316] [C161] completed header: key=Cache-Control value=no-cache
[2019-09-04 09:37:57.022][000012][trace][http] [source/common/http/http1/codec_impl.cc:316] [C161] completed header: key=User-Agent value=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36
[2019-09-04 09:37:57.022][000012][trace][http] [source/common/http/http1/codec_impl.cc:316] [C161] completed header: key=Upgrade value=websocket
[2019-09-04 09:37:57.022][000012][trace][http] [source/common/http/http1/codec_impl.cc:316] [C161] completed header: key=Origin value=chrome-extension://omalebghpgejjiaoknljcfmglgbpocdp
[2019-09-04 09:37:57.022][000012][trace][http] [source/common/http/http1/codec_impl.cc:316] [C161] completed header: key=Sec-WebSocket-Version value=13
[2019-09-04 09:37:57.022][000012][trace][http] [source/common/http/http1/codec_impl.cc:316] [C161] completed header: key=Accept-Encoding value=gzip, deflate
[2019-09-04 09:37:57.022][000012][trace][http] [source/common/http/http1/codec_impl.cc:316] [C161] completed header: key=Accept-Language value=zh-CN,zh;q=0.9,en;q=0.8
[2019-09-04 09:37:57.022][000012][trace][http] [source/common/http/http1/codec_impl.cc:316] [C161] completed header: key=Sec-WebSocket-Key value=DWGXE2nuMBZyPZDEXuomkw==
[2019-09-04 09:37:57.022][000012][trace][http] [source/common/http/http1/codec_impl.cc:412] [C161] headers complete
[2019-09-04 09:37:57.022][000012][trace][http] [source/common/http/http1/codec_impl.cc:316] [C161] completed header: key=Sec-WebSocket-Extensions value=permessage-deflate; client_max_window_bits
[2019-09-04 09:37:57.022][000012][trace][http] [source/common/http/http1/codec_impl.cc:420] [C161] codec entering upgrade mode.
[2019-09-04 09:37:57.022][000012][debug][http] [source/common/http/conn_manager_impl.cc:537] [C161][S17377287246961922087] request headers complete (end_stream=false):
':authority', '<IP_ADDRESS>'
':path', '/xlchannel.app2amlogic/sio/?EIO=3&transport=websocket'
':method', 'GET'
'connection', 'Upgrade'
'pragma', 'no-cache'
'cache-control', 'no-cache'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36'
'upgrade', 'websocket'
'origin', 'chrome-extension://omalebghpgejjiaoknljcfmglgbpocdp'
'sec-websocket-version', '13'
'accept-encoding', 'gzip, deflate'
'accept-language', 'zh-CN,zh;q=0.9,en;q=0.8'
'sec-websocket-key', 'DWGXE2nuMBZyPZDEXuomkw=='
'sec-websocket-extensions', 'permessage-deflate; client_max_window_bits'

[2019-09-04 09:37:57.022][000012][debug][router] [source/common/router/router.cc:270] [C161][S17377287246961922087] cluster 'xlchannel.app2amlogic' match for URL '/xlchannel.app2amlogic/sio/?EIO=3&transport=websocket'
[2019-09-04 09:37:57.022][000012][debug][router] [source/common/router/router.cc:328] [C161][S17377287246961922087] router decoding headers:
':authority', '<IP_ADDRESS>'
':path', '/xlchannel.app2amlogic/sio/?EIO=3&transport=websocket'
':method', 'GET'
':scheme', 'http'
'connection', 'Upgrade'
'pragma', 'no-cache'
'cache-control', 'no-cache'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36'
'upgrade', 'websocket'
'origin', 'chrome-extension://omalebghpgejjiaoknljcfmglgbpocdp'
'sec-websocket-version', '13'
'accept-encoding', 'gzip, deflate'
'accept-language', 'zh-CN,zh;q=0.9,en;q=0.8'
'sec-websocket-key', 'DWGXE2nuMBZyPZDEXuomkw=='
'sec-websocket-extensions', 'permessage-deflate; client_max_window_bits'
'content-length', '0'
'x-forwarded-for', '<IP_ADDRESS>'
'x-forwarded-proto', 'http'
'x-envoy-internal', 'true'
'x-request-id', 'e1e5b5d7-4fad-4f07-8a6e-1f7e915bfee6'
'x-envoy-expected-rq-timeout-ms', '15000'

[2019-09-04 09:37:57.022][000012][debug][client] [source/common/http/codec_client.cc:26] [C162] connecting
[2019-09-04 09:37:57.022][000012][debug][connection] [source/common/network/connection_impl.cc:634] [C162] connecting to <IP_ADDRESS>:9001
[2019-09-04 09:37:57.022][000012][debug][connection] [source/common/network/connection_impl.cc:643] [C162] connection in progress
[2019-09-04 09:37:57.023][000012][debug][http2] [source/common/http/http2/codec_impl.cc:721] [C162] setting stream-level initial window size to 268435456
[2019-09-04 09:37:57.023][000012][debug][http2] [source/common/http/http2/codec_impl.cc:743] [C162] updating connection-level initial window size to 268435456
[2019-09-04 09:37:57.023][000012][debug][pool] [source/common/http/conn_pool_base.cc:20] queueing request due to no available connections
[2019-09-04 09:37:57.023][000012][trace][http] [source/common/http/conn_manager_impl.cc:772] [C161][S17377287246961922087] decode headers called: filter=0x2502370 status=1
[2019-09-04 09:37:57.023][000012][trace][http] [source/common/http/http1/codec_impl.cc:433] [C161] message complete
[2019-09-04 09:37:57.023][000012][trace][http] [source/common/http/http1/codec_impl.cc:438] [C161] Pausing parser due to upgrade.
[2019-09-04 09:37:57.023][000012][trace][http] [source/common/http/http1/codec_impl.cc:369] [C161] parsed 586 bytes
[2019-09-04 09:37:57.023][000012][trace][http] [source/common/http/http1/codec_impl.cc:342] [C161] direct-dispatched 0 bytes
[2019-09-04 09:37:57.032][000012][trace][connection] [source/common/network/connection_impl.cc:439] [C162] socket event: 2
[2019-09-04 09:37:57.032][000012][trace][connection] [source/common/network/connection_impl.cc:507] [C162] write ready
[2019-09-04 09:37:57.032][000012][debug][connection] [source/common/network/connection_impl.cc:516] [C162] connected
[2019-09-04 09:37:57.032][000012][debug][client] [source/common/http/codec_client.cc:64] [C162] connected
[2019-09-04 09:37:57.032][000012][debug][pool] [source/common/http/http2/conn_pool.cc:83] [C162] creating stream
[2019-09-04 09:37:57.032][000012][debug][router] [source/common/router/router.cc:1023] [C161][S17377287246961922087] pool ready
[2019-09-04 09:37:57.032][000012][trace][http2] [source/common/http/http2/codec_impl.cc:554] [C162] send data: bytes=24
[2019-09-04 09:37:57.032][000012][trace][connection] [source/common/network/connection_impl.cc:376] [C162] writing 24 bytes, end_stream false
[2019-09-04 09:37:57.032][000012][trace][http2] [source/common/http/http2/codec_impl.cc:554] [C162] send data: bytes=21
[2019-09-04 09:37:57.032][000012][trace][connection] [source/common/network/connection_impl.cc:376] [C162] writing 21 bytes, end_stream false
[2019-09-04 09:37:57.032][000012][trace][http2] [source/common/http/http2/codec_impl.cc:508] [C162] sent frame type=4
[2019-09-04 09:37:57.032][000012][trace][http2] [source/common/http/http2/codec_impl.cc:554] [C162] send data: bytes=13
[2019-09-04 09:37:57.032][000012][trace][connection] [source/common/network/connection_impl.cc:376] [C162] writing 13 bytes, end_stream false
[2019-09-04 09:37:57.032][000012][trace][http2] [source/common/http/http2/codec_impl.cc:508] [C162] sent frame type=8
[2019-09-04 09:37:57.032][000012][trace][http2] [source/common/http/http2/codec_impl.cc:554] [C162] send data: bytes=505
[2019-09-04 09:37:57.032][000012][trace][connection] [source/common/network/connection_impl.cc:376] [C162] writing 505 bytes, end_stream false
[2019-09-04 09:37:57.032][000012][trace][http2] [source/common/http/http2/codec_impl.cc:508] [C162] sent frame type=1
[2019-09-04 09:37:57.032][000012][trace][connection] [source/common/network/connection_impl.cc:507] [C162] write ready
[2019-09-04 09:37:57.032][000012][trace][connection] [source/common/network/raw_buffer_socket.cc:62] [C162] write returns: 563
[2019-09-04 09:37:57.032][000012][trace][connection] [source/common/network/connection_impl.cc:439] [C162] socket event: 2
[2019-09-04 09:37:57.032][000012][trace][connection] [source/common/network/connection_impl.cc:507] [C162] write ready
[2019-09-04 09:37:57.042][000012][trace][connection] [source/common/network/connection_impl.cc:439] [C162] socket event: 3
[2019-09-04 09:37:57.042][000012][trace][connection] [source/common/network/connection_impl.cc:507] [C162] write ready
[2019-09-04 09:37:57.042][000012][trace][connection] [source/common/network/connection_impl.cc:477] [C162] read ready
[2019-09-04 09:37:57.042][000012][trace][connection] [source/common/network/raw_buffer_socket.cc:21] [C162] read returns: 56
[2019-09-04 09:37:57.042][000012][trace][connection] [source/common/network/raw_buffer_socket.cc:21] [C162] read returns: -1
[2019-09-04 09:37:57.042][000012][trace][connection] [source/common/network/raw_buffer_socket.cc:29] [C162] read error: 11
[2019-09-04 09:37:57.042][000012][trace][http2] [source/common/http/http2/codec_impl.cc:340] [C162] dispatching 56 bytes
[2019-09-04 09:37:57.042][000012][trace][http2] [source/common/http/http2/codec_impl.cc:398] [C162] recv frame type=4
[2019-09-04 09:37:57.042][000012][trace][http2] [source/common/http/http2/codec_impl.cc:398] [C162] recv frame type=4
[2019-09-04 09:37:57.042][000012][trace][http2] [source/common/http/http2/codec_impl.cc:398] [C162] recv frame type=8
[2019-09-04 09:37:57.042][000012][trace][http2] [source/common/http/http2/codec_impl.cc:398] [C162] recv frame type=3
[2019-09-04 09:37:57.042][000012][trace][http2] [source/common/http/http2/codec_impl.cc:494] [C162] remote reset: 1
[2019-09-04 09:37:57.042][000012][debug][http2] [source/common/http/http2/codec_impl.cc:563] [C162] stream closed: 1
[2019-09-04 09:37:57.042][000012][debug][client] [source/common/http/codec_client.cc:105] [C162] request reset
[2019-09-04 09:37:57.042][000012][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-09-04 09:37:57.042][000012][debug][pool] [source/common/http/http2/conn_pool.cc:222] [C162] destroying stream: 0 remaining
[2019-09-04 09:37:57.042][000012][debug][router] [source/common/router/router.cc:481] [C161][S17377287246961922087] upstream reset
[2019-09-04 09:37:57.042][000012][debug][http] [source/common/http/conn_manager_impl.cc:1188] [C161][S17377287246961922087] encoding headers via codec (end_stream=false):
':status', '503'
'content-length', '57'
'content-type', 'text/plain'
'date', 'Wed, 04 Sep 2019 09:37:56 GMT'
'server', 'envoy'
'connection', 'close'

[2019-09-04 09:37:57.042][000012][trace][connection] [source/common/network/connection_impl.cc:376] [C161] writing 153 bytes, end_stream false
[2019-09-04 09:37:57.042][000012][trace][http] [source/common/http/conn_manager_impl.cc:1304] [C161][S17377287246961922087] encoding data via codec (size=57 end_stream=true)
[2019-09-04 09:37:57.042][000012][trace][connection] [source/common/network/connection_impl.cc:376] [C161] writing 57 bytes, end_stream false
[2019-09-04 09:37:57.042][000012][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-09-04 09:37:57.042][000012][debug][connection] [source/common/network/connection_impl.cc:101] [C161] closing data_to_write=210 type=2
[2019-09-04 09:37:57.042][000012][debug][connection] [source/common/network/connection_impl.cc:153] [C161] setting delayed close timer with timeout 1000 ms
[2019-09-04 09:37:57.042][000012][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=3)
[2019-09-04 09:37:57.042][000012][trace][http2] [source/common/http/http2/codec_impl.cc:355] [C162] dispatched 56 bytes
[2019-09-04 09:37:57.042][000012][trace][http2] [source/common/http/http2/codec_impl.cc:554] [C162] send data: bytes=9
[2019-09-04 09:37:57.042][000012][trace][connection] [source/common/network/connection_impl.cc:376] [C162] writing 9 bytes, end_stream false
[2019-09-04 09:37:57.042][000012][trace][http2] [source/common/http/http2/codec_impl.cc:508] [C162] sent frame type=4
[2019-09-04 09:37:57.042][000012][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=3)
[2019-09-04 09:37:57.043][000012][trace][connection] [source/common/network/connection_impl.cc:439] [C162] socket event: 2
[2019-09-04 09:37:57.043][000012][trace][connection] [source/common/network/connection_impl.cc:507] [C162] write ready
[2019-09-04 09:37:57.043][000012][trace][connection] [source/common/network/raw_buffer_socket.cc:62] [C162] write returns: 9
[2019-09-04 09:37:57.043][000012][trace][connection] [source/common/network/connection_impl.cc:439] [C161] socket event: 2
[2019-09-04 09:37:57.043][000012][trace][connection] [source/common/network/connection_impl.cc:507] [C161] write ready
[2019-09-04 09:37:57.043][000012][trace][connection] [source/common/network/raw_buffer_socket.cc:62] [C161] write returns: 210
[2019-09-04T09:37:57.021Z] "GET /xlchannel.app2amlogic/sio/?EIO=3&transport=websocket HTTP/1.1" 503 UR 0 57 21 - "<IP_ADDRESS>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" "e1e5b5d7-4fad-4f07-8a6e-1f7e915bfee6" "<IP_ADDRESS>" "<IP_ADDRESS>:9001"

3 envoy have added this configuration
 "upgrade_configs": [
                     {"upgrade_type": "websocket"}
                  ],

@alyssawilk can you comment on this?

I assume the log you have provided is from the "front-envoy". Do you have logs from the other 2 envoy processes?

hi @yanavlasov
In the middle of envoy, if the log level is info, there is no log, if the log level is debug, too many other log interference, only see this log for this request
[source/common/http/http2/codec_impl.cc:535] [C1917] invalid frame: Invalid HTTP header field was received
The sidecar envoy no logs

I suspect you didn't enable connect support
https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/http/websocket#handling-h2-hops
https://github.com/envoyproxy/envoy/blob/master/api/envoy/api/v2/core/protocol.proto#L86
I'll also update the docs to make it more clear this is off by default and needs config to turn on.

@alyssawilk thank you。
According to your prompt, front-envoy has no 503 UR.But the middle envoy still has 503 UR。
My sidecar envoy config like this
{
                "name": "local_service_socketio_cluster",
                "hosts": [
                    {
                        "socket_address": {
                            "port_value": 8080,
                            "address": "<IP_ADDRESS>"
                        }
                    }
                ],

                "http2_protocol_options":{
                    "max_concurrent_streams":2147483647,
                    "hpack_table_size":4096,
		    "allow_connect": true
                },

                "connect_timeout": "2.5s",
                "common_http_protocol_options": {
                    "idle_timeout": "120s"
                },
                "circuit_breakers": {
                    "thresholds":[
                        {
                            "priority":"DEFAULT",
                            "max_connections": 1048576,
                            "max_pending_requests": 1048576,
                            "max_requests": 1048576
                        },
                        {
                            "priority":"HIGH",
                            "max_connections": 1048576,
                            "max_pending_requests": 1048576,
                            "max_requests": 1048576
                        }
                    ]
                },
                "lb_policy": "round_robin",
                "type": "strict_dns"
            },

"route_config": {
                                        "name": "local_route",
                                        "virtual_hosts": [
                                            {
                                                "name": "backend",
                                                "routes": [
                                                    {
                                                        "route": {
                                                            "cluster": "local_service_socketio_cluster",
							    "upgrade_configs": [{"upgrade_type": "websocket"}]
                                                        },
                                                        "match": {
                                                            "prefix": "/xlchannel.app2amlogic/sio/"
                                                        }
                                                    },
                                                    {
                                                        "route": {
                                                            "cluster": "local_service_cluster"
                                                        },
                                                        "match": {
                                                            "prefix": "/xlchannel.app2amlogic/"
                                                        }
                                                    }

When I requested, the middle envoy appeared have same log like this, and there was no log in sidecar envoy
[2019-09-09 08:11:10.912][000032][debug][http] [source/common/http/conn_manager_impl.cc:537] [C197269][S17400316266476258482] request headers complete (end_stream=false):
':authority', '<IP_ADDRESS>'
':path', '/xlchannel.app2amlogic/sio/?EIO=3&transport=websocket'
':method', 'GET'
':scheme', 'http'
'pragma', 'no-cache'
'cache-control', 'no-cache'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36'
'origin', 'chrome-extension://omalebghpgejjiaoknljcfmglgbpocdp'
'sec-websocket-version', '13'
'accept-encoding', 'gzip, deflate'
'accept-language', 'zh-CN,zh;q=0.9,en;q=0.8'
'sec-websocket-key', '8GEYDcI/OWVMqkHXzAFoEA=='
'sec-websocket-extensions', 'permessage-deflate; client_max_window_bits'
'x-forwarded-for', '<IP_ADDRESS>'
'x-forwarded-proto', 'http'
'x-envoy-internal', 'true'
'x-request-id', '4c4f7780-3034-4eb0-aeaa-9e9ee1fc7313'
'x-envoy-expected-rq-timeout-ms', '15000'
'upgrade', 'websocket'
'connection', 'upgrade'

[2019-09-09 08:11:10.912][000032][debug][lua] [source/extensions/filters/common/lua/lua.cc:37] coroutine finished
[2019-09-09 08:11:10.912][000032][debug][router] [source/common/router/router.cc:270] [C197269][S17400316266476258482] cluster 'xlchannel.app2amlogic' match for URL '/xlchannel.app2amlogic/sio/?EIO=3&transport=websocket'
[2019-09-09 08:11:10.912][000032][debug][router] [source/common/router/router.cc:328] [C197269][S17400316266476258482] router decoding headers:
':authority', '<IP_ADDRESS>'
':path', '/xlchannel.app2amlogic/sio/?EIO=3&transport=websocket'
':method', 'GET'
':scheme', 'http'
'pragma', 'no-cache'
'cache-control', 'no-cache'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36'
'origin', 'chrome-extension://omalebghpgejjiaoknljcfmglgbpocdp'
'sec-websocket-version', '13'
'accept-encoding', 'gzip, deflate'
'accept-language', 'zh-CN,zh;q=0.9,en;q=0.8'
'sec-websocket-key', '8GEYDcI/OWVMqkHXzAFoEA=='
'sec-websocket-extensions', 'permessage-deflate; client_max_window_bits'
'x-forwarded-for', '<IP_ADDRESS>,<IP_ADDRESS>'
'x-forwarded-proto', 'http'
'x-request-id', 'c831fe68-48f0-93d6-96b2-057bd0aff6df'
'upgrade', 'websocket'
'connection', 'upgrade'
'content-length', '0'
'x-envoy-external-address', '<IP_ADDRESS>'
'x-envoy-expected-rq-timeout-ms', '30000'

[2019-09-09 08:11:10.912][000032][debug][client] [source/common/http/codec_client.cc:26] [C197270] connecting
[2019-09-09 08:11:10.912][000032][debug][connection] [source/common/network/connection_impl.cc:634] [C197270] connecting to <IP_ADDRESS>:9001
[2019-09-09 08:11:10.912][000032][debug][connection] [source/common/network/connection_impl.cc:643] [C197270] connection in progress
[2019-09-09 08:11:10.912][000032][debug][http2] [source/common/http/http2/codec_impl.cc:721] [C197270] setting stream-level initial window size to 268435456
[2019-09-09 08:11:10.913][000032][debug][http2] [source/common/http/http2/codec_impl.cc:743] [C197270] updating connection-level initial window size to 268435456
[2019-09-09 08:11:10.913][000032][debug][pool] [source/common/http/conn_pool_base.cc:20] queueing request due to no available connections
[2019-09-09 08:11:10.913][000032][debug][connection] [source/common/network/connection_impl.cc:516] [C197270] connected
[2019-09-09 08:11:10.913][000032][debug][client] [source/common/http/codec_client.cc:64] [C197270] connected
[2019-09-09 08:11:10.913][000032][debug][pool] [source/common/http/http2/conn_pool.cc:83] [C197270] creating stream
[2019-09-09 08:11:10.913][000032][debug][router] [source/common/router/router.cc:1023] [C197269][S17400316266476258482] pool ready
[2019-09-09 08:11:10.913][000032][debug][http2] [source/common/http/http2/codec_impl.cc:563] [C197270] stream closed: 1
[2019-09-09 08:11:10.913][000032][debug][client] [source/common/http/codec_client.cc:105] [C197270] request reset
[2019-09-09 08:11:10.913][000032][debug][pool] [source/common/http/http2/conn_pool.cc:222] [C197270] destroying stream: 0 remaining
[2019-09-09 08:11:10.913][000032][debug][router] [source/common/router/router.cc:481] [C197269][S17400316266476258482] upstream reset
[2019-09-09 08:11:10.913][000041][debug][main] [source/server/connection_handler_impl.cc:257] [C197271] new connection
[2019-09-09 08:11:10.913][000032][debug][http] [source/common/http/conn_manager_impl.cc:1188] [C197269][S17400316266476258482] encoding headers via codec (end_stream=false):
':status', '503'
'content-length', '57'
'content-type', 'text/plain'
'date', 'Mon, 09 Sep 2019 08:11:10 GMT'
'server', 'envoy'

[2019-09-09 08:11:10.913][000032][debug][http2] [source/common/http/http2/codec_impl.cc:518] [C197269] sent reset code=0
[2019-09-09 08:11:10.913][000032][debug][http2] [source/common/http/http2/codec_impl.cc:563] [C197269] stream closed: 0

So it looks like you made it to the middle Envoy.  Unfortunately there's not much I can say without seeing logs from the Envoy that is rejecting the request because we don't (yet?) convey any details about the cause of the stream reset.  If you had trace logging on in the side car I'd expect at least logs for the connection being established, and ideally logs from nghttp2 about why it was rejecting the request.

@alyssawilk The sidecar's trace log is
[http2] [source/common/http/http2/codec_impl.cc:535] [C797] invalid frame: Invalid HTTP header field was received
more log like this
[2019-09-10 02:09:13.030][000027][debug][main] [source/server/connection_handler_impl.cc:257] [C5124] new connection
[2019-09-10 02:09:13.030][000027][trace][connection] [source/common/network/connection_impl.cc:439] [C5124] socket event: 2
[2019-09-10 02:09:13.030][000027][trace][connection] [source/common/network/connection_impl.cc:507] [C5124] write ready
[2019-09-10 02:09:13.082][000027][trace][connection] [source/common/network/connection_impl.cc:439] [C797] socket event: 3
[2019-09-10 02:09:13.082][000027][trace][connection] [source/common/network/connection_impl.cc:507] [C797] write ready
[2019-09-10 02:09:13.082][000027][trace][connection] [source/common/network/connection_impl.cc:477] [C797] read ready
[2019-09-10 02:09:13.082][000027][trace][connection] [source/common/network/raw_buffer_socket.cc:21] [C797] read returns: 117
[2019-09-10 02:09:13.082][000027][trace][connection] [source/common/network/raw_buffer_socket.cc:21] [C797] read returns: -1
[2019-09-10 02:09:13.082][000027][trace][connection] [source/common/network/raw_buffer_socket.cc:29] [C797] read error: 11
[2019-09-10 02:09:13.082][000027][trace][http2] [source/common/http/http2/codec_impl.cc:340] [C797] dispatching 117 bytes
[2019-09-10 02:09:13.082][000027][debug][http] [source/common/http/conn_manager_impl.cc:200] [C797] new stream
[2019-09-10 02:09:13.083][000027][debug][http2] [source/common/http/http2/codec_impl.cc:535] [C797] invalid frame: Invalid HTTP header field was received
[2019-09-10 02:09:13.083][000027][trace][http2] [source/common/http/http2/codec_impl.cc:355] [C797] dispatched 117 bytes
[2019-09-10 02:09:13.083][000027][trace][http2] [source/common/http/http2/codec_impl.cc:554] [C797] send data: bytes=13
[2019-09-10 02:09:13.083][000027][trace][connection] [source/common/network/connection_impl.cc:376] [C797] writing 13 bytes, end_stream false
[2019-09-10 02:09:13.083][000027][trace][http2] [source/common/http/http2/codec_impl.cc:508] [C797] sent frame type=3
[2019-09-10 02:09:13.083][000027][debug][http2] [source/common/http/http2/codec_impl.cc:518] [C797] sent reset code=1
[2019-09-10 02:09:13.083][000027][debug][http2] [source/common/http/http2/codec_impl.cc:563] [C797] stream closed: 1
[2019-09-10 02:09:13.083][000027][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-09-10 02:09:13.083][000027][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-09-10 02:09:13.083][000027][trace][connection] [source/common/network/connection_impl.cc:439] [C797] socket event: 2
[2019-09-10 02:09:13.083][000027][trace][connection] [source/common/network/connection_impl.cc:507] [C797] write ready
[2019-09-10 02:09:13.083][000027][trace][connection] [source/common/network/raw_buffer_socket.cc:62] [C797] write returns: 13
[2019-09-10 02:09:13.083][000027][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-09-10 02:09:13.132][000027][trace][connection] [source/common/network/connection_impl.cc:439] [C5124] socket event: 3
[2019-09-10 02:09:13.132][000027][trace][connection] [source/common/network/connection_impl.cc:507] [C5124] write ready
[2019-09-10 02:09:13.132][000027][trace][connection] [source/common/network/connection_impl.cc:477] [C5124] read ready
[2019-09-10 02:09:13.132][000027][trace][connection] [source/common/network/raw_buffer_socket.cc:21] [C5124] read returns: 0
[2019-09-10 02:09:13.132][000027][debug][connection] [source/common/network/connection_impl.cc:501] [C5124] remote close
[2019-09-10 02:09:13.132][000027][debug][connection] [source/common/network/connection_impl.cc:183] [C5124] closing socket: 0
[2019-09-10 02:09:13.132][000027][debug][main] [source/server/connection_handler_impl.cc:68] [C5124] adding to cleanup list
[2019-09-10 02:09:13.132][000027][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-09-10 02:09:13.132][000027][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-09-10 02:09:13.132][000027][trace][connection] [source/common/network/connection_impl.cc:376] [C640] writing 59 bytes, end_stream false
[2019-09-10 02:09:13.132][000027][trace][connection] [source/common/network/connection_impl.cc:376] [C640] writing 55 bytes, end_stream false
[2019-09-10 02:09:13.132][000027][trace][connection] [source/common/network/connection_impl.cc:439] [C640] socket event: 2
[2019-09-10 02:09:13.132][000027][trace][connection] [source/common/network/connection_impl.cc:507] [C640] write ready
[2019-09-10 02:09:13.132][000027][trace][connection] [source/common/network/raw_buffer_socket.cc:62] [C640] write returns: 114
[2019-09-10 02:09:13.190][000027][trace][connection] [source/common/network/connection_impl.cc:439] [C797] socket event: 3
[2019-09-10 02:09:13.190][000027][trace][connection] [source/common/network/connection_impl.cc:507] [C797] write ready
[2019-09-10 02:09:13.190][000027][trace][connection] [source/common/network/connection_impl.cc:477] [C797] read ready
[2019-09-10 02:09:13.190][000027][trace][connection] [source/common/network/raw_buffer_socket.cc:21] [C797] read returns: 117
[2019-09-10 02:09:13.190][000027][trace][connection] [source/common/network/raw_buffer_socket.cc:21] [C797] read returns: -1
[2019-09-10 02:09:13.190][000027][trace][connection] [source/common/network/raw_buffer_socket.cc:29] [C797] read error: 11

@alyssawilk Thank you very much, the problem has been solved。


fp8 is slower than fp16

Hi experts, I tried to use Transformer Engine to do the inference test of llama2 on H800 and found that the speed of fp8 was much slower than fp16, bellow is a small reproduction that only contains Linear. The results show that the speed of fp8.Linear < te.linear < nn.linear, can anyone tell me if this test is reasonable? Thanks.
Docker image: nvcr.io/nvidia/pytorch:23.08-py3
Test code:
import os
import numpy as np
import torch
import transformer_engine.pytorch as te
from time import perf_counter
from torch import nn
from transformer_engine.common import recipe

os.environ['CUDA_VISIBLE_DEVICES'] = "1"

batch_sizes = [1024, 10240, 10240, 10240, 51200]
hidden_sizes = [1024, 1024, 5120, 10240, 10240]
output_sizes = [1024, 1024, 5120, 10240, 10240]

fp8_recipe = recipe.DelayedScaling(margin=0, interval=1, fp8_format=recipe.Format.E4M3)
fp8_recipe.reduce_amax = False
use_te_fp8s = [[False, False], [True, False], [True, True]]

class Model(torch.nn.Module):
    def __init__(self, hidden_size, output_size, use_te):
        super().__init__()
        if not use_te:
            self.linear = nn.Linear(hidden_size, output_size)
        else:
            self.linear = te.Linear(hidden_size, output_size)

    def forward(self, x):
        for i in range(10):
            x = self.linear(x)
        return x

for i in range(len(batch_sizes)):
    for use_te_fp8 in use_te_fp8s:
        use_te = use_te_fp8[0]
        use_fp8 = use_te_fp8[1]
        batch_size = batch_sizes[i]
        hidden_size = hidden_sizes[i]
        output_size = output_sizes[i]
        model = Model(hidden_size, output_size, use_te)
        model.to('cuda')
        input = torch.randn(batch_size, hidden_size).to('cuda')

        times = []
        for j in range(20):
            if j >= 5:
                start = perf_counter()
            with te.fp8_autocast(enabled=use_fp8, fp8_recipe=fp8_recipe):
                out = model(input)
            if j >= 5:
                end = perf_counter()
                times.append(end - start)
        print(f'use_te: {str(use_te):5}, use_fp8: {str(use_fp8):5}, bs: {batch_size:5}, hid_size: {hidden_size:5}, out_size: {output_size:5}, cost_time: {1000*np.mean(times):5f}ms')
    print('=============================================')

Test result:
use_te: False, use_fp8: False, bs:  1024, hid_size:  1024, out_size:  1024, cost_time: 0.231356ms
use_te: True , use_fp8: False, bs:  1024, hid_size:  1024, out_size:  1024, cost_time: 0.874557ms
use_te: True , use_fp8: True , bs:  1024, hid_size:  1024, out_size:  1024, cost_time: 3.224556ms
=============================================
use_te: False, use_fp8: False, bs: 10240, hid_size:  1024, out_size:  1024, cost_time: 0.238261ms
use_te: True , use_fp8: False, bs: 10240, hid_size:  1024, out_size:  1024, cost_time: 0.903459ms
use_te: True , use_fp8: True , bs: 10240, hid_size:  1024, out_size:  1024, cost_time: 3.308409ms
=============================================
use_te: False, use_fp8: False, bs: 10240, hid_size:  5120, out_size:  5120, cost_time: 0.248760ms
use_te: True , use_fp8: False, bs: 10240, hid_size:  5120, out_size:  5120, cost_time: 0.917531ms
use_te: True , use_fp8: True , bs: 10240, hid_size:  5120, out_size:  5120, cost_time: 3.316881ms
=============================================
use_te: False, use_fp8: False, bs: 10240, hid_size: 10240, out_size: 10240, cost_time: 0.256033ms
use_te: True , use_fp8: False, bs: 10240, hid_size: 10240, out_size: 10240, cost_time: 0.922597ms
use_te: True , use_fp8: True , bs: 10240, hid_size: 10240, out_size: 10240, cost_time: 11.672506ms
=============================================
use_te: False, use_fp8: False, bs: 51200, hid_size: 10240, out_size: 10240, cost_time: 0.238444ms
use_te: True , use_fp8: False, bs: 51200, hid_size: 10240, out_size: 10240, cost_time: 0.908310ms
use_te: True , use_fp8: True , bs: 51200, hid_size: 10240, out_size: 10240, cost_time: 52.545521ms

Hi @WangQiangItachi, in order for this test to give you proper results, you need to synchronize with the GPU after computation (since GPU computation is asynchronous with respect to the CPU), by adding torch.cuda.synchronize() after the computation. You can see that this is a problem since the nn.Linear that you got does not increase at all with the increase of the problem size.
Adding synchronization introduces some overheads, so in order to get the accurate timing you should change the way your timing loop works a little bit, by measuring multiple invocations, synchronizing, and then averaging the times. Basically something like this:
        for j in range(5):
            # warmup
            with te.fp8_autocast(enabled=use_fp8, fp8_recipe=fp8_recipe):
                out = model(input)
        torch.cuda.synchronize()
        N = 20
        start = perf_counter()
        for j in range(N):
            with te.fp8_autocast(enabled=use_fp8, fp8_recipe=fp8_recipe):
                out = model(input)
        torch.cuda.synchronize()
        end = perf_counter()
        time = (end - start) / N
        print(f'use_te: {str(use_te):5}, use_fp8: {str(use_fp8):5}, bs: {batch_size:5}, hid_size: {hidden_size:5}, out_size: {output_size:5}, cost_time: {1000*time:5f}ms')

Hi @ptrendx, thanks for your suggestion, I modified the code and re-tested it, the results are as follows. It can be seen that when the linear size reaches a certain level, fp8 will have a greater performance improvement. Does this mean that using fp8 for inference with the LLM model only yields speed benefits when the batch size is relatively large? Because I tested chatglm-6B/llama2-13b on a single H800 card with batch sizes of 8/16/32, and the inference speed with fp8 was still slower than fp16.
use_te: False, use_fp8: False, bs:  1024, hid_size:  1024, out_size:  1024, cost_time: 0.271229ms
use_te: True , use_fp8: False, bs:  1024, hid_size:  1024, out_size:  1024, cost_time: 1.009548ms
use_te: True , use_fp8: True , bs:  1024, hid_size:  1024, out_size:  1024, cost_time: 3.510086ms
=============================================
use_te: False, use_fp8: False, bs: 10240, hid_size:  1024, out_size:  1024, cost_time: 0.729535ms
use_te: True , use_fp8: False, bs: 10240, hid_size:  1024, out_size:  1024, cost_time: 1.056175ms
use_te: True , use_fp8: True , bs: 10240, hid_size:  1024, out_size:  1024, cost_time: 3.574828ms
=============================================
use_te: False, use_fp8: False, bs: 10240, hid_size:  5120, out_size:  5120, cost_time: 15.609458ms
use_te: True , use_fp8: False, bs: 10240, hid_size:  5120, out_size:  5120, cost_time: 15.755959ms
use_te: True , use_fp8: True , bs: 10240, hid_size:  5120, out_size:  5120, cost_time: 5.614840ms
=============================================
use_te: False, use_fp8: False, bs: 10240, hid_size: 10240, out_size: 10240, cost_time: 62.995147ms
use_te: True , use_fp8: False, bs: 10240, hid_size: 10240, out_size: 10240, cost_time: 59.850148ms
use_te: True , use_fp8: True , bs: 10240, hid_size: 10240, out_size: 10240, cost_time: 20.039033ms
=============================================
use_te: False, use_fp8: False, bs: 51200, hid_size: 10240, out_size: 10240, cost_time: 320.128337ms
use_te: True , use_fp8: False, bs: 51200, hid_size: 10240, out_size: 10240, cost_time: 324.473535ms
use_te: True , use_fp8: True , bs: 51200, hid_size: 10240, out_size: 10240, cost_time: 97.020478ms

For inference (and especially for H800, since it has lower memory throughput) you should try the trick with caching FP8 weights. The forward method of TE modules has the option is_first_microbatch. When set with the default value of None, it will recompute the FP8 values of the weights every time. In order to avoid that recomputation, you can set it once to True (this will do the casting to FP8 and cache the results) and then do the actual inference with it set to False (this will use the cached values of the weights without casting again). That should limit the overheads and so boost the performance of the FP8 path.
Another tip for boosting the performance is to use the larger APIs vs just Linear layer - APIs like LayerNormLinear hide the cost of casting activations to FP8 in the LayerNorm, which is a double win - LN is faster and you do not have the overhead of the cast. Not sure if you used those in your full inference test.
Finally, the results you posted are still a little strange to me - I would have expected to see te.Linear and nn.Linear to be ~same speed for non-FP8 cases, whereas there seems to be some overhead in the small cases. Could you post the full benchmark code here? I will test it on my side and see where the problem is.

Hi @ptrendx, I will try the trick with caching FP8 weights and use LayerNormLinear to replace Linear + LayerNorm, as you mentioned, my current tests simply involve replacing nn.Linear/nn.LayerNorm with te.Linear/te.LayerNorm, as shown below.
def replace_layers_recursively(model, child, name=''):
    if isinstance(child, nn.Linear):
        has_bias = child.bias is not None
        te_module = te.Linear(child.in_features, child.out_features, bias=has_bias)
        te_module.weight.data.copy_(child.weight.data)
        if has_bias:
            te_module.bias.data.copy_(child.bias.data)
        setattr(model, name, te_module)
        child = child.cpu()
    elif isinstance(child, nn.LayerNorm):
        te_module = te.LayerNorm(child.normalized_shape[0], eps=child.eps)
        te_module.weight.data.copy_(child.weight.data)
        te_module.bias.data.copy_(child.bias.data)
        setattr(model, name, te_module)
        child = child.cpu()

    for name1, module in child.named_children():
        replace_layers_recursively(child, module, name=name1)

And the full benchmark code is as follows.
import os
import numpy as np
import torch
import transformer_engine.pytorch as te
from time import perf_counter
from torch import nn
from transformer_engine.common import recipe

os.environ['CUDA_VISIBLE_DEVICES'] = "1"

batch_sizes = [1024, 10240, 10240, 10240, 51200]
hidden_sizes = [1024, 1024, 5120, 10240, 10240]
output_sizes = [1024, 1024, 5120, 10240, 10240]

fp8_recipe = recipe.DelayedScaling(margin=0, interval=1, fp8_format=recipe.Format.E4M3)
fp8_recipe.reduce_amax = False
use_te_fp8s = [[False, False], [True, False], [True, True]]

class Model(torch.nn.Module):
    def __init__(self, hidden_size, output_size, use_te):
        super().__init__()
        if not use_te:
            self.linear = nn.Linear(hidden_size, output_size)
        else:
            self.linear = te.Linear(hidden_size, output_size)

    def forward(self, x):
        for i in range(10):
            x = self.linear(x)
        return x

for i in range(len(batch_sizes)):
    for use_te_fp8 in use_te_fp8s:
        use_te = use_te_fp8[0]
        use_fp8 = use_te_fp8[1]
        batch_size = batch_sizes[i]
        hidden_size = hidden_sizes[i]
        output_size = output_sizes[i]
        model = Model(hidden_size, output_size, use_te)
        model.to('cuda')
        input = torch.randn(batch_size, hidden_size).to('cuda')

        for j in range(5):
            # warmup
            with te.fp8_autocast(enabled=use_fp8, fp8_recipe=fp8_recipe):
                out = model(input)
        torch.cuda.synchronize()
        N = 20
        start = perf_counter()
        for j in range(N):
            with te.fp8_autocast(enabled=use_fp8, fp8_recipe=fp8_recipe):
                out = model(input)
        torch.cuda.synchronize()
        end = perf_counter()
        time = (end - start) / N
        print(f'use_te: {str(use_te):5}, use_fp8: {str(use_fp8):5}, bs: {batch_size:5}, hid_size: {hidden_size:5}, out_size: {output_size:5}, cost_time: {1000*time:5f}ms')
    print('=============================================')

Hi @WangQiangItachi , wanted to give you an update. I ran the test on the H100 and the results are similar to what you reported on H800:
use_te: False, use_fp8: False, bs:  1024, hid_size:  1024, out_size:  1024, cost_time: 0.175725ms
use_te: True , use_fp8: False, bs:  1024, hid_size:  1024, out_size:  1024, cost_time: 0.573835ms
use_te: True , use_fp8: True , bs:  1024, hid_size:  1024, out_size:  1024, cost_time: 2.072206ms
=============================================
use_te: False, use_fp8: False, bs: 10240, hid_size:  1024, out_size:  1024, cost_time: 0.716221ms
use_te: True , use_fp8: False, bs: 10240, hid_size:  1024, out_size:  1024, cost_time: 0.666609ms
use_te: True , use_fp8: True , bs: 10240, hid_size:  1024, out_size:  1024, cost_time: 2.116560ms
=============================================
use_te: False, use_fp8: False, bs: 10240, hid_size:  5120, out_size:  5120, cost_time: 16.341905ms
use_te: True , use_fp8: False, bs: 10240, hid_size:  5120, out_size:  5120, cost_time: 16.344693ms
use_te: True , use_fp8: True , bs: 10240, hid_size:  5120, out_size:  5120, cost_time: 5.692444ms
=============================================
use_te: False, use_fp8: False, bs: 10240, hid_size: 10240, out_size: 10240, cost_time: 66.155760ms
use_te: True , use_fp8: False, bs: 10240, hid_size: 10240, out_size: 10240, cost_time: 62.284888ms
use_te: True , use_fp8: True , bs: 10240, hid_size: 10240, out_size: 10240, cost_time: 20.845784ms
=============================================
use_te: False, use_fp8: False, bs: 51200, hid_size: 10240, out_size: 10240, cost_time: 333.154325ms
use_te: True , use_fp8: False, bs: 51200, hid_size: 10240, out_size: 10240, cost_time: 335.041633ms
use_te: True , use_fp8: True , bs: 51200, hid_size: 10240, out_size: 10240, cost_time: 99.745531ms
=============================================

I'm still looking into why the BF16 te.Linear is slower than nn.Linear for the smallest case. The initial investigation shows that the issue is on the CPU side - somehow the TE version for that smallest case has higher CPU overhead than the regular call, the GPU kernels launched have the same speed (and since in both TE and non-TE cases that case is CPU-limited, this corresponds to the slower reported time).

Hi @ptrendx, thanks for your update.

Hi @WangQiangItachi, I'm sorry for the delayed reply. Yes, in general we consider this a bug, although it is unclear how much we can really do to mitigate it. One of the approaches we are considering is making it easier to use CUDA graphs with TE, which should significantly lower the CPU overhead.

Hi @ptrendx, thank you for helping to clarify my doubts.


Bug: Broken Image In README

Describe the bug
There is an image in the Readme file, that has broken due to some reason.
Screenshots

Thanks for opening this issue. We highly appreciate your contribution 🎉
@all-contributors please add @verma-kunal for docs and bug


Can I use the Alexa Voice Service (AVS) Device SDK  for Android 4.4?

IMPORTANT: Before you create an issue, please take a look at our Issue Reporting Guide.
Briefly summarize your issue:

When I look at   Android Quick Start Guide ，I'm notice that the parameter, API_LEVEL  , Supported version: API 23+ or greater is supported.  The API 23  is  correspond to Android 6.0，it's that mean i can't use the SDK for Andorid 4.4 ?    Please tell  me, thanks very much !
What is the expected behavior?

What behavior are you observing?

Provide the steps to reproduce the issue, if applicable:

Tell us about your environment:

What version of the AVS Device SDK are you using?
  <x.y.z>

Tell us what hardware you're using:

[ ] Desktop / Laptop
[ ] Raspberry Pi
[ ] Other - tell us more:

Tell us about your OS (Type & version):

[ ] Linux
[ ] MacOS
[ ] Raspbian Stretch
[ ] Raspbian Jessy
[ ] Other - tell us more:

Hi!
Sorry for long response. The way SDK currently adapted for the Android requires device to run on API 23+. So yes, you will not be able to run it on older OS.

Hi @xuewenxu,
You are more than welcome to try. I don't think we use anything specific introduced on API 23. That said, this can change at any time and we only support / test devices running API 23 and above.

Thank you @celinval,
I'm trying it on for Android 4.4.

It seems that there might be an answer:
https://github.com/alexa/avs-device-sdk/issues/1092#issuecomment-441973926


Delete expired items at LocalFS backend registry

WHY
To avoid increasing registry size
WHAT
Delete old registry files on LocalFS backend registry every 10 seconds. Default expiration period is 5 days.

LGTM :+1:


SVG image update

Upgrade PNGs to SVG for consistency and retina-friendliness

:+1:

Best PR ever.


Enable "Unserializable" JSON Annotations

This no longer is needed by the F764 patch series. I'm hesitant on whether or not this is the right approach. I'm leaving this here for the time being.
This adds a trait Unserializable that indicates two things when mixed into an Annotation:

This indicates that this annotation cannot be serialized to JSON by json4s natively because it contains something "weird" like an anonymous function, e.g., () => RawModule.
By implementing an abstract method, toJsonSerializable, this Annotation can be converted to an AnnotationSeq that can be serialized.
When the JsonProtocol tries to serialize it, it will first convert it to something that is "good" for JSON, e.g., serializing () => RawModule would elaborate the module first.

JSON serialization is modified to ignore any DeletedAnnotations that contain an unserializable annotation.
This includes a test that unserializable things that the user doesn't implement properly except at run time and that an unserializable annotation is converted to what you expect.
Note: I used a type-unsafe approach here for the abstract method toJsonSerializable. I need this to return an Annotation that does not mixin Unserializable. I wasn't aware of an approach that could do that in a method signature. I fell back to having a second, final method _toJsonSerializable call toJsonSerializable and do run-time type checks. This is what is used for automatic conversion during JSON serialization.

Closing this as it was not needed for Stage/Phase.


Create Entity class

I found it cumbersome to make learning-related util functions, e.g., entity_confusion_matrix, entity_level_f1, for each and every application.
To make these functions as a first-class citizen in the package, I think the "Entity" has to be standardized.
I think these are principles and requirements for Entity class:

1-N relationship between Entity and Mention (i.e., one entity -  N mentions).
An instance of Entity can be associated with multiple types of Mention, e.g., SpanMention, CellMention, FigureMention.
An instance of Entity has to be uniquely identifiable.
The way an instance of Entity is identified should be customizable.

Close for now for inactivity.


EXC_BAD_ACCESS when trying to access memoryStorage.config.expiration

I just updated from Kingfisher 5.7.0 to 5.7.1 via Cocoapods
When I try launching my first controller, Kingfisher crashes with this message :

Thread 1: EXC_BAD_ACCESS (code=2, address=0x16ef28c40)

class func getImageCache(name: String) -> ImageCache{

        let cache: ImageCache

        if let storedCache = imageCaches[name] {
            storedCache.backgroundCleanExpiredDiskCache()
            return storedCache
        }

        cache = ImageCache(name: name)
        cache.diskStorage.config.sizeLimit = 0
        cache.diskStorage.config.expiration = StorageExpiration.seconds(60 * 60 * 6)
        cache.memoryStorage.config.expiration = StorageExpiration.seconds(900)
        imageCaches[name] = cache

        return cache
    }

crash occurs at this line : cache.memoryStorage.config.expiration = StorageExpiration.seconds(900) end a po cache.memoryStorage.config displays this :

Config

totalCostLimit : 0
countLimit : 0
▿ expiration : StorageExpiration

seconds : 2.1418258113e-314

cleanInterval : 3.041644003e-314

but po cache.memoryStorage.config.expiration :

error: Execution was interrupted, reason: EXC_BAD_ACCESS (code=1, address=0xb8).

Can you try a full clean and reinstallation? Especially if you are using Kingfisher across of targets. It sounds like you are using an older binary with a new module declaration (and there is no binary stability in Swift yet, which can cause this issue.)

@onevcat I am using another SDK that is using an older version of kingfisher and I am on 5.7.0. It causes the same exception. Is there a way I can remove the older version from that sdk and make it use the newer version that I have under my own pods? Guide please


feat: Add async support

Issue #, if available:
#17
Description of changes:
I've taken @richardsmithsfdc 's branch and merged the intermediate changes from master down into it. We'd love to be able to use the CRT with AWS Powertools for Lambda (Java), and we use this library for our SQS large message offloading.
This PR so far is just a mechanical resolution of the diff between Richard's branch and master to try and get us closer to the point we can merge this in and release. I've not gone through Richard's changes in depth yet - I expect we'll want a full review of all of this.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@ziyanli-amazon and @scottgerring , thank you!
For reference, I updated my draft PR of async payload offloading support in AWS SQS enhanced client library , in case anyone wants to do anything with that one.


Use hashlink

Hashlink is miles ahead of c++ for Haxe. Looks like it takes 1/10th of the time to compile, all into one neat file.

Although it trips up with stuff that C++ doesn't. Guess HxCPP is much more lenient with dynamic values. Leaving this as is for now
Uncaught exception: Can't cast haxe.ds.ObjectMap to haxe.ds.StringMap
Called from $Reflect.callMethod(..HaxeToolkit\haxe\std/hl/_std/Reflect.hx:83)
Called from base.transpiler._Lua.$Lua_Fields_.callInstruction(base/transpiler/Lua.hx:305)
Called from base.transpiler.$Instructions.instr_root(base/transpiler/Lua.hx:18)
Called from $Reflect.callMethod(..HaxeToolkit\haxe\std/hl/_std/Reflect.hx:83)
Called from base.transpiler._Lua.$Lua_Fields_.callInstruction(base/transpiler/Lua.hx:305)
Called from _Main.$Main_Fields_.main(Main.hx:16)
Called from fun$606(?:1)


disconnect people from voice when muting

denying someone speak permissions in a voice channel only makes them unable to speak in voice chat upon joining and does not affect people who already are connected to voice
disconnect is required to properly mute them

Apparently this didn’t close with #417.


onTap function

can you add to ButtonTabBar onTap functionality ?  for example onTap is have in TabBar

Hi!
Do you mean onTap on the Button?

Yes

Hello,
You can already do that using a TabController. Just attach a listener to it like so:
_tabController.addListener(() {
  print(_tabController.index);
});

Thanks a lot man.!


Pending Tasks for version 4

[ ] Select: Define and create Form tests.

[x] #525

[x] #524

[x] #502

[x] #490

[x] #489

[x] #480

[x] #481

[x] #458

[x] #482

[x] #483

[x] #484

[x] #485

[x] Renaming duplicated components:

[x]  DxcNewInputText -> DxcTextInput
[x]  DxcPassword -> DxcPasswordInput
[x]  DxcNumber -> DxcNumberInput
[x]  DxcNewDate -> DxcDateInput
[x]  DxcNewTextarea -> DxcTextarea
[x]  DxcNewSelect -> DxcSelect
[x]  DxcInput -> DxcInputText
[x]  DxcTextarea -> V3DxcTextarea
[x]  DxcSelect -> V3DxcSelect
[x]  Apply new Changes on app, doc, examples… etc.
[x]  In doc site, autocomplete has to be missing and it has to be replaced by the autosuggest examples.
[x]  In previous input text component. Define an example oof autocomple.

[x] #486

[x] #487

[x] #488

[x] Create an inner component for doc site to define a components as deprecated. Tag deprecated. https://xd.adobe.com/view/668dbf0a-70e2-4ac4-b99e-1afba65efa43-33de/
Estilos Tag:

   box-sizing: border-box;
    height: 24px;
    border: 1px solid #C59F07;
    border-radius: 0.5rem;
    padding: 4px 8px;
    font-size: 0.75rem;
    font-weight: 600;
    color: #624F04;
    background-color: #FFFDF5;
<a> sidenav:
a::before {
    display: table;
    content: "";
    margin-bottom: 0.21875rem;
a::after {
    display: table;
    content: "";
    margin-top: 0.21875rem;
Div sidenav:
    display: flex;
    align-items: center;
    gap: 0.5rem;

all completed in version 4 but "Select: Define and create Form tests." which won't be in the release 4.


[FLINK-13170][table-planner] Planner should get table factory from ca…

…talog when creating sink for CatalogTable

What is the purpose of the change
Planner should first try getting table factory from catalog when creating table sinks for CatalogTable.
Brief change log

Make planner use TableSinkFactory returned by Catalog::getTableFactory.

Verifying this change
Existing tests.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? NA

cc @xuefuz @bowenli86 @zjuwangg

can we add some test coverage?

@flinkbot attention @KurtYoung @JingsongLi @twalthr @dawidwys
Can you guys help to take a look?

@godfreyhe please take a look at this

Moved the logic to a util method and updated some tests to make sure the change is covered.

@flinkbot attention @godfreyhe

thanks for this pr @lirui-apache. StreamPlanner in flink-table-planner module should also be updated, just as PlannerBase in flink-table-planner-blink module. and add some tests for also for blink planner?

@godfreyhe thanks for your suggestions. I have updated StreamPlanner  and added a test for blink planner in SinkTest. Please have a look.

Thanks @lirui-apache , overall looks good to me, just reminder that the TableEnvironment#sql has been reverted, and we should still use sqlQuery for select and sqlUpdate for insert into and ddl.

Thanks @danny0405 , I've updated accordingly.
@godfreyhe @bowenli86 please take another look, thanks.

LGTM, +1 to merge

Couldn't reproduce the test failure. Rebase to try again.

I don't think the StandaloneResourceManagerTest test failure is related. It's reported in https://issues.apache.org/jira/browse/FLINK-13242
merging


Default value of initialSize not working?

The doc says:

initialSize is the initial size of cropping area. 1.0 (or null, by default) fits the size of image, which means cropping area extends as much as possible. 0.5 would be the half. This value is also referred when aspectRatio changes via CropController.aspectRatio.

I tried setting initialSize to 1.0 or null. Neither worked. The default crop size doesn't seem to fill the image.

@mgenware
Thank you for your feedback.
It seems initialSize: 1 would fit the image with the aspect ratio of 1, meaning square.
I believe It should fit the entire image unless aspectRatio is given. I'll fix this.

Crop(
  controller: cropController,
  image: imageData,
  onCropped: onCropped,
  initialSize: 0.8,
  interactive: true,
  // aspectRatio: 1,
  withCircleUi: true,
  baseColor: Colors.black,
  cornerDotBuilder: (size, edgeAlignment) =>
    const SizedBox.shrink(),
  progressIndicator: const CircularProgressIndicator(
    color: Colors.white,
  ),
  fixArea: true,
);

In my case, I'm using the same code as above, and as you can see, the size of the CropArea is different when the image is horizontal and vertical. How do I fix this to 90% of the device width?
I would like to satisfy the above request while satisfying the following conditions.
initialSize : 0.8,
interactive: true,
withCircleUi : true,
fixArea: true

Width > Height Image [Horizontal Image]

Height > Width Image [Vertical Image]


Add Grafana dashboard for ETCD Backups

Signed-off-by: Shreyas Rao<EMAIL_ADDRESS>What this PR does / why we need it:
This PR adds a grafana dashboard for etcd-backup-restore metrics, such as snapshots taken, latest revision, defragmentations and restorations.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Release note:

Added grafana dashboard for ETCD backups.

cc @swapnilgm


Validation turns on midway during training instead of epoch end - Progress Bar

Validation is starting before epoch end. Validation starts during the first training epoch.
I've attached a snippet of the command prompt

EDIT: Epoch progress bar includes validation progress. Please close the issue

Exactly, the first bar contains both the count of training and validation data.


irmin-bench: Upgrade progress library version

Let's wait for progress=0.2.1 before merging

Thanks!


kicking off a $destroy from mouseclick may cause vue devtools to block main thread waiting on stringify() to complete

I have a dynamically mounted component which is destroyed when the user clicks a cancel button. It appears that in some cases where vue devtools does some stringify / encode that recurses and takes a very long time, blocking the main thread until it is completed. This is accompanied by an error message in the console along the lines of: app.js:6120 [Vue warn]: Property or method "toJSON" is not defined on the instance but referenced during render. Make sure to declare reactive data properties in the data option. (found in <Root>).
The console error and deep encode code do not occur when the vue devtools tab has not been visited after reloading chrome devtools (I assume it hasnt hooked into vue at that point).
In my case this occurs when using an elementui select component within my manually mounted component. Destroying my manually mounted component in turn destroys the children. Each el-select (and el-option presumably) that is destroyed adds ~125ms to block the main thread.
Here's part of the flame graph. Those encodes go waaaaayyy down and there are much more to the right. You can see the main thread being blocked for close to 5.5s here. Green is vue.esm, blue is backend.js.

I can easily work around the blocking by putting the destroy function call in a set timeout, but I am also a little annoyed by the "toJSON" warnings, so it'd be good to fix them both properly.
Please let me know if you need more detail. Thanks!

In issue #460 the same problem is encountered when using vuex to hold onto a reference to a component. I am doing the same with this dynamically mounted component.

Duplicate of #460
Let's track it there 🙂

@posva sure thing. should I comment on #460 with my findings?


Possibility to bump minor version

Hi!
As you kindly merged the API modification of the format function, do you mind bumping the minor version of the crate ?
Thank you!

I'm going to bump the minor version when I'll be doing a new release - I'm totally aware I merged an incompatible change. I'm also considering to temporary revert this change - depending if I have more incompatible changes at release time.
Is it a problem for you that the repository has still an unchanged version?

Nop, that's more a question of fetching it directly from cargo and dropping the support of a local modified version in our project.

Hmm, now that I looked into this issue more carefully to try to avoid breaking changes I realized Read, Write and Seek traits are implemented for references to respective traits. This means you should be able to pass ReadWriteSeek mutable reference to format_volume function without API change introduced in https://github.com/rafalh/rust-fatfs/pull/24
Have you tested it? Or maybe I'm wrong and you have some usecase when API change is needed?

Hi @rafalh ,
You are right. I totally missed the fact Read/Write/Seek are also implemented for the mutable reference.
Sorry for this @rafalh: I am a bit confused this bad fix has been merged into master :cry:

No problem. I will revert the change today. Have a nice day!


Which installs affects Microsoft 365 Admin Center Channel selection?

Hi, the Microsoft 365 Admin Center mention that change of update channel will: "Your choice will apply to both new and existing installations." but this documentation where Admin Center links does not mention what happen.
If its change only installs made by users or which are not driven by GPO/MECM/MDM. This is what I suspect. But what happen when I change it manually as user? Will it revert?
And what is precedence? Admin Center > Office Config Service > MECM > MDM > GPO or else?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0cd85e3b-e780-457e-9502-856bc6315bca
Version Independent ID: 416e351f-13c5-e951-ea81-1650910a7c25
Content: Overview of update channels for Microsoft 365 Apps - Deploy Office
Content Source: DeployOffice/overview-update-channels.md
Service: o365-proplus-itpro
GitHub Login: @DHB-MSFT
Microsoft Alias: danbrown

@Kazzan -- Thanks for your question.  Yes, the channel setting in the Microsoft 365 admin center only applies to "un-managed" devices.
That is, devices where you aren't using Group Policy or Office cloud policy service to manage Office on the device in some way. For example, to specify an update channel or update location (but actually could be any Office policy setting that's implemented on the device). Also, doesn't apply if you're using Configuration Manager to manage Office updates on the device. (If you've set OfficeMgmtCOM="True" in your configuration.xml file or are using the "Management of Microsoft 365 Apps for enterprise" policy setting in Group Policy.)
FYI, we're in the process of updating our documentation to make clearer the impact of the setting in the Microsoft 365 admin center.
As for precedence, a setting from Office cloud policy service takes precedence over the same setting in Group Policy. That is mentioned in the Office cloud policy service documentation. Group Policy takes precedence over everything else.
We'll double-check on the scenario where the user has manually changed their channel on their device, but fairly certain it will revert back to the setting specified in the Microsoft 365 admin center, since that device would probably still be considered as un-managed.
Hope that helps.

@Kazzan -- Thanks for your question.  Yes, the channel setting in the Microsoft 365 admin center only applies to "un-managed" devices.
That is, devices where you aren't using Group Policy or Office cloud policy service to manage Office on the device in some way. For example, to specify an update channel or update location (but actually could be any Office policy setting that's implemented on the device). Also, doesn't apply if you're using Configuration Manager to manage Office updates on the device. (If you've set OfficeMgmtCOM="True" in your configuration.xml file or are using the "Management of Microsoft 365 Apps for enterprise" policy setting in Group Policy.)
FYI, we're in the process of updating our documentation to make clearer the impact of the setting in the Microsoft 365 admin center.
As for precedence, a setting from Office cloud policy service takes precedence over the same setting in Group Policy. That is mentioned in the Office cloud policy service documentation. Group Policy takes precedence over everything else.
We'll double-check on the scenario where the user has manually changed their channel on their device, but fairly certain it will revert back to the setting specified in the Microsoft 365 admin center, since that device would probably still be considered as un-managed.
Hope that helps.

Thanks for all the bits in answer and such clarification. Good to know because we exactly want to run some pilot groups setup by policies with different channels. And good to clarify the precedence order in one place, because someone can easily ask for manual or ODT config if it also get reverted by Microsoft 365 Admin Center.

Thanks for all the bits in answer and such clarification. Good to know because we exactly want to run some pilot groups setup by policies with different channels. And good to clarify the precedence order in one place, because someone can easily ask for manual or ODT config if it also get reverted by Microsoft 365 Admin Center.

To make it easier for you to submit feedback on articles on learn.microsoft.com, we're transitioning our feedback system from GitHub Issues to a new experience.
As part of the transition, this GitHub Issue will be moved to a private repository. We're moving Issues to another repository so we can continue working on Issues that were open at the time of the transition. When this Issue is moved, you'll no longer be able to access it.
If you want to provide additional information before this Issue is moved, please update this Issue before December 15th, 2023.
With the new experience, you no longer need to sign in to GitHub to enter and submit your feedback. Instead, you can choose directly on each article's page whether the article was helpful. Then you can then choose one or more reasons for your feedback and optionally provide additional context before you select Submit.
Here's what the new experience looks like.
Note: The new experience is being rolled out across learn.microsoft.com in phases. If you don't see the new experience on an article, please check back later.
First, select whether the article was helpful:

Then, choose at least one reason for your feedback and optionally provide additional details about your feedback:

Article was helpful
Article was unhelpful

Finally, select Submit and you're done!