Redpanda 23.2.x and lower won't work with redpanda-5.7.41 chart due to audit_enabled flag

What happened?
User is reporting that they're getting the following error from core redpanda when trying to use the 5.7.41 charts. They're running ~23.2.8~ 23.2.28 and trying to work through two separate upgrade flows, one for redpanda and one for the charts.
ERROR 2024-06-03 18:15:04,707 [shard 0] main - application.cc:389 - Failure during startup: std::invalid_argument (Unknown property audit_enabled)

Best guess is that the audit support check is incorrectly passing audit_enabled=false into the redpanda.yaml for versions lower than 23.3.0.
https://github.com/redpanda-data/helm-charts/blob/49f4661c41f97a275c4579ef9faded84217c2dc9/charts/redpanda/templates/_configmap.tpl#L130-L132
What did you expect to happen?
Chart should not pass audit_enabled: false to redpanda.yaml as that causes redpanda to choke on startup.
How can we reproduce it (as minimally and precisely as possible)?. Please include values file.
Run 23.2.x image of redpanda with 5.7.41 (and possibly earlier).
Anything else we need to know?
No response
Which are the affected charts?
Redpanda
Chart Version(s)
5.7.41
Cloud provider
N/A - Self-hosted kubernetes.
JIRA Link: K8S-249

also @chrisseto

Might also be coming from here.
https://github.com/redpanda-data/helm-charts/blob/49f4661c41f97a275c4579ef9faded84217c2dc9/charts/redpanda/templates/_configmap.tpl#L226
It's not clear though, user pointed out that either of those appear to be inside the 23-3-0 if/then block, so wanted to call it out specifically

Using the following check I can fairly confidently state that helm chart is working as expected.
$ helm repo add redpanda https://charts.redpanda.com
$ helm template redpanda redpanda/redpanda --version=5.7.41 --dry-run=client --kube-version=v1.21.0 --debug --set=image.tag=v23.2.28 | grep audit_enabled
install.go:214: [debug] Original chart version: "5.7.41"
install.go:231: [debug] CHART PATH: /Users/rafalkorepta/Library/Caches/helm/repository/redpanda-5.7.41.tgz

$ helm template redpanda redpanda/redpanda --version=5.8.7 --dry-run=client --kube-version=v1.21.0 --debug --set=image.tag=v23.2.28 | grep audit_enabled
install.go:214: [debug] Original chart version: "5.8.7"
install.go:231: [debug] CHART PATH: /Users/rafalkorepta/Library/Caches/helm/repository/redpanda-5.8.7.tgz

$ # With new image.tag that is `v23.3.2` it is working as expected.
$ helm template redpanda redpanda/redpanda --version=5.8.7 --dry-run=client --kube-version=v1.21.0 --debug --set=image.tag=v23.3.2 | grep audit_enabled
install.go:214: [debug] Original chart version: "5.8.7"
install.go:231: [debug] CHART PATH: /Users/rafalkorepta/Library/Caches/helm/repository/redpanda-5.8.7.tgz

    audit_enabled: false
      audit_enabled: false

Can user, that is templating Redpanda helm chart, specify image.tag version as overwrite of the default value?
P.S.
The audit logging was introduced in https://github.com/redpanda-data/helm-charts/pull/934.
Without image.tag value overwrite it would be enabled starting from 5.7.3 version of the Redpanda helm chart.
$ helm repo add redpanda https://charts.redpanda.com
$ helm template redpanda redpanda/redpanda --version=5.7.2 --dry-run=client --kube-version=v1.21.0 --debug | grep audit_enabled
install.go:214: [debug] Original chart version: "5.7.2"
install.go:231: [debug] CHART PATH: /Users/rafalkorepta/Library/Caches/helm/repository/redpanda-5.7.2.tgz

$ helm template redpanda redpanda/redpanda --version=5.7.3 --dry-run=client --kube-version=v1.21.0 --debug | grep audit_enabled
install.go:214: [debug] Original chart version: "5.7.3"
install.go:231: [debug] CHART PATH: /Users/rafalkorepta/Library/Caches/helm/repository/redpanda-5.7.3.tgz

    audit_enabled: false
      audit_enabled: false

the version check seems to depend on us having the default RP image repo configured. user in this case is working with a local image.repository so the version check is not resolving right because (for some unknown historical reason), we treat only our tagged versions as trusted.

I'm hesitant to just rip out the image check all at once because the blast radius could be quite large. (It'll significantly change the generation for a given input.)
Instead I think we'll have a smoother/safer experience if we add a new field that is explicitly the redpanda version, redpanda compatibility version, or something of that nature. We can have that field fallback to attempting to parse the version from the provided image tag. If that doesn't work, we'll fail the chart generation and ask the user to explicitly provide the version.


Invalid-meta warning?? (Is bower.json required in package.tar.gz?)

This is breaking dependency injection (e.g. Grunt's wiredep task):

Yet, I have both of these present in my bower.json:
{
  "name"        : "scrollreveal",
  "description" : "Easy scroll animations for web and mobile browsers.",
  "homepage"    : "https://scrollrevealjs.org",
  "main"        : ["dist/scrollreveal.js"],
  "authors"     : ["Julian Lloyd"],
  "license"     : "MIT",

  "keywords": [
    "scroll",
    "reveal",
    "css",
    "animation",
    "javascript",
    "mobile"
  ],

  "ignore": [
    "**/.*",
    "node_modules",
    "bower_components",
    "app/bower_components"
  ]
}

I have a .gitattributes file configured to filter what ends up in the release, so my package.tar.gz will only have:

README
LICENSE
dist/scrollreveal.js
dist/scrollreveal.min.js

After a bower install a .bower.json is generated without any of my bower.json contents:
{
  "name": "scrollreveal",
  "homepage": "https://github.com/jlmakes/scrollreveal.js",
  "version": "3.0.1",
  "_release": "3.0.1",
  "_resolution": {
    "type": "version",
    "tag": "v3.0.1",
    "commit": "6fc1af488b9cec5763e998efbbee8864d3892555"
  },
  "_source": "git://github.com/jlmakes/scrollreveal.js.git",
  "_target": "~3.0.1",
  "_originalSource": "scrollreveal",
  "_direct": true
}

I’m starting to think that my package.tar.gz needs to have the bower.json in order to fix this—is this correct? I don’t have a way to validate this without bumping arbitrarily to 3.0.2, so I’m hoping someone can confirm/deny.
Thanks! :bow:

Well found a new feature to implement and bumped to 3.0.2—I included bower.json in the package release file and everything is right as rain.
Cheers!


Allow specifying duration for event drop

Time points are great!  But durations are too.  I'd love to be able to specify a duration and make an event ball that stretches on the timeline to show the endpoint.

I think this is a duplicate of #138. You were the first, yet the other one seems more complete. :)

I agree; #138 is a better description of the request


Main: Fix Dropdown list in CustomDropdown

Fix Dropdown list in CustomDropdown

/azp run

/azp run

@rchauhan18, Rebasing the branch may help in running the pipeline.

/azp run


Handle text_alignment in Milestone

This change addresses issue #75 by shifting text 1/2 it's length left or right when "left"/"right" are passed as text_alignment.

This is a very simple version of aligning Milestone text. I think a better version would be to allow pixels or % as additional arguments like text_alignment="left:100" for 100 pixels to the left or text_alignment="left:50%" for 50% of text length to the left. If this sounds good I can update my pull request to implement these.

I know too little about the implementation of Roadmapper's functionality, and I highly doubt that we have a useful example for unit tests. However, I highly recommend that when you modify a feature, you write a simple unit test that ensures this functionality works as expected. Besides explaining your actual code changes, such tests ensure that your functionality does not regress with future changes.

I agree that in general unit tests are a good idea but in this case I haven't any idea how to make a useful assertion about this feature and don't want to write a meaningless test for the sake of having a test. If you can suggest a way to assert that a text has been offset I am happy to write a test.

``> This is a very simple version of aligning Milestone text. I think a better version would be to allow pixels or % as additional arguments like text_alignment="left:100" for 100 pixels to the left or text_alignment="left:50%" for 50% of text length to the left. If this sounds good I can update my pull request to implement these.
For my own use I wanted the ability to do alignment by percent or pixel so I implemented it. If you wold rather see this code (or this code with additional requested changes) in the pull request please let me know:
text_width, _ = painter.get_text_dimension(text=self.text, font=self.font, font_size=self.font_size)

# Text is already "centered" if we change nothing
# For "left" and "right", move text_x by 1/2 text_width.
if ":" in self.text_alignment:
    alignment, modifier = self.text_alignment.split(":")
else:
    alignment = self.text_alignment
    modifier = None

if modifier is None:
    offset = .5 * text_width
elif "%" in modifier:
    offset = (int(modifier.strip("%")) / 100) * text_width
else:
    offset = int(modifier)

if alignment == "right":
    self.text_x = self.text_x + offset
elif alignment == "left":
    self.text_x = self.text_x - offset

I had a look at your edited class and method. Testing this method is indeed slightly weird as the method does not really have an output but generates a file.
My suggestion is to create an integration test for this. We already have quite generic tests which basically compare the generated roadmaps with how the roadmap should look.
Your task would be to create the code for a roadmap similar to ColourTheme. Then, add the class of your roadmap to the same lists as ColourTheme and ColourThemeExtensive. Please pay attention to giving appropriate names.
After you commited your changes, we can help you to generate the example roadmaps on the GitHub servers. Alternatively, you execute the workflow to generate the roadmaps from your personal account.

Hi,
Sorry for the late response, as I am currently on holiday.
Thanks for the feedback and improvement suggestions. I really appreciate it.
@kajuberdut It is indeed a very simple text alignment implementation. I looked at your suggestion, and I think it is a good idea. However, as @felix-seifert was saying, it would be great if you could write the corresponding test cases. To be honest, I am pretty bad at writing test cases and would definitely need some help.
CS

I'll implement the test case over the coming weekend. Thanks!

Ok, as of today I have re-worked it to allow the options I mentioned above like "left:100" for 100 units left, or "left:75%" for 75% of text width to the left. This change means there is some logic for unit tests so I also added unit tests in: src/tests/test_milstone.py
I also added a src/tests/roadmap_generators/milestone_text_alignment.py to generate a roadmap with examples of the text alignment in milsetones and added it to the same lists that ColourTheme was in.
The unit tests are all passing:

@felix-seifert I will now look into running the workflows or you are welcome to jump in on those. Thank you.

Ok, it took a few iterations but I have both workflows passing in my fork: https://github.com/kajuberdut/roadmapper/actions
@felix-seifert or @csgoh if either of you could kick off the workflows when convenient or let me know if I need to do something else first that would be great. Thanks.

In my update just now I refactored alignment out to it's own class so that these offset by units / percent etc options can be easily used in other items that have alignment (however, percent is a bit of a problem since in other cases it is the percent of the bounding object instead of the percent of the text.)
I have added unit tests for the new class and updated Milestone to use the Alignment class to handle most of the logic for alignment. However, I have failing tests and also need to address @felix-seifert's comment about updating the documentation so this is not ready for review again. Also I have run out of weekend so this might have to wait a week for me to complete it. Thanks for the useful feedback so far.

I had some time so I fixed the bug I had introduced that was failing unit tests and I also added a validation to avoid the impossible case of center + offset. I think this is ready to review and all tests are passing.


Typography characters

Description
Ensure that the fonts we are using support a wide range of charsets. If it doesn't, decide on fallback fonts.
Acceptance Criteria

[ ] Check the charset range of the fonts we chosen and react accordingly

We analysed both fonts,IBM Plex Mono and GT Sectra and they grant support for the following character sets, such as:

Latin characters: English, Spanish, French, Italian, German, Dutch…
Latin script: Czech, Polish, Estonian, Lithuanian…
(non-Latin) Greek/Cyrillic
Arabic
Hebrew
Indic scripts
Chinese
Japanese
Korean
Russian


Start a roadmap

This is very much a suggestion, apart from I think we should aim for v1 to have OME-zarr 0.4 validation. Open questions:

Should v1 include metadata writing?
What else should happen after v1?

At some point in the future, I think it would be nice if the versions of this package (a reference implementation) would align with the versions of the NGFF spec (such that v0.4 would contain the data models for OME-Zarr 0.4, and v0.5 would contain the data models for both OME-Zarr 0.4 and 0.5, etc.), just to avoid confusion with all the different packages and versions. Or do you think this is asking for too much?

I think we have to version our package independently from the OME-Zarr versions. If the versions are locked together, then it's hard for us to issue patch releases or bugfixes.

To add, our plan is to have different namespaces for different versions of the OME-zarr spec, so currently we have a ome_zarr_models.v04 namespace, but we'll add v05, v06 etc. to this in time.

Should v1 include metadata writing?

Yes, and in a simple form we have metadata writing today -- any class that inherits from GroupSpec has a to_zarr(store, path) method that will create an OME Zarr group (and zarr arrays) in the provided storage backend. we could make this fancier, e.g. by only writing group attributes (but still checking that the entire zarr group is valid).
I think a MVP for this repo should be sufficient to easily read an OME-Zarr image from storage, make some changes to that metadata, and save a new OME-Zarr image to a different storage backend with those changes. That should cover the spectrum of basic use cases, I think, and I don't think we are too far from that tbh

I'm sticking my neck out a bit here and have decided to put mutating and reading/writing in v2. In the interests of getting a proof of concept done, I think having something that can read, validate, and provide a nice Python interface to the metadata and data is already a big win, so that's what I've put in the roadmap.
This shouldn't discourage anyone from working on mutation and saving metadata back to disk though - in theory it shouldn't be too hard!

maybe I just need to write up an example? since we have classes that inherit from GroupSpec, and GroupSpec has a to_zarr method that saves zarr group / array metadata to disk, then our classes that inherit from GroupSpec already support writing.

In theory it's not tricky to just write out whatever metadata we have, but in practice I think we have to be quite careful, because we want to avoid writing out invalid metadata, which means we have to carefully forbid things like:

Creating a multiscales that has a path that doesn't exist in the group
Adding more axes than dimensions in a zarr array
... probably many more

we had validation for these things already, did we remove it?

ok we still have it, but it's been moved:
https://github.com/BioImageTools/ome-zarr-models-py/blob/83f2c491735440fd78bcb977c332bf346c85f700/src/ome_zarr_models/v04/_image_old.py#L57
this class already reads and writes valid ome-ngff metadata, with the validation routines you mentioned. Here's an example:
# /// script
# requires-python = ">=3.13"
# dependencies = [
#     "ome-zarr-models",
#     "pydantic-zarr",
#     "rich"
# ]
#
# [tool.uv.sources]
# ome-zarr-models = { git = "https://github.com/BioImageTools/ome-zarr-models-py.git" }
# ///

from pydantic_zarr.v2 import ArraySpec
from ome_zarr_models.v04._image_old import Image
from ome_zarr_models.v04.coordinate_transformations import _build_transforms
from ome_zarr_models.v04.multiscales import Multiscale, Dataset
from ome_zarr_models.v04.axes import Axis
from zarr import MemoryStore
from rich import print

# define some arrays
names = ('s0', 's1')
shapes = ((10,10), (5,5))
dtypes = ("uint8", "uint8")
scales = ((1,1), (2,2))
translates = ((0,0), (0.5, 0.5))

axes = [Axis(name='a', type='space'), Axis(name='b', type='space')]
datasets = [Dataset(path=k, coordinateTransformations=_build_transforms(scale=s, translation=t)) for k, s, t in zip(names, scales, translates)]
multiscale_meta = Multiscale(
    axes=axes,
    datasets=datasets
    )

arrays = {name: ArraySpec(shape=shape, dtype=dtype, chunks=shape) for name, shape, dtype in zip(names, shapes, dtypes)}
group = Image(attributes={'multiscales': [multiscale_meta]}, members=arrays)
# a model of the zarr hierarchy we will create
print(group.model_dump())

# create the actual zarr metadata
store = MemoryStore()
zgroup = group.to_zarr(store, path='test')
print(zgroup.attrs.asdict())

# check that the arrays are correct
assert zgroup['s0'].shape == (10, 10)
assert zgroup['s1'].shape == (5, 5)

# create a brand new Image model, based on the one we created
group2 = Image.from_zarr(zgroup)
assert group2.model_dump() == group.model_dump()

when I run that, I see this:
(ome-zarr-models) ➜  ome-zarr-models-py git:(main) ✗ uv run example.py
Reading inline script metadata from `example.py`
 Updated https://github.com/BioImageTools/ome-zarr-models-py.git (83f2c49)
{
    'zarr_version': 2,
    'attributes': {
        'multiscales': [
            {
                'axes': [{'name': 'a', 'type': 'space', 'unit': None}, {'name': 'b', 'type': 'space', 'unit': None}],
                'datasets': [
                    {
                        'path': 's0',
                        'coordinateTransformations': (
                            {'type': 'scale', 'scale': [1.0, 1.0]},
                            {'type': 'translation', 'translation': [0.0, 0.0]}
                        )
                    },
                    {
                        'path': 's1',
                        'coordinateTransformations': (
                            {'type': 'scale', 'scale': [2.0, 2.0]},
                            {'type': 'translation', 'translation': [0.5, 0.5]}
                        )
                    }
                ],
                'version': None,
                'coordinateTransformations': None,
                'metadata': None,
                'name': None,
                'type': None
            }
        ],
        'omero': None,
        'image_labels': None
    },
    'members': {
        's0': {
            'zarr_version': 2,
            'attributes': {},
            'shape': (10, 10),
            'chunks': (10, 10),
            'dtype': '|u1',
            'fill_value': 0,
            'order': 'C',
            'filters': None,
            'dimension_separator': '/',
            'compressor': None
        },
        's1': {
            'zarr_version': 2,
            'attributes': {},
            'shape': (5, 5),
            'chunks': (5, 5),
            'dtype': '|u1',
            'fill_value': 0,
            'order': 'C',
            'filters': None,
            'dimension_separator': '/',
            'compressor': None
        }
    }
}
{
    'multiscales': [
        {
            'axes': [{'name': 'a', 'type': 'space', 'unit': None}, {'name': 'b', 'type': 'space', 'unit': None}],
            'datasets': [
                {
                    'path': 's0',
                    'coordinateTransformations': (
                        {'type': 'scale', 'scale': [1.0, 1.0]},
                        {'type': 'translation', 'translation': [0.0, 0.0]}
                    )
                },
                {
                    'path': 's1',
                    'coordinateTransformations': (
                        {'type': 'scale', 'scale': [2.0, 2.0]},
                        {'type': 'translation', 'translation': [0.5, 0.5]}
                    )
                }
            ],
            'version': None,
            'coordinateTransformations': None,
            'metadata': None,
            'name': None,
            'type': None
        }
    ],
    'omero': None,
    'image_labels': None
}


Added and documented some more HTTP-Methods

Added and documented some more HTTP-Methods
Just to update the existing ones.

nice !
just need to edit changelog i think.

Do these work on gwt at all?

@intrigus sur ethis is just adding new fields to a static class that had not all HTTP-Methods as constants before, (all checks passed)

There is more to it than just adding a few constants. Please look at the code:
https://github.com/libgdx/libgdx/blob/master/gdx/src/com/badlogic/gdx/net/NetJavaImpl.java#L175

To make it work on gwt, there has to be made some changes for the new methods, if they are supported. Else, it should be documented that they are not supported on gwt
https://github.com/libgdx/libgdx/blob/11a53e2d9d573e2a0a94457b309dbf7044623497/backends/gdx-backends-gwt/src/com/badlogic/gdx/backends/gwt/GwtNet.java#L129-L145

On GWT, we use RequestBuilder, which supports DELETE, GET, HEAD, POST, PUT. Any others should be documented as unavailable on GWT.

Unless GWT can support the other methods (sorry not too familiar?), it probably makes sense to drop this pull request for compatiblity's sake?

@SimonIT
I just extended the constants, but you are free to implement it.
This Line should tell that it's not supported:
https://github.com/libgdx/libgdx/blob/11a53e2d9d573e2a0a94457b309dbf7044623497/backends/gdx-backends-gwt/src/com/badlogic/gdx/backends/gwt/GwtNet.java#L144

@noblemaster Compatibility does not break

@Monarchis If you want this pull request to continue, please (1) make sure those methods work on all backends. Also, (2) given we add a lot of methods, let's also allow custom HTTP methods in addition to the ones you proposed supporting both ones that do output (ditto POST) and those that don't (ditto GET). I think that make the most sense.
Also, don't forget to (3) write JUnit test for your methods and the custom methods testing them on all backends.

@noblemaster The NetGwt is using google lib that does not support PATCH; OPTIONS, TRACE and CONNECT so there will be a problem mapping them to the other 5 existing ones, HEAD is possible to implement properly.

One thing that i can do is marking the one that do not work on GWT as not supported by gtw, and removing the rest that does not work right now.

I see. In that sense it probably makes little sense to add them besides probably HEAD which is of little use if you ask me. libGDX shouldn't offer methods that are not supported on all backends.

@noblemaster
They can be implemented on all Platforms, only gwt has problems.

ok only HEAD is left as change.

There is some duplicate code you added in GwtNet. Also, what about the unit tests for all the platform to make sure it actually works?

@noblemaster the only thing i added was the HEAD-Method, and setting the UriConnection.setDoInput() to false if it's head. (GwtNet).
The redundancies, idk maybe i'll remove them.
But currently I'm still not sure if head also needs an Input (the headers that are responded to the client).
May i really have to test this Method, otherwise, nothing else has changed.


statelessProgrammableSwitchService throws error on null switch value

Describe The Bug:
Homebridge log is filled with errors for all devices with switches "Missing button status from [device name]"
To Reproduce:
I have a range of z-wave and zigbee relays that have switch inputs that i'm not using.
Expected behavior:
No errors in logs
Logs:
[26/06/2023, 17:58:15] [Smartthings Plug (IK)] Button value from Front Garden Uplighter: null
[26/06/2023, 17:58:15] [Smartthings Plug (IK)] Missing button status from Front Garden Uplighter

Plugin Config:
{
    "bridge": {
        "name": "Homebridge 9AD8",
        "username": "0E:A5:61:46:9A:D8",
        "port": 51749,
        "pin": "483-66-557",
        "bind": [
            "Ethernet"
        ]
    },
    "accessories": [
    "platforms": [
        {
            "name": "Config",
            "port": 8581,
            "platform": "config",
            "sessionTimeout": 28800
        },
        {
            "name": "Smartthings Plug (IK)",
            "BaseURL": "https://api.smartthings.com/v1/",
            "AccessToken": ""
            "GarageDoorMaxPoll": 40,
            "PollLocksSeconds": 10,
            "PollDoorsSeconds": 10,
            "PollSensorsSeconds": 5,
            "PollSwitchesAndLightsSeconds": 10,
            "IgnoreDevices": [
                "Dehumidifier",
                "Beas side light",
                "Hatties side light"
            ],
            "platform": "HomeBridgeSmartThings"
        }
    ],
    "disabledPlugins": [
    ]
}

Screenshots:

Environment:

Plugin Version: 1.5.9
Homebridge Version: 1.6.0
Node.js Version: 18.15.0
NPM Version:
Operating System: Windows

I too have this symptom. Would be nice to have a way of suppressing error messages from a ST device.


[Fix]: Fix wrong word.

Fix kwargss to kwargs.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Include Standalone Tesseract for Mac

I recently discovered you could update the dynamic shared libraries of tesseract with otool. All it takes is a simple shell script to change your installed tesseract into a standalone folder. I added it for my own app and thought it might help this project as well.
#!/bin/sh

if ! command -v tesseract &> /dev/null
then
    brew install tesseract
fi

if [ "$EUID" -ne 0 ]
  then echo "Please run as root"
  exit
fi

mkdir tesseract_standalone
cp -r /usr/local/Cellar/tesseract tesseract_standalone/
cp -r /usr/local/Cellar/leptonica tesseract_standalone/
cp -r /usr/local/Cellar/libpng tesseract_standalone/
cp -r /usr/local/Cellar/libtiff tesseract_standalone/
cp -r /usr/local/Cellar/webp tesseract_standalone/
cp -r /usr/local/Cellar/openjpeg tesseract_standalone/
cp -r /usr/local/Cellar/jpeg tesseract_standalone/
cp -r /usr/local/Cellar/giflib tesseract_standalone/

cd tesseract_standalone/tesseract/4.1.1/bin/
install_name_tool -change /usr/local/opt/leptonica/lib/liblept.5.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib tesseract
install_name_tool -change /usr/local/Cellar/tesseract/4.1.1/lib/libtesseract.4.dylib @executable_path/../lib/libtesseract.4.dylib tesseract
install_name_tool -change /usr/local/opt/leptonica/lib/liblept.5.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib ../../../tesseract/4.1.1/lib/libtesseract.4.dylib
install_name_tool -change /usr/local/opt/libpng/lib/libpng16.16.dylib ../../../libpng/1.6.37/lib/libpng16.16.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/jpeg/lib/libjpeg.9.dylib ../../../jpeg/9d/lib/libjpeg.9.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/giflib/lib/libgif.7.dylib ../../../giflib/5.2.1/lib/libgif.7.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/libtiff/lib/libtiff.5.dylib ../../../libtiff/4.2.0/lib/libtiff.5.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/webp/lib/libwebp.7.dylib ../../../webp/1.2.0/lib/libwebp.7.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/openjpeg/lib/libopenjp2.7.dylib ../../../openjpeg/2.4.0/lib/libopenjp<IP_ADDRESS>.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/jpeg/lib/libjpeg.9.dylib ../../../jpeg/9d/lib/libjpeg.9.dylib ../../../libtiff/4.2.0/lib/libtiff.5.dylib

Thanks for this! I'll take a look at it over the weekend hopefully!
On Thu, 8 Apr 2021, 1:56 am Mathew Chan, @.***> wrote:

I recently discovered you could update the dynamic shared libraries of
tesseract with otool. All it takes is a simple shell script to change your
installed tesseract into a standalone folder. I added it for my own app and
thought it might help this project as well.
#!/bin/sh
if ! command -v tesseract &> /dev/nullthen
brew install tesseractfi
if [ "$EUID" -ne 0 ]
then echo "Please run as root"
exitfi
mkdir tesseract_standalone
cp -r /usr/local/Cellar/tesseract tesseract_standalone/
cp -r /usr/local/Cellar/leptonica tesseract_standalone/
cp -r /usr/local/Cellar/libpng tesseract_standalone/
cp -r /usr/local/Cellar/libtiff tesseract_standalone/
cp -r /usr/local/Cellar/webp tesseract_standalone/
cp -r /usr/local/Cellar/openjpeg tesseract_standalone/
cp -r /usr/local/Cellar/jpeg tesseract_standalone/
cp -r /usr/local/Cellar/giflib tesseract_standalone/
cd tesseract_standalone/tesseract/4.1.1/bin/
install_name_tool -change /usr/local/opt/leptonica/lib/liblept.5.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib tesseract
install_name_tool -change /usr/local/Cellar/tesseract/4.1.1/lib/libtesseract.4.dylib @executable_path/../lib/libtesseract.4.dylib tesseract
install_name_tool -change /usr/local/opt/leptonica/lib/liblept.5.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib ../../../tesseract/4.1.1/lib/libtesseract.4.dylib
install_name_tool -change /usr/local/opt/libpng/lib/libpng16.16.dylib ../../../libpng/1.6.37/lib/libpng16.16.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/jpeg/lib/libjpeg.9.dylib ../../../jpeg/9d/lib/libjpeg.9.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/giflib/lib/libgif.7.dylib ../../../giflib/5.2.1/lib/libgif.7.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/libtiff/lib/libtiff.5.dylib ../../../libtiff/4.2.0/lib/libtiff.5.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/webp/lib/libwebp.7.dylib ../../../webp/1.2.0/lib/libwebp.7.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/openjpeg/lib/libopenjp2.7.dylib ../../../openjpeg/2.4.0/lib/libopenjp<IP_ADDRESS>.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/jpeg/lib/libjpeg.9.dylib ../../../jpeg/9d/lib/libjpeg.9.dylib ../../../libtiff/4.2.0/lib/libtiff.5.dylib
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/cfculhane/AnkiOCR/issues/19, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJRYIUYMRLSLH66YNOYDNZDTHR6BLANCNFSM42RCCQHA
.

Great. One more thing, you can directly set a relative path for Tessdata in Tesseract/Pytessteract without exporting TESSDATA_PREFIX.

This has now been committed, thanks for the suggestion, seems to work great. I have split tessdata out to a separate directory to make it easier for people to add their own language files.
https://github.com/cfculhane/AnkiOCR/commit/bacd7dfe388b4cf2b2f9d5cb1b7473d7cbc5ad5a

Closing this for now, please open an issue if it doesn't work going forward :)


How to extract the 'amqp10.auth.basic.password' from a message queue binding

For our integration, we only need the password (or dedicated parts like user/password) from the credentials. Currently we use the user/password from the amqp10.auth.basic - however, we have to manually extract it and store it in our credential store. We now want to switch to the btp operator to mount the password in our kubernetes cluster. The operator is connected to btp service manager, we created a "serviceInstance" and a "serviceBinding" but now fail to extract only the password.
we tried multiple variants: this is a sample of the direction we were looking at:
apiVersion: services.cloud.sap.com/v1
kind: ServiceBinding
metadata:
  name: btp-message-queue-binding
  namespace: spm
spec:
  serviceInstanceName: mqs-avengers-3
  secretName: btp-message-queue-secret
  secretTemplate: |
    apiVersion: v1
    kind: Secret
    metadata:
    stringData:
      PASSWORD:  {{ .credentials.amqp10.auth.basic.password }}

Doing it this way, the operator gives an error: invalid template: [].auth interface is nil (i don't have the exact error message, but it points to auth cannot be dereferenced).
We tried some alternatives like
PASSWORD:  {{ .credentials.amqp10 }}
that returns some rendered string as map[auth:map[basic:map[password: gdsadsaghjgsad
that at least looks like you should be able to navigate into the maps further.
Error:
2024-04-16T11:30:49Z	ERROR	servicebinding-resource	failed to create secret from template	{"error": "the Secret template is invalid: template: spm/btp-message-queue-binding:5:41: executing \"spm/btp-message-queue-binding\" at <.credentials.amqp10.auth.basic>: nil pointer evaluating interface {}.auth", "errorVerbose": "template: spm/btp-message-queue-binding:5:41: executing \"spm/btp-message-queue-binding\" at <.credentials.amqp10.auth.basic>: nil pointer evaluating interface {}.auth\nthe Secret template is invalid\ngithub.com/SAP/sap-btp-service-operator/api/common/utils.CreateSecretFromTemplate\n\t/workspace/api/common/utils/secret_template.go:36\ngithub.com/SAP/sap-btp-service-operator/api/v1.(*ServiceBinding).validateSecretTemplate\n\t/workspace/api/v1/servicebinding_validating_webhook.go:154\ngithub.com/SAP/sap-btp-service-operator/api/v1.(*ServiceBinding).ValidateUpdate\n\t/workspace/api/v1/servicebinding_validating_webhook.go:97\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*validatingHandler).Handle\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/webhook/admission/validator.go:103\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/webhook/admission/webhook.go:169\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/webhook/admission/http.go:98\nsigs.k8s.io/controller-runtime/pkg/webhook/internal/metrics.InstrumentedHook.InstrumentHandlerInFlight.func1\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.17.0/prometheus/promhttp/instrument_server.go:60\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2166\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.17.0/prometheus/promhttp/instrument_server.go:147\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2166\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.17.0/prometheus/promhttp/instrument_server.go:109\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2166\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2683\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3137\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2039\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"} 38 github.com/SAP/sap-btp-service-operator/api/v1.(*ServiceBinding).validateSecretTemplate 37 	/workspace/api/v1/servicebinding_validating_webhook.go:156 36 github.com/SAP/sap-btp-service-operator/api/v1.(*ServiceBinding).ValidateUpdate 35 	/workspace/api/v1/servicebinding_validating_webhook.go:97 34 sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*validatingHandler).Handle 33 	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/webhook/admission/validator.go:103 32 sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle 31 	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/webhook/admission/webhook.go:169 30 sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP 29 	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/webhook/admission/http.go:98 28 sigs.k8s.io/controller-runtime/pkg/webhook/internal/metrics.InstrumentedHook.InstrumentHandlerInFlight.func1 27 	/go/pkg/mod/github.com/prometheus/client_golang@v1.17.0/prometheus/promhttp/instrument_server.go:60 26 net/http.HandlerFunc.ServeHTTP 25 	/usr/local/go/src/net/http/server.go:2166 24 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1 23 	/go/pkg/mod/github.com/prometheus/client_golang@v1.17.0/prometheus/promhttp/instrument_server.go:147 22 net/http.HandlerFunc.ServeHTTP 21 	/usr/local/go/src/net/http/server.go:2166 20 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2 19 	/go/pkg/mod/github.com/prometheus/client_golang@v1.17.0/prometheus/promhttp/instrument_server.go:109 18 net/http.HandlerFunc.ServeHTTP 17 	/usr/local/go/src/net/http/server.go:2166 16 net/http.(*ServeMux).ServeHTTP 15 	/usr/local/go/src/net/http/server.go:2683 14 net/http.serverHandler.ServeHTTP 13 	/usr/local/go/src/net/http/server.go:3137 12 net/http.(*conn).serve 11 	/usr/local/go/src/net/http/server.go:2039
What is the best way to extract only the password?

Hi @TorgeHarbig
please share the credentials json structure of the service you are consuming.
Thanks,
Keren.


test: add NPM_CONFIG_TARBALL check

This commit adds a check to verify that the environment variable
NPM_CONFIG_TARBALL matches the target node version.
It also moves the check of the node version of the base image to the
start of the test to allow for early failure if the version don't match.

LGTM :+1:

@lance Just a friendly reminder about this. Would be good to get this in soonish I think.

@danbev sorry - for some reason I thought this had already been merged. Thanks for the reminder!


[MM-60227] Allow plugins to ask for desktop sources for screen sharing

Summary
PR adds support for more than just Calls to ask for desktop screen-sharing sources. Since CallsWidgetWindow is a singleton, we can easily reuse most of the existing logic without having to make significant changes.
While we are here, I am also making the API call properly throw errors so that plugins can handle any permission issue without requiring them to listen to Calls-specific errors.
Note: we still expect the plugin side to implement a selection modal. I am still considering whether we should simplify this process as well, but if so it would be in another PR.
Example
https://github.com/user-attachments/assets/675048df-81eb-44e1-8df1-dfbced1a3188
Ticket Link
https://mattermost.atlassian.net/browse/MM-60227
Release Note
Added support for plugins to ask for desktop source for screen sharing through the desktopAPI.getDesktopSources call

@streamer45 - Are dev docs updates needed for this PR from your perspective?

@cwarnermm Not at this time. However, ideally, we should have some documentation on how plugins can integrate with our Desktop App, and this sort of functionality would fit in there. That's a larger effort, though, that likely needs to start from the eng side.


Getting The X11 connection broke: I/O error (code 1) After started using V3.5

Hey, I am using splash in kebernetes, started getting The X11 connection broke: I/O error (code 1) after i changed the version from 3.4 to 3.5.
I get the following log when the pod gets restarted.
The X11 connection broke: I/O error (code 1)
XIO:  fatal IO error 22 (Invalid argument) on X server ":755893205"
after 751 requests (747 known processed) with 0 events remaining.
This happens after crawling few pages continuously. there is no much difference in the page crawled, its the same page with different post parameters. The pod gets restarted randomly after few requests [like around 20-30 pages]
Pod setup:
image:  scrapinghub/splash:3.5
commandArgs:
args: ["--slots=10", "--max-timeout=3600", "--maxrss=1200"]
resources:
cpu: "100m"
memory: "1400Mi"
Any help to know what causes this issue and solution for it, will be appreciated!

Any update on this?

I also have this error since 3.5. Did some testing and seems to me that it occurs due to XVFB being killed on container restart. I noticed that it occurs mostly when the maxrss limit is reached and twisted is restarted.

I have also started to experience this issue recently. Restarting the Docker container does fix it, but its not practical unfortunately.

@astam006, with aquarium the container restart is being applied. but as you said, it is pretty impractical. I also have some containers that randomly hang on encountering this issue. The server receives requests and i can see splash downloading stuff. But on rendering the XVFB just stalls consistently. This happens every now and then, and as long as this issue persisted, the container wont actually serve any requests, it will accept it though, but give error as respose.

I tried using 3.4, and I have the same problem described here https://github.com/TeamHG-Memex/aquarium/issues/1 . This time it did not say X11 connection broke, but everything else was the same


use the usual directory for CNI as AWS VPC CNI does not like alternate directories

  Warning  FailedCreatePodSandBox  10m                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "715301634ac099a9643293f42cffa758b4faa9c16600bc800be1d626f9ae610e": plugin type="aws-cni" name="aws-cni" failed (add): failed to find plugin "aws-cni" in path [/home/containerd/opt/cni/bin]

potentially the aws vpc cni can look at containerd configuration and pick up the bin_dir from config dump command and use that to drop the "aws-cni" binary
root@ip-172-31-18-214:/home/ec2-user# /home/containerd/usr/local/bin/containerd config default | grep cni
    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
          cni_conf_dir = ""
          cni_max_conf_num = 0
root@ip-172-31-18-214:/home/ec2-user# /home/containerd/usr/local/bin/containerd config dump | grep cni
    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/home/containerd/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = "/home/containerd/opt/containerd/cluster/gce/cni.template"
          cni_conf_dir = ""
          cni_max_conf_num = 0
          cni_conf_dir = ""
          cni_max_conf_num = 0


Put link paramset name wrong

I agree to the following

[X] I have read the documentation right now
[X] I have read the FAQs right now
[X] I am aware of the latest release notes
[X] I am aware that an integration that was installed via HACS also needs to be updated via HACS
[X] The Backend (CCU/Homegear/...) is working as expected, devices are taught in and are controllable by its own UI.
[X] I am running the latest version of the custom_component (and Home Assistant)

The problem

Should be "put link paramset" instead of put "paramset" for put_link_paramset
What version of HomematicIP (local) has the issue?
latest
What was the last working version of HomematicIP (local)?
No response
What type of installation are you running?
Home Assistant OS
What type of installation are you running for your homematic backend?
RaspberryMatic Standalone
Which version of your homematic backend are you running?
No response
What hardware are you running for your system?
No response
Which config details do you use

[ ] TLS
[ ] callback data (see)

Which interfaces do you use?

[ ] Homematic IP
[ ] Homematic (Bidcos-RF)
[ ] Groups (Heating-Group)
[ ] BidCos-Wired (HM-Wired)

Diagnostics information (no logs here)
No response
Log file extract. Anything in the logs that might be useful for us? The log (Setting/System/Logs -> load full log) is the best source to support trouble shooting!
No response
Additional information
No response

Warum stellst du keinen PR ein?

Erledigt.


Python Debugger closes after only 3 seconds

Issue Type: Bug

open python file
Start debugger on current file
After only 2-3 seconds, the debugger closes (control bar disappears and orange status bar goes back to blue)

VS Code version: Code 1.35.1 (c7d83e57cd18f18026a8162d042843bda1bcf21f, 2019-06-12T14:30:02.622Z)
OS version: Windows_NT x64 10.0.17763

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz (8 x 1800)

GPU Status
2d_canvas: enabledchecker_imaging: disabled_offflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: disabled_softwarerasterization: enabledsurface_synchronization: enabled_onvideo_decode: enabledwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
15.84GB (7.82GB free)

Process Argv

Screen Reader
no

VM
0%

Extensions (3)

Extension
Author (truncated)
Version

anaconda-extension-pack
ms-
1.0.1

python
ms-
2019.5.18875

vscode-yaml
red
0.4.1

Sounds like an issue with an extension.
A plain install of VSCode does not include Python support, so you must have added an extension to give this. On the VSCode Activity Bar click the Extensions icon. Find the installed extension that's giving you your Python features. Look for the "Repository" link in its header. Click that. Go to the Issue section of its GitHub repo and open your ticket there. Then close this issue here, optionally including a reference to the issue you created in the other repo.


Fix # 16475 : Support dashboards field in chart entity

Describe your changes:
Fixes  #16475 #16390

Support dashboards field in chart entity

UI :

support chart in suggestion box with redirect to dashboard page
support charts in glossary asset with relevant data in right side panel
supported playwright test for charts add as glossary assets

Will require indexing  once to see the correct changes

https://github.com/open-metadata/OpenMetadata/assets/66266464/57608830-bcd0-4952-b09e-469bebeffd6d
https://github.com/open-metadata/OpenMetadata/assets/66266464/0cdeadd9-4619-4fe9-a362-59a007b0bb1a

Type of change:

[x] Bug fix
[x] Improvement
[ ] New feature
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation

Checklist:

[x] I have read the CONTRIBUTING document.
[ ] My PR title is Fixes <issue-number>: <short explanation>
[ ] I have commented on my code, particularly in hard-to-understand areas.
[ ] For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

@Sachin-chaurasiya  changes made as per comments !


releases-tab throws error if no releases present

When looking at a repo with no previous releases like e.g. https://github.com/prisma/studio the releases-tab feature breaks:
❌ releases-tab
content.js:2838 TypeError: Cannot read property 'before' of null
    at releases_tab_init (content.js:3624)
    at async features_run (content.js:2833)

(Windows, Chrome Version 76.0.3809.132 (Official Build) (64-bit), extension version 19.9.4)

Or maybe it doesn't... strange.


[WebAPI] TC dev for ImageBitmap

Impacted tests(approved): new 3, update 0, delete 0
Unit test platform: Crosswalk Project for Android 20.50.533.0
Unit test result summary: pass 3, fail 0, block 0
BUG=https://crosswalk-project.org/jira/browse/CTS-1746

@Honry , ptal, thanks~

LGTM, and please also cherry-pick to Crosswalk-20 branch.


Infinite loop if client call ListTunnels with specific name and auto-pagination

Confirmation

[X] My issue isn't already found on the issue tracker.
[X] I have replicated my issue using the latest version of the library and it is still present.

cloudflare-go version
0.63.0
Go environment
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="<hidden>/Library/Caches/go-build"
GOENV="<hidden>/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="<hidden>/.asdf/installs/golang/1.20.1/packages/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="<hidden>/.asdf/installs/golang/1.20.1/packages"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="<hidden>/.asdf/installs/golang/1.20.1/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="<hidden>/.asdf/installs/golang/1.20.1/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.1"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="<hidden>/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/8w/3sgs6h1s55773snm6gqgw8780000gn/T/go-build1902134300=/tmp/go-build -gno-record-gcc-switches -fno-common"

Expected output
Return List of Tunnel.
Actual output
Infinite loop.
Code demonstrating the issue
var context, cancelFn = context.WithCancel(context.Background())
var tunnels, _, err = c.api.ListTunnels(context, "", cloudflare.TunnelListParams{
Name: "name",
})
Steps to reproduce

Call api.ListTunnels() with below parameter:

var tunnels, _, err = c.api.ListTunnels(context, "<UUID>", cloudflare.TunnelListParams{
    Name: "name",
})

above function will never result.

Debugging result
This happen because if we call https://api.cloudflare.com/client/v4/accounts/<AccountID>/cfd_tunnel with ?name=<name>

the api didn't result ResultInfo as picture below:

On this line: ResultInfo always be zero as image below.

Then ResultInfo.Next() src will increase Page to 1
However, on ResultInfo.Done() src, Page must more than 1 and more than TotalPages
This cause infinite loop

References
No response

thanks, this is a known issue at the moment as the backend service doesn't supply the pagination envelope. the internal team will be addressing it.
you can work around it at the moment by supplying the page parameter and checking if the result size is > 0.

I didn't saw on our opened issues. This will help others understand what happen.
Thank you.

that's fine; it was noticed in another repository and i notified the internal team. i've leave this one here for posterity. once this is fixed, everything will work by itself again and you shouldn't need to upgrade (only remove your workaround).

Thanks @jacobbednarz! This is also affecting https://github.com/cloudflare/terraform-provider-cloudflare/pull/2364. I will use a large per_page value for now.

infinite loop is addressed by #1327. the auto pagination will default to a single page until the service team


Added the word "Vagary"

The Pull Request resolves Issue #638
Description:
Added the word "Vagary"

Hey @raymondTheDev
This issue is already resolved. Please contribute in the open issues.
Thanks for contributing.


A "tag *" value in "   type tail" section yields errors in the log when fluent container starts

I am using fluentd  v0.12.32.
I am tailing log files written to /var/lib/docker/containers/*
If I have tag *  in my  <source> section of type tail in order to pass on the entire tag to filter and match sections, there seems to be an error occuring as noted below in the log when I start my fluent container.
If I change the tag to a value other than * (for example tag doug or tag doug.*) no errors occur, everything works as expected.
When I prepend the tag like this tag doug.*  the full tag, prepended with doug is passed on to the <filter> and <match> sections.
From the documentation, I was expecting to be able to specify tag * and just pass along the input tag as is and not have to prepend it with anything.  Should it work this way?
<source>
  type tail
  path /var/lib/docker/containers/*/*-json.log
  tag *
  pos_file /var/lib/docker/containers/d4b.pos
  format json
  time_format %Y-%m-%dT%H:%M:%S
</source>
<filter fluent.*>
  @type record_transformer
  <record>
    message FLUENT:, ${record["message"]}
    loglevel ${tag_parts[1]}
    dlb_container_id ${hostname}
  </record>
</filter>
<filter var.lib.docker.containers.**>
  @type record_transformer
  <record>
    dlb_container_id ${hostname}
    dlb_tag ${tag}
  </record>
</filter>
<match **>
  type ibmlogging
  flush_interval 10s
  buffer_queue_full_action exception
  buffer_queue_limit 256
  buffer_chunk_limit 8m
  buffer_path /var/fluentd/log_buffer
  secrets_path /run/secrets/d4b-logmet-fluentd
  targetHost temp_value
  targetPort 0000
  retry_limit 8
  retry_wait 1s
  max_retry_wait 10s  # default is infinite
</match>

When I start my fluent container, it loads all the plugins, reads the above conf, starts tailing the logs of 4 running containers...then has a probelm.
Any idea what may be wrong with my fluent.conf?
{"log":"2017-03-20 18:37:36 +0000 [info]: fluent/supervisor.rb:471:read_config: reading config file path=\"/fluentd/etc/fluent.conf\"\n","stream":"stdout","time":"2017-03-20T18:37:36.748747383Z"}
{"log":"2017-03-20 18:37:36 +0000 [info]: fluent/supervisor.rb:337:supervise: starting fluentd-0.12.32\n","stream":"stdout","time":"2017-03-20T18:37:36.748850771Z"}
...
<load plugins>
<read fluent.conf>
...
{"log":"2017-03-20 18:37:38 +0000 [info]: plugin/in_tail.rb:567:initialize: following tail of /var/lib/docker/containers/d2dcaef0fda9e642b45e9915f138c939266812178ee13015652c218f18f0e967/d2dcaef0fda9e642b45e9915f138c939266812178ee13015652c218f18f0e967-json.log\n","stream":"stdout","time":"2017-03-20T18:37:38.089572586Z"}
{"log":"2017-03-20 18:37:38 +0000 [info]: plugin/in_tail.rb:567:initialize: following tail of /var/lib/docker/containers/3dc27558e10c45486201b0d3e4ea24336b272c078a15140371bce76e31732444/3dc27558e10c45486201b0d3e4ea24336b272c078a15140371bce76e31732444-json.log\n","stream":"stdout","time":"2017-03-20T18:37:38.089578468Z"}
{"log":"2017-03-20 18:37:38 +0000 [info]: plugin/in_tail.rb:567:initialize: following tail of /var/lib/docker/containers/822d56de1ccdf53476a10d3656ae7740bd309bf5617deb0940a826c8bcebfe02/822d56de1ccdf53476a10d3656ae7740bd309bf5617deb0940a826c8bcebfe02-json.log\n","stream":"stdout","time":"2017-03-20T18:37:38.089584208Z"}
{"log":"2017-03-20 18:37:38 +0000 [info]: plugin/in_tail.rb:567:initialize: following tail of /var/lib/docker/containers/6c9972bbc49d9d56cf09abc10d7cb627e6bb545b220b6a4eba3d4c8f5de05956/6c9972bbc49d9d56cf09abc10d7cb627e6bb545b220b6a4eba3d4c8f5de05956-json.log\n","stream":"stdout","time":"2017-03-20T18:37:38.089590218Z"}
{"log":"2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:616:rescue in on_notify: undefined method `+' for nil:NilClass\n","stream":"stdout","time":"2017-03-20T18:37:39.091939765Z"}
{"log":"  2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:288:in `receive_lines'\n","stream":"stdout","time":"2017-03-20T18:37:39.091967354Z"}
{"log":"  2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:411:in `wrap_receive_lines'\n","stream":"stdout","time":"2017-03-20T18:37:39.09197415Z"}
{"log":"  2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:606:in `on_notify'\n","stream":"stdout","time":"2017-03-20T18:37:39.091980226Z"}
{"log":"  2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:437:in `on_notify'\n","stream":"stdout","time":"2017-03-20T18:37:39.091996285Z"}
{"log":"  2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:522:in `on_timer'\n","stream":"stdout","time":"2017-03-20T18:37:39.092002859Z"}
{"log":"  2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run_once'\n","stream":"stdout","time":"2017-03-20T18:37:39.092008615Z"}
{"log":"  2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run'\n","stream":"stdout","time":"2017-03-20T18:37:39.092014037Z"}
{"log":"  2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:277:in `run'\n","stream":"stdout","time":"2017-03-20T18:37:39.092019617Z"}
{"log":"2017-03-20 18:37:40 +0000 [error]: plugin/in_tail.rb:616:rescue in on_notify: undefined method `+' for nil:NilClass\n","stream":"stdout","time":"2017-03-20T18:37:40.09018477Z"}
{"log":"  2017-03-20 18:37:40 +0000 [error]: plugin/in_tail.rb:522:on_timer: suppressed same stacktrace\n","stream":"stdout","time":"2017-03-20T18:37:40.090212339Z"}
{"log":"2017-03-20 18:37:41 +0000 [error]: plugin/in_tail.rb:616:rescue in on_notify: undefined method `+' for nil:NilClass\n","stream":"stdout","time":"2017-03-20T18:37:41.090561325Z"}
{"log":"  2017-03-20 18:37:41 +0000 [error]: plugin/in_tail.rb:522:on_timer: suppressed same stacktrace\n","stream":"stdout","time":"2017-03-20T18:37:41.091272632Z"}
{"log":"2017-03-20 18:37:42 +0000 [error]: plugin/in_tail.rb:616:rescue in on_notify: undefined method `+' for nil:NilClass\n","stream":"stdout","time":"2017-03-20T18:37:42.091080643Z"}
{"log":"  2017-03-20 18:37:42 +0000 [error]: plugin/in_tail.rb:522:on_timer: suppressed same stacktrace\n","stream":"stdout","time":"2017-03-20T18:37:42.091109684Z"}
{"log":"2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:616:rescue in on_notify: undefined method `+' for nil:NilClass\n","stream":"stdout","time":"2017-03-20T18:37:43.090042188Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:522:on_timer: suppressed same stacktrace\n","stream":"stdout","time":"2017-03-20T18:37:43.090067049Z"}
{"log":"2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:616:rescue in on_notify: undefined method `+' for nil:NilClass\n","stream":"stdout","time":"2017-03-20T18:37:43.097276805Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:538:on_change: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:288:in `receive_lines'\n","stream":"stdout","time":"2017-03-20T18:37:43.097751368Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:538:on_change: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:411:in `wrap_receive_lines'\n","stream":"stdout","time":"2017-03-20T18:37:43.097769381Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:538:on_change: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:606:in `on_notify'\n","stream":"stdout","time":"2017-03-20T18:37:43.097775425Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:538:on_change: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:437:in `on_notify'\n","stream":"stdout","time":"2017-03-20T18:37:43.097781471Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:538:on_change: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:538:in `on_change'\n","stream":"stdout","time":"2017-03-20T18:37:43.097787237Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:538:on_change: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run_once'\n","stream":"stdout","time":"2017-03-20T18:37:43.097805606Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:538:on_change: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run'\n","stream":"stdout","time":"2017-03-20T18:37:43.098485994Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:538:on_change: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:277:in `run'\n","stream":"stdout","time":"2017-03-20T18:37:43.098503068Z"}
{"log":"2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:616:rescue in on_notify: undefined method `+' for nil:NilClass\n","stream":"stdout","time":"2017-03-20T18:37:44.09030848Z"}
{"log":"  2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:288:in `receive_lines'\n","stream":"stdout","time":"2017-03-20T18:37:44.090891622Z"}
{"log":"  2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:411:in `wrap_receive_lines'\n","stream":"stdout","time":"2017-03-20T18:37:44.090908959Z"}
{"log":"  2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:606:in `on_notify'\n","stream":"stdout","time":"2017-03-20T18:37:44.090915223Z"}
{"log":"  2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:437:in `on_notify'\n","stream":"stdout","time":"2017-03-20T18:37:44.090920823Z"}
{"log":"  2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:522:in `on_timer'\n","stream":"stdout","time":"2017-03-20T18:37:44.090926303Z"}
{"log":"  2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run_once'\n","stream":"stdout","time":"2017-03-20T18:37:44.090931927Z"}
{"log":"  2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run'\n","stream":"stdout","time":"2017-03-20T18:37:44.090937427Z"}
{"log":"  2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:277:in `run'\n"
...

Thanks for the report.
@tag_prefix ||= '' is needed here:
https://github.com/fluent/fluentd/blob/85d51905accfce7158470f3e4ab99641a4cfd5e9/lib/fluent/plugin/in_tail.rb#L130


Deep Clean & Re-Structure of Repo

As suggested from the data api doc we need a separation of concerns between different parts of the application.
In general, the repo structure should treat the following implementations independently:

How datasets get defined, loaded, and sanitized - libs/datasets/sources
How datasets get defined as a canonical source for that type of data - libs/datasets/models
How functions and general transformations are defined - libs/functions/transforms
How statistical functions are defined libs/functions/models
How pipelines are defined as steps for functions and dataset steps - libs/pipelines
How the resultant API schema is defined and written to - api
How the pipelines are triggered - cli
How triggerable pipelines are automated - .github/workflows/
Where notebooks & exploratory analyses are kept - analyses

As a result, I propose we use the following structure to more cleanly separate the varying responsibilities needed within this repo.
├── .github/workflows
├── analyses
├── api
├── cli
├── libs
│   ├── datasets
│   │   ├── sources
│   │   └── api
│   └── models
│   │   ├── ensemble
│   │   ├── inference
│   │   └── seir
│   └── pipelines
└── tests
    ├── data
    └── results

Much of this work would require moving the existing repo to match this structure, including moving some functions around to live in their target homes.
EDIT:  Taking in notes and updating the hierarchy above

Couple questions/comments:

functions feels a bit broad do we need this structure?
I'm a bit leery of separating "transforms" away from data sources/data models. I'm just not 100% sure what I expect to live there.
Why are models in functions? If i'm looking at libs/ without knowing anything, seeing datasets and models at the top level feels clearer than models living in functions.
What goes in dataset/models? the word model will always be overloaded in a project that involves actual models :P.

In general I liked the flatter structure a bit more, to me the data structures that will define these matter a bit more than the folder structure, and I don't quite have a handle on that.
It might be helpful for someone to do a quick run through of applying this structure (or other) to the dod "pipeline" as a draft PR to see a bit more concretely how those changes manifest/what feels right.

done


rm PILOT_USE_TARGET_PORT_FOR_GATEWAY_ROUTES

Please provide a description of this PR:
According to the comments, it can be removed in 1.12

@hzxuzhonghu: PR needs rebase.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@hzxuzhonghu: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

integ-ipv6-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-ipv6-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

integ-ipv6-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-ipv6-k8s-tests_istio

unit-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test unit-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

integ-ipv6-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-ipv6-k8s-tests_istio

unit-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test unit-tests_istio

lint_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test lint_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

integ-ipv6-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-ipv6-k8s-tests_istio

unit-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test unit-tests_istio

lint_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test lint_istio

integ-pilot-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-pilot-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

integ-ipv6-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-ipv6-k8s-tests_istio

unit-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test unit-tests_istio

lint_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test lint_istio

integ-pilot-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-pilot-k8s-tests_istio

integ-security-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-security-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

integ-ipv6-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-ipv6-k8s-tests_istio

unit-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test unit-tests_istio

lint_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test lint_istio

integ-pilot-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-pilot-k8s-tests_istio

integ-security-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-security-k8s-tests_istio

integ-telemetry-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-telemetry-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

integ-ipv6-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-ipv6-k8s-tests_istio

unit-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test unit-tests_istio

lint_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test lint_istio

integ-pilot-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-pilot-k8s-tests_istio

integ-security-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-security-k8s-tests_istio

integ-telemetry-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-telemetry-k8s-tests_istio

integ-helm-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-helm-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

integ-ipv6-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-ipv6-k8s-tests_istio

unit-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test unit-tests_istio

lint_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test lint_istio

integ-pilot-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-pilot-k8s-tests_istio

integ-security-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-security-k8s-tests_istio

integ-telemetry-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-telemetry-k8s-tests_istio

integ-helm-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-helm-tests_istio

release-test_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test release-test_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

integ-ipv6-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-ipv6-k8s-tests_istio

unit-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test unit-tests_istio

lint_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test lint_istio

integ-pilot-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-pilot-k8s-tests_istio

integ-security-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-security-k8s-tests_istio

integ-telemetry-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-telemetry-k8s-tests_istio

integ-helm-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-helm-tests_istio

release-test_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test release-test_istio

integ-cni-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-cni-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

integ-ipv6-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-ipv6-k8s-tests_istio

unit-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test unit-tests_istio

lint_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test lint_istio

integ-pilot-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-pilot-k8s-tests_istio

integ-security-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-security-k8s-tests_istio

integ-telemetry-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-telemetry-k8s-tests_istio

integ-helm-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-helm-tests_istio

release-test_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test release-test_istio

integ-cni-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-cni-k8s-tests_istio

analyze-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test analyze-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

integ-ipv6-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-ipv6-k8s-tests_istio

unit-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test unit-tests_istio

lint_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test lint_istio

integ-pilot-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-pilot-k8s-tests_istio

integ-security-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-security-k8s-tests_istio

integ-telemetry-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-telemetry-k8s-tests_istio

integ-helm-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-helm-tests_istio

release-test_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test release-test_istio

integ-cni-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-cni-k8s-tests_istio

analyze-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test analyze-tests_istio

release-notes_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test release-notes_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

integ-ipv6-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-ipv6-k8s-tests_istio

unit-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test unit-tests_istio

lint_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test lint_istio

integ-pilot-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-pilot-k8s-tests_istio

integ-security-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-security-k8s-tests_istio

integ-telemetry-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-telemetry-k8s-tests_istio

integ-helm-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-helm-tests_istio

release-test_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test release-test_istio

integ-cni-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-cni-k8s-tests_istio

analyze-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test analyze-tests_istio

release-notes_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test release-notes_istio

integ-distroless-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-distroless-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

integ-ipv6-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-ipv6-k8s-tests_istio

unit-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test unit-tests_istio

lint_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test lint_istio

integ-pilot-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-pilot-k8s-tests_istio

integ-security-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-security-k8s-tests_istio

integ-telemetry-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-telemetry-k8s-tests_istio

integ-helm-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-helm-tests_istio

release-test_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test release-test_istio

integ-cni-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-cni-k8s-tests_istio

analyze-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test analyze-tests_istio

release-notes_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test release-notes_istio

integ-distroless-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-distroless-k8s-tests_istio

gencheck_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test gencheck_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

integ-ipv6-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-ipv6-k8s-tests_istio

unit-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test unit-tests_istio

lint_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test lint_istio

integ-pilot-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-pilot-k8s-tests_istio

integ-security-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-security-k8s-tests_istio

integ-telemetry-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-telemetry-k8s-tests_istio

integ-helm-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-helm-tests_istio

release-test_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test release-test_istio

integ-cni-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-cni-k8s-tests_istio

analyze-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test analyze-tests_istio

release-notes_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test release-notes_istio

integ-distroless-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-distroless-k8s-tests_istio

gencheck_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test gencheck_istio

integ-pilot-multicluster-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-pilot-multicluster-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

integ-ipv6-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-ipv6-k8s-tests_istio

unit-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test unit-tests_istio

lint_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test lint_istio

integ-pilot-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-pilot-k8s-tests_istio

integ-security-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-security-k8s-tests_istio

integ-telemetry-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-telemetry-k8s-tests_istio

integ-helm-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-helm-tests_istio

release-test_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test release-test_istio

integ-cni-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-cni-k8s-tests_istio

analyze-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test analyze-tests_istio

release-notes_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test release-notes_istio

integ-distroless-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-distroless-k8s-tests_istio

gencheck_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test gencheck_istio

integ-pilot-multicluster-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-pilot-multicluster-tests_istio

integ-telemetry-mc-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-telemetry-mc-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

integ-ipv6-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-ipv6-k8s-tests_istio

unit-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test unit-tests_istio

lint_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test lint_istio

integ-pilot-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-pilot-k8s-tests_istio

integ-security-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-security-k8s-tests_istio

integ-telemetry-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-telemetry-k8s-tests_istio

integ-helm-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-helm-tests_istio

release-test_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test release-test_istio

integ-cni-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-cni-k8s-tests_istio

analyze-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test analyze-tests_istio

integ-distroless-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-distroless-k8s-tests_istio

gencheck_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test gencheck_istio

integ-pilot-multicluster-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-pilot-multicluster-tests_istio

integ-telemetry-mc-k8s-tests_istio
2058de723d655e79cd6a7e60d66101787e7ac257
link
/test integ-telemetry-mc-k8s-tests_istio

release-notes_istio
9bc6f638b4aeb1d344e0405222b5e14fa1efce60
link
/test release-notes_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test release-notes_istio


x/net/http: Re-connect with upgraded HTTP2 connection fails to send Request.body

What version of Go are you using (go version)?

$ go version
go version go1.12.5 darwin/amd64

Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/bfife/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/bfife/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/x7/yphtc2sj4ml2lrflpwdblvrxsmcq9d/T/go-build801671760=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?
This is a follow up of issues #25009 #28206
We are sending requests as follows:
proto.transport = &http.Transport{}
proto.client = &http.Client{Transport: proto.transport}
req, err := http.NewRequest(request.Method, url, bytes.NewBuffer(request.Body))
resp, err := proto.client.Do(req)

What did you expect to see?
Good behaviour
With the above client configuration, I was able to determine the following code was sending out the request using the RoundTrip function noted here:
https://github.com/golang/go/blob/release-branch.go1.12/src/net/http/transport.go#L427
	if t.useRegisteredProtocol(req) {
		altProto, _ := t.altProto.Load().(map[string]RoundTripper)
		if altRT := altProto[scheme]; altRT != nil {
			if resp, err := altRT.RoundTrip(req); err != ErrSkipAltProtocol {
				return resp, err
			}
		}
	}

In our case, for the first and following requests t.useRegisteredProtocol(req) returns true, so altRT.RoundTrip(req) tries to send via h2_bundle's RoundTrip.  As no connection exists at this point in time, this will fail due to error "http2: no cached connection was available", bubbling up ErrSkipAltProtocol "net/http: skip alternate protocol", and the code continues on.  n.b. subsequent requests will be sent via altRT.RoundTrip(req) if there are cached connections available, however a GoAway received from the server will trigger the same behaviour of advancing to the next section of code.
https://github.com/golang/go/blob/release-branch.go1.12/src/net/http/transport.go#L447
	for {
		select {
		case <-ctx.Done():
			req.closeBody()
			return nil, ctx.Err()
		default:
		}

		// treq gets modified by roundTrip, so we need to recreate for each retry.
		treq := &transportRequest{Request: req, trace: trace}
		cm, err := t.connectMethodForRequest(treq)
		if err != nil {
			req.closeBody()
			return nil, err
		}

		// Get the cached or newly-created connection to either the
		// host (for http or https), the http proxy, or the http proxy
		// pre-CONNECTed to https server. In any case, we'll be ready
		// to send it requests.
		pconn, err := t.getConn(treq, cm)
		if err != nil {
			t.setReqCanceler(req, nil)
			req.closeBody()
			return nil, err
		}

		var resp *Response
		if pconn.alt != nil {
			// HTTP/2 path.
			t.decHostConnCount(cm.key()) // don't count cached http2 conns toward conns per host
			t.setReqCanceler(req, nil)   // not cancelable with CancelRequest
			resp, err = pconn.alt.RoundTrip(req)
		} else {
			resp, err = pconn.roundTrip(treq)
		}
		if err == nil {
			return resp, nil
		}
...

In the above for loop, for the first request without any connections available, a new transport connection is created, and pconn.alt.RoundTrip(req) or pconn.RoundTrip(req) is then used to submit and fulfill the Request, and this likely completes successfully.  So far, so good.
As noted above, subsequent requests from our client continue to be sent from altRT.RoundTrip(req), at which point they are returned, without reaching the for loop.
Bad behaviour
After some time, our load balancer may respond with a GoAway response after a full request is transmitted, including data from the Request.Body, as seen with the following debug:
2019/06/04 02:22:26.750150 http2: Framer 0xc000470a80: wrote HEADERS flags=END_HEADERS stream=17 len=12
2019/06/04 02:22:26.750321 http2: Framer 0xc000470a80: wrote DATA stream=17 len=125 data="STRINGOFDATA"
2019/06/04 02:22:26.750386 http2: Framer 0xc000470a80: wrote DATA flags=END_STREAM stream=17 len=0 data=""
2019/06/04 02:22:26.753144 http2: Framer 0xc000470a80: read GOAWAY len=8 LastStreamID=15 ErrCode=NO_ERROR Debug=""
2019/06/04 02:22:26.753217 http2: Transport readFrame error on conn 0xc0001a3e00: (*errors.errorString) EOF
2019/06/04 02:22:26.753291 http2: Transport failed to get client conn for server:443: http2: no cached connection was available

This results in another transport being created in the for loop shown above, however the Request.Body is not being reset before the request is sent on the newly created transport connection.
The rewind code at https://github.com/golang/go/blob/release-branch.go1.12/src/net/http/transport.go#L497 doesn't get executed if pconn.alt.RoundTrip(req) or pconn.RoundTrip(req) are successful in sending a request after a connection is re-established.  From my limited testing, I think the only time the rewind is reached is when TLSNextProto is manually set as it is in TestTransportBodyAltRewind, because this skips setting up the altProto map in transport.go/onceSetNextProtoDefaults(), as altRT is nil.
It looks like moving the rewind code from L497 to L455 would satisfy the case where an existing connection receives a GoAway frame and the Body had been sent to the server (as described above), however it would also execute in the case of the initial request where a transport does not exist and the request hasn't been sent (which makes that execution not necessary).
I believe this test case demonstrates the issue described above:
func TestTransportBodyGoAwayReceived(t *testing.T) {
        var err error
        var resp *Response
        req, _ := NewRequest("POST", "https://example.org/", bytes.NewBufferString("request"))
        tr := &Transport{DisableKeepAlives: true }
        tr.RegisterProtocol("https", httpsProto{})
        c := &Client{Transport: tr}
        resp, err = c.Do(req)
        if err != nil {
                t.Error(err)
        }
        if resp != nil {
                defer resp.Body.Close()
                ioutil.ReadAll(resp.Body)
        }
}

type httpsProto struct{}

func (httpsProto) RoundTrip(req *Request) (*Response, error) {
        defer req.Body.Close()
        ioutil.ReadAll(req.Body)
        return nil, ErrSkipAltProtocol
}

output
../bin/go test -v -run=TestTransportBodyGoAwayReceived net/http
=== RUN   TestTransportBodyGoAway
2019/06/04 21:36:24 Error enabling Transport HTTP/2 support: protocol https already registered
--- FAIL: TestTransportBodyGoAway (0.42s)
    transport_internal_test.go:274: Post https://example.org/: http: ContentLength=7 with Body length 0
FAIL
FAIL	net/http	0.495s

Moving the rewind code to the top of the for loop as described above results in success:
./bin/go test -v -run=TestTransportBodyGoAwayReceived net/http
=== RUN   TestTransportBodyGoAway
2019/06/04 21:35:54 Error enabling Transport HTTP/2 support: protocol https already registered
--- PASS: TestTransportBodyGoAway (0.56s)
PASS
ok  	net/http	0.642s

As I am not in a position to sign the CLA and submit a fix, it would be good to get some maintainers to have a look at this issue.

/cc @bradfitz @FiloSottile

Sorry to bump, but is anyone able to have a look at this ? @bradfitz @FiloSottile

Thanks for opening this issue @brianfife.  Also, thanks for linking to #25009 and #28206, as they appear to be related (even though they were marked as closed as of Go 1.12).  I'm on 1.12.7 and I'm seeing this issue with clients connecting to the API I work on.
You did a good job of showing the issue, in the actual transport code, but I figured I'd add some of my client debug logs to the issue for additional context.
As you described, I always see this issue manifest after the server sends a GOAWAY frame after a request body has been sent. The internal retry logic in http2shouldRetryRequest in h2_bundle.go calls .GetBody() on the request in order to "recreate" the body to send on a new TCP connection, but the bytes never make it onto the wire:
...
2019/07/18 12:51:41 http2: Framer 0xc00027a000: read GOAWAY len=8 LastStreamID=1999 ErrCode=NO_ERROR Debug=""
--
2019/07/18 12:51:41 http2: Transport received GOAWAY len=8 LastStreamID=1999 ErrCode=NO_ERROR Debug=""
2019/07/18 12:51:41 http2: Transport failed to get client conn for api.foobar.com:443: http2: no cached connection was available
...
2019/07/18 12:51:41 http2: Transport creating client conn 0xc000508900 to <IP_ADDRESS>:443
2019/07/18 12:51:41 http2: Transport encoding header "content-type" = "application/x-www-form-urlencoded"
2019/07/18 12:51:41 http2: Transport encoding header "content-length" = "51"
2019/07/18 12:51:41 http2: Transport encoding header "accept-encoding" = "gzip"
2019/07/18 12:51:41 http2: Framer 0xc000409500: wrote HEADERS flags=END_HEADERS stream=1 len=135
2019/07/18 12:51:41 http2: Framer 0xc000409500: wrote DATA flags=END_STREAM stream=1 len=0 data=""

In the above, we can clearly see that the request contained a 51 byte, form-encoded body but the H/2 framer never sent it, setting the END_STREAM flag after sending 0 bytes of data.
We typically see this happen when the client is using a moderate amount of concurrency, since the chances are increased that the periodic GOAWAY frames from the server will arrive after a request body has been sent.
Interestingly, the Go bug manifests when the transport is left to choose H/2 on it's own via the connection upgrade codepath (aka TLSNextProto).
So, either of these options will result in a client that does not repro this bug:
// Force H/2 explicitly
client := &http.Client{
    Transport: &http2.Transport{},
}

// OR Force HTTP/1.1 explicitly
client := http.Client{
    Transport: &http.Transport{
        TLSNextProto: make(map[string]func(string, *tls.Conn) http.RoundTripper),
    },
}

// OR, run the repro code with GODEBUG="http2client=0" to disable H/2

To echo @brianfife, I'm also sorry to bump this, but I'm wondering if there's any movement or thoughts on how to move forward with this particular issue? @bradfitz @FiloSottile

To echo @brianfife, I'm also sorry to bump this, but I'm wondering if there's any movement or thoughts on how to move forward with this particular issue? @bradfitz @FiloSottile

Correct, body is not re-winded after altRT.RoundTrip here:
https://github.com/golang/go/blob/cf06b9aa81004a017e5c98422fce2fafd1f24921/src/net/http/transport.go#L427-L434
I've missed that during #25009 fix because the trigger condition is less frequent and was not covered by my go-only test-case.
This condition triggers when http2Transport.RoundTripOpt is failing to get a connection during retry (e.g. on attempt to retry after GOAWAY when all other connections in pool are terminated), which results to http2noCachedConnError that is transformed into ErrSkipAltProtocol in http2noDialH2RoundTripper.RoundTrip.
If we change to something like that, everything should be OK:
	if t.useRegisteredProtocol(req) {
		altProto, _ := t.altProto.Load().(map[string]RoundTripper)
		if altRT := altProto[scheme]; altRT != nil {
			if resp, err := altRT.RoundTrip(req); err != ErrSkipAltProtocol {
				return resp, err
			} else if req.GetBody != nil {
				// Rewind the body after ErrSkipAltProtocol, because it can
				// be already used.
				newReq := *req
				var err error
				newReq.Body, err = req.GetBody()
				if err != nil {
					return nil, err
				}
				req = &newReq
			}
		}
	}

I've tested this change with my repro and got zero "400 Bad Request" errors, but I'm not sure that it is correct solution, probably we can refactor something to make this rewind more explicit.
/cc @bradfitz

the above looks reasonable to me, @ernado do you want to open a PR?

@ernado @bcmills any updates on this?

@brianfife, I left a comment on @ernado's CL 210123 indicating what I think is the way forward: namely, we need to add an optional method that a RoundTripper can implement in order to decline a request without reading its Body.

/cc
Could we fix this issue before next release? As an important feature(GOAWAY) in Kubernetes API Server rely on that.

No ETA from me for CL 210123, so please submit other CL if anybody wants, just check that commends from my CL are addressed.
Also current workaround is to use http2-only client.

See also https://golang.org/cl/234358 from #39086

Thanks, let me see if there is anything I can do to fix this issue ASAP.

This issue was incorrectly titled (x/net/http instead of net/http) and therefore incorrectly milestoned (Unreleased) and left out of Go 1.15 release planning. I've retiteld and remilestoned. Since this bug is affecting an important existing use case (Kubernetes), I've also marked it a release blocker. I also sent a fix.
The code change is fairly straightforward, but net/http itself is quite subtle, so it would be best to get the change into beta1 to get the most possible testing. Therefore I did not add the "okay-after-beta1" label.

Thanks for correcting the title - glad it's finally getting some traction!

This issue was incorrectly titled (x/net/http instead of net/http) and therefore incorrectly milestoned (Unreleased) and left out of Go 1.15 release planning. I've retiteld and remilestoned. Since this bug is affecting an important existing use case (Kubernetes), I've also marked it a release blocker. I also sent a fix.
The code change is fairly straightforward, but net/http itself is quite subtle, so it would be best to get the change into beta1 to get the most possible testing. Therefore I did not add the "okay-after-beta1" label.

Is it possible make this bugfix backport to old version like v1.14.x or v1.13.x ? Since kubernetes is still using go1.13 to build itself.

Thanks @rsc for your PR, which is very similar with my idea to fix this issue.

Indeed, @answer1991, thanks for that CL as a starting point.

@gopherbot please backport, as Kubernetes is affected.