direct users to metpy for Tdew to RH conversions

Is your feature request related to a problem? Please describe.
In case weather file doesn't have RH or Pwat, I would like to be able to use Tdew to calculate spectral mismatch
Describe the solution you'd like
This capability is already implemented in MetPy relative_humidity_from_dewpoint
Describe alternatives you've considered
reimplement in pvlib? import metpy and call it? ignore b/c most weather files already include RH or Pwat, or uncertainty in chain of conversions too much?
Additional context
I thought this topic came up somewhere else but I couldn't find it in google group, discussions, or issues/pr's

I would prefer to duplicate very minor models/functions in pvlib rather than expand the list of dependencies.

I find using metpy annoying due to the requirement of specifying units; thus I agree that it would be nice to have these simple functions in pvlib.

I agree with Anton that these models and implementations are simple enough that it's better to just copy (with attribution/license) than to introduce a non-trivial dependency. I also agree with Adam that the metpy units requirement is annoying.
The dewpoint to relative humidity calculation is essentially exact for this purpose so I don't have any concerns about introducing additional uncertainty.
If we agree that we should add this function then I might have someone that would be interested in contributing a PR.

I suggest looking up the equation in a textbook and writing a new function referencing that.  I did this years ago using the meteonorm documentation as reference.

A coworker shared this reference: http://www.bom.gov.au/climate/averages/climatology/relhum/calc-rh.pdf, which is a calculation of the ratio of vapor to saturation vapor pressures based on ambient and dew point temperatures

Another reference that may be useful for saturation vapor pressures.  PlantPredict uses the AEKR coefficients over water for the magnus form equations.
https://www.osti.gov/servlets/purl/548871-PjpxAP/webviewable/

I'm currently working on this. There are three different options. MetPy uses Bolton1980, @kurt-rhee mentions PlantPredict uses the AEKR, @mikofski pointed us to the document that uses Abbot 1985. It's unclear to me whether it is preferable to choose one approach over another. Any thoughts?

https://github.com/Unidata/MetPy/issues/508 also relevant. I suspect all of the above is fine for PV modeling applications.

I'm currently working on this. There are three different options. MetPy uses Bolton1980, @kurt-rhee mentions PlantPredict uses the AEKR, @mikofski pointed us to the document that uses Abbot 1985. It's unclear to me whether it is preferable to choose one approach over another. Any thoughts?

Personally I'd love to see all three of them. If they have similar input arguments they could all be bundled in one function that has a model argument, like this:
temp_dew_from_relative_humidity(relative_humidity, model='aekr')

I would prefer to see just one good one rather than a range of options since this is peripheral to PV modelling.

@uakkoseo
I don't have a strong preference for either implementation, I believe that the model accuracy in either case must be quite rough.  If it helps, here is the C# implementation of the equation mentioned above:
/*
 * These magnus coefficients below are the AEKR coefficients which reduce overall errors the most between -40 and 50 degrees C according to research by NOAA:
 * https://www.osti.gov/servlets/purl/548871-PjpxAP/webviewable/
 *
 * magnusA variable is not explicity needed since the value is cancelled out by a division operation, but the variable is kept here
 * for the sake of keeping the complete equation.
 *
 */
double magnusA = 6.1094;
double magnusB = 17.625;
double magnusC = 243.04;

// e:  amount of water vapor in the air
var e = magnusA * Math.Exp((magnusB * Temperature) / (magnusC + Temperature));

// es:  amount of water vapr in the air if the air were entirely saturated at the same temperature and pressure
var es = magnusA * Math.Exp((magnusB * dewpoint) / (magnusC + dewpoint));

return 100 * (es / e);

Thanks @kurt-rhee !  I think it is easy enough to bundle them all in one function and give the option to choose the model, unless one is sufficient or deemed better than others. From what I understand, all three of these models are used often, the uncertainties seem to be similar for different temperature ranges. The AEKR model has the largest range for temperature out of the three. If used in the stated temperature range Bolton and Abbot seem to be as applicable. (I understand as the issue @wholmgren linked also points to this as well)

More models = more problems. I agree with @adriesse that it's not worth it in this case. My inclination is to copy what MetPy has already implemented.

I don't have a strong stance to not use MetPy but for PV modelling, AEKR might be the better of the two as it seems to work better above 35C (noting worse below -50C compared to MetPy).

One approach could be to provide one reasonably validated model obeying a well defined interface with a mechanism that allows users to plug in their own model.

Copying in the error plot from that MetPy issue...

If I understand correctly, @uakkoseo and @kurt-rhee are proposing to implement the model in blue. Let's just do that and move on with life!

proposing to implement the model in blue.

I have a lot of trouble figuring out from this discussion what's what.  Is the blue line AEKR?  The MetPy discussion links to a WMO report from 1966.
I would be content with a formulation that is currently used by some entity like WMO, ECMWF, NOAA, or even Meteonorm.  If those entities use different formulations, then feel free to choose randomly from among them!

Let's just do that and move on with life!

@adriesse thank you, I tried to find some guidance from the entities you listed above. I was able to find NOAA formula following this page to be using Tetens formula

(MetPy uses Bolton, a modified version of Tetens with values from the more recent formula of Wexler)  and Meteonorm has different coefficients as well using DWD 1979 (different than what is used in MetPy or AEKR equation). I hope the different formulations listed here can mean AEKR equation is okay to proceed with.

I hope the different formulations listed here can mean AEKR equation is okay to proceed with.

The AEKR equation was published 27 years ago but not adopted by major institutions.  Should pvlib?

@kurt-rhee honestly I was a bit stuck on how to proceed on this! If you have a better grasp on it than me please contribute as I wasn't sure if there was a final agreement on the formula. I won't be able to get back to this earlier than next month or so otherwise.

Adding comments from the PR to this thread for continuity sake:
Reasoning
There are some occasions where dew point temperature is available, but relative humidity is not.
Relative humidity can be converted to precipitable water in pvlib via guemard94.
Precipitable water is used by first solar to calculate spectral correction factors.
This function would allow a user with dew point temperature to convert it to relative humidity.

Caveats
Caveat 1
In the issue thread

https://github.com/pvlib/pvlib-python/issues/1744 there is some controversy about the simpleness of this approach. Some users would prefer a more well known solution.
I would argue that because this approach was used in the creation of the First Solar spectral correction function, that it has historical significance and should therefore be included.
If any user should desire, they could add an alternative function and then create a controller function which allows the user to choose different relative humidity calculation methods, which is a pattern than exists already inside pvlib.

Caveat 2
I placed these functions inside of spectrum, but I could also see them living inside of pvlib.atmosphere.py where the gueymard94 function exists. I'd like the maintainers opinion before proceeding to update the docs and what's new file.

@kurt-rhee Thanks for reactivating this issue.
I don't think there is any controversy about simpleness, nor is this a discussion about user preference.  The content of pvlib needs to stand on firm foundations.
So, I did some research.
Calculating RH from the ratio of two saturation vapor pressures, $e_s$ is fine.
There are many different equations for $e_s$ and actually there is one already in pvlib--it's embedded in gueymard94_pw!  The most common equation form for $e_s$ is the Magnus aka Tetens form.  It is recommended by the WMO Handbook and used among others by: Meteonorm, DWD, Lufft, MetPy, NOAA, and of course PlantPredict.  It seems appropriate for pvlib despite the fact that more accurate formulas exist.
The only question remaining is which set of three coefficients to use because there are many sets to choose from.  Superficially they all look very similar, but due to the exponential function in the equation, small differences could become significant.
The coefficients shown in recent editions of the WMO Guide to Meteorological Instruments and Methods of Observation  are: (6.112, 17.62, 243.12).   The guide states that "These formulae are convenient for computation and sufficiently accurate for all normal meteorological applications" in the range -45 to 60 C. These values were first published in Sonntag 1990, but I can only find Sonntag 1994.
Based on my research, I recommend using these WMO coefficients to create a generic tdew_to_rh function for pvlib.  The saturation vapor pressure equation could be a separate function also, and optionally, an inverse function rh_to_tdew could be created at the same time.
The question about historical significance could be assessed by further investigation (@kurt-rhee?):

Was Tdew used in publications by First Solar, or were RH data available?
Is the model spectral factor affected substantially by the choice of Magnus equation coefficients--in particular AEKR vs. WMO?

Ah I totally misunderstood the original asks in the conversation, I'm happy to change the default coefficients to the ones shown above and then change the formulation to the function to allow for users to change the input coefficients as they desire.
As for the last two questions:

My understanding is that First Solar tested with both Tdew and RH to ensure that either data set would give similar results.
Given how rough the model is for converting Tdew to RH, I think that many (reasonable) coefficient sets would give similar results.  I did a bit of testing in my PlantPredict days and the input coefficient set didn't make a huge difference in the resulting energy model.

I'll create a new commit with the desired changes this week!

+1 to @adriesse's plan; basing our function on the WMO guide seems like the way to go to me.  Anton, thank you for figuring out a good path forward here and breaking up the logjam!


Revive base path

Current implementation drops the base file path. For example, if you specify launchable optimize minitest tests, the command outputs models/open_class_user_test.rb without test directory.  However, minitest needs full path such as test/models/open_class_user_test.rb to test run.  That is why I deleted the line removing a base path component. Instead of the line, I added file path normalization with`sourc option.

Thanks for discovering this subtle and important problem, this is indeed broken. We need to be careful what these strings represent, indeed. But this change now breaks Gradle side, so we need to come up with something else.
Does Python have a type that represents a path name, ala java.io.File or java.nio.file.Path? If so, using that would tighten up the abstraction.
Several competing requirements for this scan method:

The base parameter is assumed to be resolvable path (or else glob.iglob won't work). We will state this.
Test runners like Mini Test, needing to receive resolvable paths. For those, the existing behaviour of silently dropping the base would be rather surprising.
Test runners like Gradle & Bazel, which manipulates the obtained paths into something else such as class name and labels. They need to first drop the base portion from the enumerated paths, then do the string manipulation to the rest. Ideally they need enumerated paths relative to base, and if that's difficult, at least they need a guarantee that the enumerated path contains base as the prefix. Here, os.path.normpath would break that.

I think one way to do this is to modify path_builder to take two arguments, base and an enumerated path resolvable with base (IOW, t[len(base)+1:]) and expect that method to combine them back into one resolvable path, which shall be the default behavior, or build a custom TestPath. The only down side I can think of is that the caller who just wants to do filtering now at least needs to do the path combination, though so far we haven't come across such a caller yet.
Perhaps another way is to split those two into two separate methods. Maybe that is less confusing.
Oh, and if the intent of relpath is to capture test paths that are portable, then we need the reverse operation in the formatter side or else the path names the command eventually produce will not be resolvable.
And finally, in my mind this subtle problem and nuances in fixes really underscore the need to write tests to ensure these key invariants are maintained going forward.

We decided to go clam everything into path_builder


Writing Excel fails to preserve template

I basically use the standard example to write to a file using a template, however the template is not being preserved, and the cells do not observe the text formatting (See the picture).
In the picture, the top is the template, and the first 2 rows contains static information, a header if you will.
However, it's completely missing in the output file, as well as the formatting & lines.
Is there a setting or something I'm missing?

I'm having the same issue.
Sadly the template file is strictly required for my purpose and it can't be changed.
I need to modify the data in template only.
Did you ever find an solution?

I switched to NPOI, then perfection!


Problem: BatchTxs does not sort by most profitable batch

Need to fix this grpc query to return the most profitable batch first or support pagination

fix by https://github.com/crypto-org-chain/gravity-bridge/commit/4495f330b4c49210dd4d2fd0aa9a9e6b2f30f0a2


Update to Cadence v1.0.0-preview.12

Description
Automatically update to:

onflow/cadence v1.0.0-preview.12
onflow/flow-go-sdk v1.0.0-preview.10
onflow/flow-go 25d0a9f6e292

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 51.09%. Comparing base (a925945) to head (7166e0a).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #608      +/-   ##
==========================================
+ Coverage   51.06%   51.09%   +0.03%
==========================================
  Files          32       32
  Lines        4189     4192       +3
==========================================
+ Hits         2139     2142       +3
  Misses       1881     1881
  Partials      169      169

Flag
Coverage Δ

unittests
51.09% <100.00%> (+0.03%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


ROOT installation issue using conda

Issue:
trying to install root on conda with following command:
conda create -c conda-forge --name myenv root
Following error:
PackagesNotFoundError: The following packages are not available from current channels:

root

Current channels:

https://conda.anaconda.org/conda-forge/win-64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/win-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/win-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/msys2/win-64
https://repo.anaconda.com/pkgs/msys2/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

Environment (conda list):

$ conda list
EnvironmentLocationNotFound: Not a conda environment: C:\Users\Lenovo\anaconda3\envs\myenv

Details about  conda and system ( conda info ):

$ conda info

 active environment : myenv
    active env location : C:\Users\Lenovo\anaconda3\envs\myenv
            shell level : 2
       user config file : C:\Users\Lenovo\.condarc
 populated config files : C:\Users\Lenovo\.condarc
          conda version : 4.8.4
    conda-build version : 3.18.11
         python version : 3.7.7.final.0
       virtual packages :
       base environment : C:\Users\Lenovo\anaconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\Lenovo\anaconda3\pkgs
                          C:\Users\Lenovo\.conda\pkgs
                          C:\Users\Lenovo\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\Lenovo\anaconda3\envs
                          C:\Users\Lenovo\.conda\envs
                          C:\Users\Lenovo\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.8.4 requests/2.24.0 CPython/3.7.7 Windows/10 Windows/10.0.18362
          administrator : False
             netrc file : None
           offline mode : False

Unfortunately ROOT doesn't support 64-bit Windows builds so I can't provide a conda package for it natively.
I've heard that the Linux build works very well inside Windows Subsystem for Linux so that is probably your best option.


Eth1 green link light inconsistent at boot

I've noticed a weird issue with the green link light on the 0.6.36-0.6.37 x64 bionic builds (first i've used of these builds).
Everything seems to work fine, but it seems to be a random coin toss whether or not the green link light will stay on past boot.
When you plug it in, they both light up, then as the os starts to load they go off and the lights blink a few times, then half the time the green one wont come back, just the blinking amber light.
Everything works perfectly regardless, so this seems very minor. I'm just curious if anyone else has noticed this, or if it somehow could be my hardware (which would seem odd since it lights up at initial boot, but i won't disregard that possibility).
For thoroughness i've tried different cables and switches (tries both the external switch and the 4 port built into the router).

Well. It is random, as the leds are software controlled, and the pinctrl for that is not configured :)

Ah, or do you talk about gigabit ethernet?

Just the led on the actual rock64, not the gigabit light on the switch.
Haha good, thought it might of been my hardware, i could care less if the light is on or off as long as it did not indicate a potential hardware issue :P

I saw a similar behaviour on 0.5.x releases. It's been a while mine is lit but I saw this randomness a while back on 3 of my boards.

And we won't mention that the other green light is most definitely NOT software controlled, and it being inconsistent would be a sign of a major power issue since it is hard-wired to the VCC-SYS voltage regulator output... :-D


How to setup meta titles and descriptions on blog posts?

Description
I'd like to setup meta titles and descriptions for SEO on blog posts. I'm using Netlify and the files generated are .md files.
I set them up in normal pages fine, but when using the same format in .md files, they don't work.
Any idea how?

You can add meta info to your post template
  metaInfo() {
    return {
      title: this.$page.post.title,
      meta: [
        { name: "description", content: this.$page.post.excerpt }
     ]
   }
}

You can check out my blog for more > https://github.com/danvega/danvega-dev/blob/master/src/templates/Post.vue

Awesome! Thanks @danvega

Hey @danvega, I followed your blog to customize the og:image meta tag using cover in graphQL to get the post-compiled image path.
But it's not working for me, the GraphQL is returning ‘undefined’.
Could you take a peep, maybe you'll see what's wrong. https://github.com/arjunkalburgi/writing/blob/vue/src/templates/Post.vue

@arjunchib You have to have a property in your markdown frontmatter called cover. I looked at one your posts and I didn't see it there. Are you adding this property to your posts?
---
slug: "vue3-ref-vs-reactive"
title: "Vue 3 Composition API: Ref vs Reactive"
date: "2020-02-12T16:27:46.472Z"
published: false
excerpt: "One question I keep hearing over and over is what method should I use to declare reactive data in the Vue 3 Composition API? In this article, I explain what ref() and reactive() are and try to answer that question."
author: "Dan Vega"
tags:
  - "vue"
cover: "./vue3-ref-vs-reactive-cover.png"
video: https://www.youtube.com/embed/cRwG96MOHp0
---

@danvega yeah I’m only using it on specific posts, I didn’t do it to all of them. For example here
---
title: Life Story Update 2020
description: >-
  Over the last 5 years I've been learning strong UX skills to build web interfaces and products that connect and enable people. I want to look back on this journey, thinking about where I've been and some of the specific memories I have.
date: '2020-03-23T14:20:33.815Z'
slug: /life-story-update-2020
cover: ./arjun2020.jpg
---

I’ve also changed my getCoverImage function:
getCoverImage() {
        let path = "";
        const cover = this.$page.post.cover;
        if (cover != null && typeof(cover) != 'string') {
          console.log('cover: ', cover);
          path = `${this.getBaseUrl}${cover.src}`;
        } else if (cover != null && cover.includes('http')) {
          path = cover;
        } else {
          path = `${this.getBaseUrl}/writing/sra.jpg?${cover}`;
        }
        return path;
      }

For me the cover.src returns as either undefined or some weird long path. I added the ?${cover} on the default path to test this
http://debug.iframely.com/?uri=https%3A%2F%2Fwww.arjunkalburgi.com%2Fwriting%2Flife-story-update-2020%2F
This is showing that cover is /home/runner/work/writing/writing/content/posts/Life%20Story%202020%20Update/arjun2020.jpg
When It should be something like https://www.arjunkalburgi.com/writing/assets/static/arjun2020.e6856d4.13bcd529ed3aac3554657e40c24e5ddc.jpg

@danvega do I have to use it in more of my posts for it to work? I have no idea why graphql would just quit on me for that one property.


Some improvements for v156

As title.

Other improvement could be using map hash for email/user combinations


Allow setting user-agent header for api methods.

Allow setting user-agent header for api methods.
Now if one need to call an API method with his own user agent, he may
just use user_agent keyword argument for method call.
Resolves: #342

I've found that my problems can be solved another way. Sorry for making a pull request.

No worries. Always happy to take a look.
On Wed, Feb 1, 2017, 2:11 AM Andrew<EMAIL_ADDRESS>wrote:

I've found that my problems can be solved another way. Sorry for making a
pull request.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/google/google-api-python-client/pull/344#issuecomment-276619678,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPUc8iJsjDNbKfKdN8L5OJtsjIzn84mks5rYFpBgaJpZM4Lyk-T
.


Changes layout of a BPSwimlane according to user feedback

Changes the layout of

to

It consists of changing the color scheme, the default label, the title, several new alignments, making form morphs smaller and a line between values

Nice! This looks much better!
Joana Be. @.***> schrieb am Sa., 10. Juli 2021, 17:55:

Merged #62
https://github.com/hpi-swa-lab/babylonian-programming-smalltalk/pull/62
into liveprog21.
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
https://github.com/hpi-swa-lab/babylonian-programming-smalltalk/pull/62#event-5003360501,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEI3YHSXX6SJFWX656VZHLTXBUONANCNFSM5AELFCVQ
.


Is it possible to add the function of direct download? I don't want to confirm the direct download when downloading

I want to add the ability to block websites, some websites don't support downloading files using multi-threaded download tools

I think current mode is very good~~

Hi there, Thank you for your suggestion. However, the title does not accurately reflect the description, which has made it difficult for me to properly address the issues.

If I understand you correctly the issue mentioned in the title is already covered by #89 .

and the issue mentioned in the description is also covered by #241 .

If I have misunderstood your issue, please reopen it with an updated title and description. so we can discuss about it further.


Update configure-server-endpoints.md

updated the required of having server licenses in the tenant
As some users are getting confused as to where to assign them

Issue#https://github.com/MicrosoftDocs/microsoft-365-docs/issues/6081

@JohanFreelancer9 Please copy review

@JohanFreelancer9


Pre-defined 'destination' array

I am getting a empty destination box, but source is filled. I have confirmed both, selected and remaining, have data in them.
<dual-list [sort]="default" [filter]="default"
    [source]="remaining" [key]="key" [display]="display" height="265px"
    [(destination)]="selected"></dual-list>

export class AdvancedCustomerConfigComponent {
    customerId: number;
    selected: Array<any>;
    remaining: Array<any>;
    key: string = "Id";
    display: string = "Description";
    default: boolean = true;

    ngOnInit() {
        if (this.customerId != null) {
            this.getTestUsers();
        }
    }

    getTestUsers() {
        this._customersService.getCustomerConfigOptions(this.customerId)
            .subscribe(
                data => {
                    this.selected = data.Result.SelectedOptions;
                    this.remaining = data.Result.RemainingOptions;
                }
            )
    }

export class ConfigurationOption {
    Id: number;
    Name: string;
    Description: string;
}

Sorry it didnt play with my code inserts... :( first time poster.

The source is a list of all available options, not just remaining. In order for the list to display, the selected options must be part of the source.
If you cannot provide a master list of all possible options for your source, then you could do something like this:
    getTestUsers() {
        this._customersService.getCustomerConfigOptions(this.customerId)
            .subscribe(
                data => {
                    this.selected = data.Result.SelectedOptions;
                    this.remaining = data.Result.RemainingOptions.concat(data.Result.SelectedOptions);
                }
            )
    }

But, it'd be in the spirit of the dual-list component to provide a master list of options rather than a remaining list. The dual-list component figures out what is remaining based on the master list and what has been selected.


Switch current view to OnionSkin view, button on UI

I tought, it would make things faster to have 1  toggle button on the onion skin UI , that directly switches the current viewport between Vp2, and OnionSkin render, so while animating i can simply use 1 UI button to take care of it.

implemented.
Note this only works when toggling from viewport 2.0
It also saves the viewport you toggled.
So in case you have two modelling views, where one is your controller less camera view and one is the one you work in, it will toggle the one you set to onionSkinRenderer last
This breaks of course when having multiple windows with onion skin renderer, but then it breaks anyway, because that's not supported


[BUG] requestBody with text/plain gets wrapped in quotes when sent to server

Describe the bug
When specifying a request body as just a plain string, the request that is sent to the server is wrapped with double-quotes rather than the raw string being sent.
To Reproduce
Steps to reproduce the behavior:

Use the spec below
Hit a simple server endpoint that just dumps the request body (mine is in rust, so not that helpful)
Observe the data being wrapped in quotes.

---
openapi: 3.0.0
info:
  title: Service
  version: '1.0.0'
paths:
  '/v1/parse':
    post:
      operationId: ParsePipeline
      requestBody:
        description: Pipeline syntax
        content:
          text/plain:
            schema:
              type: string
            examples:
              success:
                summary: 'Legit'
                value: 'hello world'

      responses:
        '200':
          description: Successfully parsed

Expected behavior
The raw string would be what I expected for sure :smile_cat:
Environment (please complete the following information):

OS:  Linux
Python version: 3.8.5
Schemathesis version: 2.7.1

Additional context
It looks like the use of is_multlpart in the Case class is the culprit here, since a plain string is viewed as "not" multipart and as such passed to requests as a json payload, rather than a data keyword argument.
I'm not sure what the right solution here is however. Monkey-patching such that a str body type is sent to requests as data definitely solves the problem but I'm sure that will have unintended consequences for other data types.
FWIW the swagger editor takes the above spec and sends the raw hello world string along to the server as I would expect

Hi @rtyler !
Thank you for the detailed report! :)

It looks like the use of is_multlpart in the Case class is the culprit here, since a plain string is viewed as "not" multipart and as such passed to requests as a json payload, rather than a data keyword argument.

Currently we handle only application/json, multipart/form-data and application/x-www-form-urlencoded :( There are long-term plans to generate data for more media types (#690 and #697 ) and allow users to register their own strategies. However, it might be better to explicitly fail on an unknown media type (or skip it and keep only explicit examples) rather than handling it like it is now (there is probably an else branch somewhere)
I will change the code, so plain text data is not encoded as JSON :)

Hm, probably the most straightforward way to send generated payload would be like this (from the highest priority):

Send forms if they are defined
then check if the tested operation expects application/json (and variations) and use the  json keyword
use the data keyword and add the expected 'content-type' header value

I.e., if Schemathesis generated some payload, then it knows how to do it, and it should be ok to send it this way. However, I'll check for more corner cases anyway

Now it should work with the "latest" Docker image :) I still need to add more changes before a stable release, but it should already be possible to test operations with the text/plain media type.
Let me know if it doesn't cover your use case (it should - I added a couple of test cases for it).
Sorry for the delay!


Expose clear method for packed arrays

Expose clear() method so that packed arrays are consistent with regular arrays.

Thanks!


chore: make weave exit 0 on SIGTERM

I'm working on running the core integration tests as Kubernetes Jobs, and I need all the support containers to exit 0 when they receive a SIGTERM -- otherwise, Kubernetes interprets the shutdown as a failure and cancels the rest of the tests. Since SIGTERM isn't mentioned anywhere else in the repo, I'm guessing there's no graceful shutdown behavior to preserve -- if so, then it should be safe to add this handler to immediately exit 0 when SIGTERM is sent.

Im nervous about intercepting sigterm generally, this could mess things up for any users who import weave.
Let's make this signal handler only installed conditionally

Ah, yeah, that makes sense. I'll put it behind an environment variable


Digital twin Registry secret for development needed

Hello,
I would like to receive a digital twin registry secret as instructed here and here to setup my development environment.

Thanks for your comment.


Confusing results for small text

I was trying  5W1H for smaller texts (2-3 lines). The results sometimes

don't show why/whats/hows/whens/wheres (whs) that could have been drawn easily by a human.
eg. what could have been - taxes, financial year,bill
when - year, last month

whs are empty when there could have been atleast something
eg. why - can be not revised

whs hard to interpret.
eg. see hows below (to be recalculated fits but it is accompanied by noisy text)

I am attaching a test case I tried. I understand this is not a bug/technical glitch and may have something to with the logic of the code. Even so, I'd like to know why I am getting such results. Is there something on my part that I could improve/tune? Does the text show a significant improvement on increase in the length of the text? Is the code designed specifically for a certain size or type of text?
{
"dId": "bdeb60d977fa2f95e871198ca96204d7928da3b35b40288011b288f9",
"title": "taxes on bill not revised after new financial year. taxes applicable for last month to be recalculated.",
"text": "",
"description": "",
"category": "billing",
"filename": "5.txt",
"fiveWoneH": {
"who": {
"extracted": [],
"label": "who"
},
"what": {
"extracted": [],
"label": "what"
},
"where": {
"extracted": [],
"label": "where"
},
"when": {
"extracted": [],
"label": "when"
},
"why": {
"extracted": [],
"label": "why"
},
"how": {
.................
"text": "after new financial year ."
"text": "new financial year ."
"text": "taxes applicable for last month to be recalculated .",
"text": "new financial",
...................
}

We had given up on this (otherwise excellent) library because we also faced the same problem with smaller texts. It would be excellent if the author of the library could improve it or guide us on parameters that we could play to make it work with smaller text.
Thanks for posting this issue.

In general, the methods are optimized for larger texts, i.e., will perform better on such. This is mostly because longer texts contain more "hints" what the main actors and actions, including their causes, places, times, etc. are. However, from the exemplary sentence you provided, Giveme5W1H should be capable of extracting the actor and action. I'll have a look into that.

fyi https://github.com/stanfordnlp/CoreNLP/issues/791#issuecomment-442242964


Add docs for IDE0110.

Corresponding docs PR for https://github.com/dotnet/roslyn/pull/49141.

@CyrusNajmabadi thanks a lot for the PR! I would suggest waiting for @gewarren's review before merging.

@CyrusNajmabadi thanks a lot for the PR! I would suggest waiting for @gewarren's review before merging.

Will do, thanks.


[move-compiler] Improve CFG traversal

Use reverse post-order traversal for source language CFG
Will allow for warnings in the future on type-system checks that require fix-point iteration

Motivation

We want to change the rules of the bytecode verifier to not require fix-point iteration for its rules
This refactor will allow for these sorts of constraints to exist for the source language too

Test Plan

internal only change
ran the tests

/land


[ZEPPELIN-1679] Invalid thrift implementation (developer critical)

What is this PR for?
Recently, zeppelin added some amazing features.
https://github.com/apache/zeppelin/pull/1613
You can easily access spark ui from the zeppelin web.
However, in the implementation of the function, the Thrift interface seems to have arbitrarily modified the automatically generated code.
So if you recreate the Thift interface by calling genthrift.sh.
Zeppelin is not built.
I modified this to get the same result with auto-generated code.
What type of PR is it?
Bug Fix
Todos

[x] add event META_INFOS in RemoteInterpreterService.thrift
[x] import generate code

What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1679
How should this be tested?

First you need to install thift 0.9.2.
Run zeppelin-interpreter / src / main / thrift / generate_thrift.sh.
Build full zeppelin.
After build, please check the interpreter spark setting in our web ui.
Try using the spark ui button.

If everything is running normally, it is a success.
Problems occur in the current master 3.
Questions:

Does the licenses files need update? no
Is there breaking changes for older versions? no
Does this needs documentation? no

The reason why travis failed is irrelevant to my pr.
09:44:43,974  INFO org.apache.zeppelin.ZeppelinITUtils:33 - Caller: org.apache.zeppelin.integration.ZeppelinIT.testAngularDisplay(ZeppelinIT.java:187)
09:44:44,074  INFO org.apache.zeppelin.ZeppelinITUtils:41 - Finished.
09:44:44,074  INFO org.apache.zeppelin.integration.ZeppelinIT:189 - testCreateNotebook Test executed
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 126.934 sec - in org.apache.zeppelin.integration.ZeppelinIT

Results :

Tests in error:
  ParagraphActionsIT.testEditOnDoubleClick:443 » ElementNotVisible Element is no...

Tests run: 17, Failures: 0, Errors: 1, Skipped: 0

[INFO]
[INFO] --- scalatest-maven-plugin:1.0:test (test) @ zeppelin-server ---
Discovery starting.
Nov 17, 2016 9:44:46 AM org.openqa.selenium.safari.SafariDriverServer start
INFO: Server started on port 10555
Nov 17, 2016 9:44:46 AM org.openqa.selenium.safari.SafariDriverCommandExecutor stop
INFO: Shutting down
Nov 17, 2016 9:44:46 AM org.openqa.selenium.safari.SafariDriverCommandExecutor stop
INFO: Stopping server
Nov 17, 2016 9:44:46 AM org.openqa.selenium.safari.SafariDriverServer stop
INFO: Stopping server
Nov 17, 2016 9:44:46 AM org.openqa.selenium.safari.SafariDriverCommandExecutor stop
INFO: Shutdown complete
Discovery completed in 3 seconds, 131 milliseconds.
Run starting. Expected test count is: 1
Zeppelin is already running
WelcomePageSuite:
- Welcome sign is correct
Zeppelin stop [  OK  ]
Run completed in 18 seconds, 840 milliseconds.
Total number of tests run: 1
Suites: completed 3, aborted 0
Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (stop-zeppelin) @ zeppelin-server ---
[INFO] Executing tasks

main:
     [exec] Zeppelin is not running
[INFO] Executed tasks
[INFO]

Thanks @cloverhearts for taking care.
LGTM and merge to master as a hotfix.

@cloverhearts Thanks for taking care of this. Will be careful next time

@karup1990 👍 Thank you for making great features.


API add cpu and ram usage Endpoint

Hello,
would it be possible to get the current cpu and ram usage of a Minecraft Server by a API endpoint?
Thank you
With best regards
Dieter

Please do not open Pull Requests that do not change anything. Create an issue instead, and please actually read through the issue list and API code before requesting these things as they already exist.


test: compiling multiple files using the Frontend

This test is currently expected to fail. The RFC #2408 explains the full context. The PR https://github.com/leanprover/lean4/pull/2398 also contains a copy of this test, adapted to use the mechanism proposed there.

This test has been incorporated into the proposed fix at https://github.com/leanprover/lean4/pull/2423.


CSCFAIRMETA-30 & CSCFAIRMETA-32: [ADD] Use DOI option + issued field

This includes CSCFAIRMETA-32 and CSCFAIRMETA-30
CSCFAIRMETA-30: New issued field (not required as default, but required if DOI is used instead of URN), simple date field
CSCFAIRMETA-32: Add option to use DOI instead of URN, when adding files from IDA, done with a DOI checkbox, which also evaluates whether the issued date is set, if checked = true
CSCFAIRMETA-32: For published datasets, the DOI checkbox is hidden
General improvement: Improved dataset loading display while publishing. Now showing the "loading" spinner immediately after "submit dataset" request, to show to the user that something's happening
This loading display is useful especially if there are errors, to reset the error banner

Don't merge until tomorrow at least, since we are doing some experiments with etsin-test --> etsin-test shouldn't be updated.

Needs to be review together with the PO

This can be reviewed


SOLR-17321:  Remove Deprecated URL API in Preparation for Java 21

SOLR-17321

As we started preparing (https://github.com/apache/solr/pull/2497 ) the move to change the Java version for the main branch, It has been decided to merge the deprecated API changes first, and thus reducing the change burden for https://github.com/apache/solr/pull/2497.
Here we are replacing the deprecated URL with new Syntax using URI.
//Deprecated
URL url = new URL(urlStr);

to
URL url = URI.create(urlStr).toURL();

Checklist
Please review the following and check all that apply:

[x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
[x] I have created a Jira issue and added the issue ID to my pull request title.
[x] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
[x] I have developed this patch against the main branch.
[x] I have run ./gradlew check.
[ ] I have added tests for my changes.
[ ] I have added documentation for the Reference Guide

Though new Locale is deprecated, however the newer syntax Locale.of is not available in Java 11. Therefore It can only be done after migrating to Java21 or along with it.
Should we use forbidden APIs to Suppress the API which is forbidden in Java 21? Thread#getID and some other stuff related to SecurityManager.

Seems a shame to add a to. Of suppress for locale and then remove it...  might be cleaner to do that one as part of the upgrade.

As far as security manager goes, don't we need to just tackle that and solve the issue? Versus suppress it and ignore the problem?

Removing a SecurityManager will not be a simple task as some of these settings, I believe, were put in place to fix Security issues. For now, using forbidden would clear the path to the upgradation, Lucene did the same thing. However, now that the JDK has marked it for removal, future JDK upgradation would only be possible after removing it So we should start looking into it right after upgrading to Java 21.
And that's why I suggested marking SecurityManager, and related stuff, forbidden which is a viable solution for now.

As far as security manager goes, don't we need to just tackle that and solve the issue? Versus suppress it and ignore the problem?

Removing a SecurityManager will not be a simple task as some of these settings, I believe, were put in place to fix Security issues. For now, using forbidden would clear the path to the upgradation, Lucene did the same thing. However, now that the JDK has marked it for removal, future JDK upgradation would only be possible after removing it So we should start looking into it right after upgrading to Java 21.
And that's why I suggested marking SecurityManager, and related stuff, forbidden which is a viable solution for now.

yeah..   It sort of feels like that is the opposite of "get the code ready for java 21 first" like you are doing here with Url.   However I also understand that it may spiral into a bigger thing.    The important thing is to keep moving forward!   Thanks for this.

Actually I started looking into it, the first issue that I found was "XXE attack in SolrConfig.xml" and a proposed fix (https://issues.apache.org/jira/browse/SOLR-12316). Afterwards the more improved version of fix involved SecurityManager as in running Solr in sandbox via SecurityManager (https://issues.apache.org/jira/browse/SOLR-13984).
And this was the comment!

The correct way to fix all issues we have seen the last time is very simple: LET'S RUN SOLR INSIDE A SECURITY MANAGER IN PRODUCTION (like in tests). Elasticsearch is doing this, so please please let's do this instead. But this requires to finally get rid of the webapplication and start.jar and add our own bootstrapping (like in tests) that configure Jetty and Security Manager from our own org.apache.solr.bootstrap.Main.java (or similar).  -- Uwe

But I will keep on investigating, How and where Solr using SecurityManager?

I like the forbidden api addition!

All the usages of new URL (except one) has been replaced with URI.create(url).toURL(). The exception was one specific case which is testing the presence of caret in the URL as It was an old issues (https://issues.apache.org/jira/browse/SOLR-5971) that failed due to the caret. Now replacing it with new Syntax fails because caret is in the disallowed list for URI and thus failed. I do not want to encode the character, because the whole point of test is to test caret without encoding it, instead I used @SuppressForbidden.
Because #2497 is still in progress, So, meanwhile,  I added "new URL" to the forbidden api to make sure no one would use it.

I added the changes under 9.7, I did not discussed this. So Should we consider this change for 10 only or release it in 9.7 as well?

Backport it to 9.7 IMO

Agreed.   When I added basic auth to the solr cli I did NOT backport it, and now it's a constant source of pain.

Going to merge this and back port to 9x on Friday, around 7PM IST.

Why use ForbiddenAPI for a deprecated thing?

As we are still in migration phase, It will not be deprecated until we move onto Java 21. Meanwhile, If someone tries to use it , they will get the warning message with an alternative API recommendation.

Why use ForbiddenAPI for a deprecated thing? The JDK (with IDE support too) and javac already handle it in its own way. ForbiddenAPI principally exists for non-deprecated things we insist on not using. cc @uschindler

In Java 11 it is not yet deprecated. So this allows to forbid it now.
Actually once we are on Java 21, the forbiddenapis signatures will complain, too (as deprecated APIs are reported, too). If you ask why forbiddenapis reports deprecated stuff: It's because normally during compilation, JDK only reports a warning for all deprecations (not only in the JDK also in your own code or dependencies). Forbiddenapis allows to restrict only JDK-forbidden stuff and you can keep other deprecations supressed. This is especially important for API developers, because you deprecate your own stuff but still have tests for it.

What I love about our community is that we have so much expertise to call upon; thanks Uwe :-)

Forbiddenapis allows to restrict only JDK-forbidden stuff

Then lets do that and not single out a specific thing (URL).

Ran the failed test multiple times and it's passing.  Flaky!
ERROR: The following test(s) have failed:
  - org.apache.solr.client.solrj.TestLBHttp2SolrClient.testTwoServers (:solr:solrj)
    Test history: https://ge.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.client.solrj.TestLBHttp2SolrClient&tests.test=testTwoServers http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.client.solrj.TestLBHttp2SolrClient.testTwoServers
    Test output: /tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.TestLBHttp2SolrClient.txt
    Reproduce with: ./gradlew :solr:solrj:test --tests "org.apache.solr.client.solrj.TestLBHttp2SolrClient.testTwoServers" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=BC6F7BFC1228CAE8 -Ptests.timeoutSuite=600000! -Ptests.file.encoding=ISO-8859-1
FAILURE: Build failed with an exception.

It recently started failing (I hope I am looking at it in a right way!)

PS :- Thank You @dsmiley. Test history URL in Action!

If there are no more comments/issues, I would like to merge this by Friday. Thanks!


feat 1398: Introduction of UI tests into the Configuration UI project

closes #1398
Includes:

jest and testing library setup
sample tests of LoginView
workflow 'Run Jest'
fixing some lint-errors of existing files that caused the build to fail

This change is

This is how it looks like with the original solution.
The alternative would look like this:


refactor: Fix project structure

CodeFactor found an issue: Unnecessary "elif" after "raise"
It's currently on:
simple_imaging\image.py:28
Commit fea650c6d7daee23d631018a85210ab8bbc384d8


How to 'accuracy' is calculated in different cases in sequence learning using RNN?

I am wondering how the accuracy in training and validation is calculated in sequence learning using RNN? Let's say I have two cases just examples for showing my problems) as following:
(1).
X_train       |    y_train
ABCDEFGHIJ    |       K
BCDEFGHIJK    |       L
CDEFGHIJKL    |       M
DEFGHIJKLM    |       N
EFGHIJKLMN    |       O
......

(2).
X_train       |    y_train
ABCDEFGHIJ    |    BCDEFGHIJK
BCDEFGHIJK    |    CDEFGHIJKL
CDEFGHIJKL    |    DEFGHIJKLM
DEFGHIJKLM    |    EFGHIJKLMN
EFGHIJKLMN    |    FGHIJKLMNO
......

And this is how I understand the accuracy. In general, accuracy=#correct_output_y / #total_y
So, in the first case, as long as the outputted y is equal to the targeted y, then it is correct, where there is only one char to be considered.
However, in the second case, only when all chars in one outputted y are equal to all chars in targeted y, the output will be considered as correct one.
Find the accuracy calculation here:
def categorical_accuracy(y_true, y_pred):
    return K.mean(K.equal(K.argmax(y_true, axis=-1),
                  K.argmax(y_pred, axis=-1)))

But when calculating the loss in the second case, even though most of the chars in one outputted y is correct, which leads to a low loss value, the output y is still considered as wrong when calculating the accuracy.
Am I correct?

The categorical accuracy is doing pairwise matches for item-wise sequence classifications.   So if you had

y_true
y_pred

the fish swam
the bird swam

then this would be 0.66 accuracy.   I'm not sure what you mean by "the output y is still considered wrong".   It's not a binary metric.
Be careful if you use masking, though. #2260 .

@braingineer Oh. It seems like my understanding is wrong.
How I understand is like this:

y_true
y_pred

aaa
aaa

bbb
bbb

ccc
ccc

ddd
ddc

eee
eee

There are 5 data points, 4 of them are correct, then the accuracy is 4/5=80% .
Thanks.

Nope.  It does pairwise comparisons.
K.argmax(y_pred, axis=-1) is over the prediction dimension.
Your dimensions should be:
( batch, sequence length, vocabulary size)
The last dimension classification layer on top of your RNN should be of output size vocabulary size.
Additionally, your y_truth should be of shape batch, sequence length, vocabulary size as well, but the vocabulary size should be a 1-hot.  there is a function to aid in this: keras.utils.np_utils.to_categorical which takes as input a vector of integers and outputs a matrix of 1-hots.
For example, I have encoded as such:

    def serve_sentence(self, data):
        for data_i in np.random.choice(len(data), len(data), replace=False):
            in_X = np.zeros(self.max_sequence_len)
            out_Y = np.zeros(self.max_sequence_len, dtype=np.int32)
            bigram_data = zip(data[data_i][0:-1], data[data_i][1:])
            for datum_j,(datum_in, datum_out) in enumerate(bigram_data):
                in_X[datum_j] = datum_in
                out_Y[datum_j] = datum_out
            yield in_X, out_Y

    def serve_batch(self, data):
        dataiter = self.serve_sentence(data)
        V = self.vocab_size
        S = self.max_sequence_len
        B = self.batch_size

        while dataiter:
            in_X = np.zeros((B, S), dtype=np.int32)
            out_Y = np.zeros((B, S, V), dtype=np.int32)
            next_batch = list(itertools.islice(dataiter, 0, self.batch_size))
            if len(next_batch) < self.batch_size:
                raise StopIteration
            for d_i, (d_X, d_Y) in enumerate(next_batch):
                in_X[d_i] = d_X
                out_Y[d_i] = to_categorical(d_Y, V)

            yield in_X, out_Y

Sorry, I should also have mentioned that it matters if your RNN is serving the last output or the sequence output.   If return_sequences=True then it behaves as I said.  If not, then it only returns the last output.


flogo expressions support for rule conditions

Support flogo expressions syntax for rule condition evaluation such as:
$.t2.p2 > $.t1.p1 where t2 and t1 are tuple types and p2 and p1 are their respective properties
(See ruleapi/tests/expr_* tests for usage)
Additionally, when run as a flogo actiontype, support for environment variables and flogo properties via
$env['envvar'] and $property['name']
(See examples/flogo/simple/flogo.json for usage in a flogo app)

This PR can be ignored as the same content is available in other PR #97


Feature Request: Configurable Line Endings (I can make the pull request)

It looks like that the plugin always writes project files using LF line-endings.
We normally have CRLF for our csproj files - so I would like to be able to configure or detect what line endings a file have - and stick to that.
It sadly looks like that xml-js does not have a setting to convert the line endings - so we need to find some recommended way to do that after the fact.
I am ready to give it a try and make a pull request for it - but I wanted to raise the issue on GitHub - so we can discuss the right solution.

I have a working prototype using the eol module. It simple just calls that after creating the XML. If that library is okay to add as a dependency - when it should be easy.

Any PR will be welcome :)


Intermittent "connection reset by peer" while pushing image

We're running docker-registry-v2 on an AWS EC2 instance - backed by an EBS volume (switched from S3 since we thought that might be the underlying issue).
From another AWS EC2 instance we're running a Bamboo CI agent which are building docker images and pushes them to our docker-registry.
Several times each day we get failed builds caused by docker push getting connection reset by peer.
[info] time="2015-07-30T10:00:31+02:00" level=fatal msg="Error pushing to registry: Put https://docker-registry.**.**/v2/userhq/blobs/uploads/e60ca766-eb26-4adf-8cdb-fe7a127e3e4c?_state=a7vvRPLZLleaqCBw4xGmaxJZ-Z0Jc0SsFUUtcrnlqft7Ik5hbWUiOiJ1c2VyaHEiLCJVVUlEIjoiZTYwY2E3NjYtZWIyNi00YWRmLThjZGItZmU3YTEyN2UzZTRjIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE1LTA3LTMwVDA4OjAwOjI2LjA0MzcwNzk2WiJ9&digest=sha256%3A1407c3b1319f21131f9da23c859ac406d2ae1051190611046c1666fc86dc5376: read tcp <IP_ADDRESS>:443: connection reset by peer"

Configuration and host information below, how can I debug this issue?
The docker registry is fronted by nginx configured as below:
user  nginx;
worker_processes  1;

events {
  worker_connections 1024;
  use epoll;
  multi_accept on;
}

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

http {
    include               /etc/nginx/mime.types;
    default_type          application/octet-stream;

    log_format            main  '$remote_addr - $remote_user [$time_local] "$request" '
                            '$status $body_bytes_sent "$http_referer" '
                            '"$http_user_agent" "$http_x_forwarded_for"';

    access_log            /var/log/nginx/access.log  main;

    sendfile              on;
    keepalive_timeout     65;
    ssl_session_timeout   10m;

    ssl_certificate       /etc/nginx/ssl/server.crt;
    ssl_certificate_key   /etc/nginx/ssl/server.key;

    ssl_ciphers                 "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
    ssl_protocols               TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers   on;
    ssl_session_cache           shared:SSL:10m;
    add_header                  Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
    add_header                  X-Frame-Options DENY;
    add_header                  X-Content-Type-Options nosniff;
    ssl_session_tickets         off;
    ssl_stapling                on;
    ssl_stapling_verify         on;

    server {
        listen 443 ssl;
        server_name myregistrydomain.com;

        client_max_body_size 0;

        chunked_transfer_encoding on;

        client_body_buffer_size 100m;

        location /v2/ {
            # Do not allow connections from docker 1.5 and earlier
            # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
            if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
                return 404;
            }

            auth_basic "registry.localhost";
            auth_basic_user_file /etc/nginx/registry.htpasswd;
            add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

            proxy_pass                          http://docker-registry:5000;
            proxy_set_header  Host              $http_host;   # required for docker client's sake
            proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
            proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
            proxy_set_header  X-Forwarded-Proto $scheme;
            proxy_read_timeout                  900;
        }
    }
}

Docker registry configuration:
version: 0.1
log:
  level: info
  formatter: text
  fields:
    service: registry
    environment: production
storage:
  filesystem:
    rootdirectory: /var/
  cache:
    layerinfo: redis
  maintenance:
    uploadpurging:
      enabled: true
      age: 72h
      interval: 8h
      dryrun: false
reporting:
  newrelic:
    licensekey: xxxxx
    name: docker-registry
    verbose: false
http:
  addr: :5000
redis:
  addr: cache:6379
  db: 0
  dialtimeout: 10ms
  readtimeout: 10ms
  writetimeout: 10ms
  pool:
    maxidle: 16
    maxactive: 64
    idletimeout: 300s

docker compose is used to start it all:
cache:
  image: redis

nginx:
  build: ./nginx
  links:
    - registry:docker-registry
  ports:
    - 443:443

registry:
  build: ./registry
  volumes:
    - /data/docker:/var/docker
  links:
    - cache

Docker registry host:
$ docker info
Containers: 3
Images: 73
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 79
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-58-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 1
Total Memory: 3.676 GiB
Name: ip-172-31-24-143
ID: OTJI:CTOH:DLAP:JVAG:RFEP:4VWV:RR2U:SMJL:E5LU:PYOZ:CYMF:DRBA
WARNING: No swap limit support
ubuntu@ip-172-31-24-143:~$ docker info
Containers: 3
Images: 73
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 79
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-58-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 1
Total Memory: 3.676 GiB
Name: ip-172-31-24-143
ID: OTJI:CTOH:DLAP:JVAG:RFEP:4VWV:RR2U:SMJL:E5LU:PYOZ:CYMF:DRBA
WARNING: No swap limit support

$ docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

Bamboo CI host:
$ docker info
Containers: 0
Images: 106
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 106
 Dirperm1 Supported: false
Execution Driver: native-0.2
Kernel Version: 3.13.0-52-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 8
Total Memory: 14.69 GiB
Name: ip-172-31-9-199
ID: ZTUG:L4YS:6JUT:KLH4:4PPE:5CGO:GBHD:4UI4:IA6F:OI3B:T5GY:7DX2
WARNING: No swap limit support

$ docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 7c8fca2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64

@pliljenberg can you get nginx error / access log, and registry logs as well?
Something in the http stack is closing the connection (probably timeouting).

@dmp42 I'll see what I can dig out. Log settings is INFO I think so it might not say much.
Do you use any sort of proxy?
If not, that points out to network issues between your bamboo host and your registry.
Any chance you can mtr there and figure if you have packet loss?

Had to open for ICMP trafic (guess that's not required for the docker client networking to work correctly?), report:
Start: Tue Aug  4 09:23:33 2015
HOST: ip-172-31-28-41             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  2.|-- ec2-52-17-235-32.eu-west-  0.0%    10    0.4   0.4   0.4   0.4   0.0

Not sure if it is related but I upgraded the docker-client (on the CI host) to 1.7.1 this weekend and so far we haven't seen any connection reset by peer?

These issues are likely unrelated. i/o timeout most of the time really is just an indication that you are having connectivity issues.
The MTR output doesn't look right. Any chance you can run it again?

@dmp42 I ran mtr -rw hostand got the results above...
$ ping docker-registry-host
PING docker-registry-host (<IP_ADDRESS>) 56(84) bytes of data.

64 bytes from ec2-52-17-235-32.eu-west-1.compute.amazonaws.com (<IP_ADDRESS>): icmp_seq=1 ttl=63 time=0.495 ms
Running against the raw IP gives the same result, am I doing it wrong?
mtr -r <IP_ADDRESS>

Start: Wed Aug  5 08:57:35 2015
HOST: ip-172-31-19-9              Loss%   Snt   Last   Avg  Best  Wrst StDev
1.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
2.|-- ec2-52-17-235-32.eu-west-  0.0%    10    0.5   0.6   0.5   0.8   0.0

I seem to be having the same issue. Also on Amazon ec2 with docker 1.7.1 (I am getting the connection reset by peer message).
I just enabled s3 logs, waiting for it to take affect to see those log outputs

I can't see any s3 logs about connection reset or timeouts..

@jasonf20 for @pliljenberg there is no logs either on the server side. This further point to network issues in between.
That kind of issue is hard to diagnose or even pinpoint - and it's unlikely Amazon support would investigate in my experience.
Running MTR and spotting packet loss is the only way to get there I think.

I am having the same issue. I've got a registry-v2 on an ec2 instance behind nginx. I've got another instance running Jenkins building images. When I push an image I get the Error pushing to registry: Put <url> read tcp <ip>: connection reset by peer. Some layers push without a problem, but it appears that mostly the larger ones fail. When pushing from my own machine I never had the error.

If that happens only on jenkins, then the issue is likely there... is there any proxy there? virtual machine involved? running in a container?

@dmp42 In my case I have a server running teamcity (amazon ec2 server - with no inner vm or docker) which sends directly to the docker-registry (also directly on ec2 instance). No nginx, proxy or the likes. I do have TLS enabled on the registry.
I think that even if this is a network related issue, (I'll also try to run mtr soon - but suspect no packet loss) then docker should be more resilient towards it. For example perform a couple of retries, or resend of failed data.

Additionally, I also ran mtr at the same time as pushing (in both directions) and had no package loss at the time of a push error ("connection reset")

I did some more testing. I've got 2 EC2 instances. One is running a registry-v2 container, the other has an image which I am trying to push to the registry. I also have my own machine which is not in AWS, it is my laptop.

Push to local address from other ec2 instance

I tagged the image I am trying to push with the local ip addess and port of the registry:
docker tag someplace/someimage 172.x.x.x:5000/someimage
I added the registry as an insecure registry to the docker opts and restarted the daemon. I can push the image successfully (it was not present in the registry).
I removed all data from the registry and pushed the image again successfully (I did this 3 times to be sure it wasn't a one-off)

Push to external address from other ec2 instance

I tagged the image I am trying to push with the external ip addess and port of the registry:
docker tag someplace/someimage 52.x.x.x:5000/someimage
I added the registry as an insecure registry to the docker opts and restarted the daemon. When I push the image I get an Error pushing to registry: Put <url> read tcp <ip>: connection reset by peer
I removed all data from the registry and pushed the image again without success (I did this 3 times to be sure it wasn't a one-off)
Some layers are pushed successfully, but then one will fail. When I try again some more will be pushed when I am lucky. There is a layer of 200MB which I have not been able to push.

Push to external address from a machine outside of AWS

I tagged the image I am trying to push with the external ip addess and port of the registry:
docker tag someplace/someimage 52.x.x.x:5001/someimage
I added the registry as an insecure registry to the docker opts and restarted the daemon. When I push the image I get an Error pushing to registry: Put <url> read tcp <ip>: connection reset by peer
I removed all data from the registry and pushed the image again without success (I did this 3 times to be sure it wasn't a one-off)
Some layers are pushed successfully, but then one will fail. When I try again some more will be pushed when I am lucky. There is a layer of 200MB which I have not been able to push.

Push to hostname + nginx from other ec2 instance

I tagged the image I am trying to push with the hostname of the registry:
docker tag someplace/someimage registry.example.com/someimage
I added the registry as an insecure registry to the docker opts and restarted the daemon. When I push the image I get an Error pushing to registry: Put <url> read tcp <ip>: connection reset by peer
I removed all data from the registry and pushed the image again without success (I did this 3 times to be sure it wasn't a one-off)
Some layers are pushed successfully, but then one will fail. When I try again some more will be pushed when I am lucky.

Push to hostname + nginx from machine outside of AWS

I tagged the image I am trying to push with the hostname of the registry:
docker tag someplace/someimage registry.example.com/someimage
I added the registry as an insecure registry to the docker opts and restarted the daemon. I can push the image successfully (it was not present in the registry).
I removed all data from the registry and pushed the image again successfully (I did this 3 times to be sure it wasn't a one-off)
Final notes:

I am not pushing the image from within a container.
docker version 1.7.1 and registry 2.1.1 (although we have had this issue since docker 1.5 and registry version 1)
The ec2 instance are virtual machines of course (ubuntu)
The registry uses an S3 bucket for storage

So it appears to be an issue with Amazon resetting connections when using external addresses between instances. An not a docker or registry or nginx issue. Is somebody able to reproduce this?

@VogonogoV smells like ELB timeout?
If you are using ELB, I suggest you look into them settings, or even switch to TCP proxy mode.

@dmp42 I don't use ELB. Any other suggestions?

I have the same issue exactly, but do not use elb, so I don't think it's
related
On Aug 25, 2015 5:49 PM, "VogonogoV"<EMAIL_ADDRESS>wrote:

@dmp42 https://github.com/dmp42 I don't use ELB. Any other suggestions?

—
Reply to this email directly or view it on GitHub
https://github.com/docker/distribution/issues/785#issuecomment-134610718
.

Ok. Keep us posted if you get any infos from Amazon / forums.
tcpdumping the traffic and capturing the traffic might help figuring out "what" "where" is the connection being closed/timeouting.

There is a layer of 200MB which I have not been able to push.

I haven't had time to read through the full issue but I suspect that the client_body_buffer_size 100m; configuration directive may be contributing to the failures. 100MB is way too big for most connections. I'd recommend removing this directive or setting to anywhere from 1-4MB, depending on available bandwidth.

@stevvooe Thanks for the suggestion, but nginx is not the problem. I'll summarize the issue as I am seeing it.

From
To
Registry address
nginx
Does it work?

ec2 machine
ec2 hosted registry
172.x.x.x
no
yes

ec2 machine
ec2 hosted registry
52.x.x.x
no
no

ec2 machine
ec2 hosted registry
registry.ourcompany.com
yes
no

my laptop
ec2 hosted registry
52.x.x.x
no
yes

my laptop
ec2 hosted registry
registry.ourcompany.com
yes
yes

When it doesn't work I get Error pushing to registry: Put <url> read tcp <ip>: connection reset by peer. Some layers will be pushed successfully. When I retry the push, more layers will be pushed. I have not been able to push a 200MB layer from an ec2 machine.
My conclusion: it is not a docker, registry, nor nginx problem. I think it's an AWS problem.
@dmp42 I'll try tcpdumping when I have the time. I'll post updates here.

@VogonogoV That table makes it pretty clear that the issue has something to do with ec2. I wonder if you need to use a different dns entry for inside ec2. The issue may be that routing using the externally available 52.x.x.x IP is not optimal for host to host ec2 transfers.

@stevvooe As a workaround I am now building images on the ec2 machine and tagging them with the local address: docker build -t 172.x.x.x:5000/some-image. When I pull the image on my machine it just uses the name you pulled it from so: docker pull registry.example.com/some-image. It works, although it is kinda freaky. Thanks for the help, nobody at the Amazon forums replies :cry:

@VogonogoV Did you try tweeting? Their support tends to work better with motivation. ;)
@RichardScothern @dmp42 @aaronlehmann Should we add a blurb in documentation about this when using s3?

I hit the same problem. Any progress on the AWS side? I would like to avoid the workaround from @VogonogoV

@robertfirek Unfortunately, this is an AWS issue and we don't really have the means to investigate. I'd recommend you open an issue with their support team.
@dmp42 Any ideas here?

I haven't heard anything from AWS. I didn't tweet because I don't have Twitter. So anyone who wants to, feel free. Opening an issue might give some more insights.
I've been using the workaround for 3 weeks now an I haven't looked back at it again.

@stevvooe @VogonogoV Thank you for your update.

@VogonogoV @robertfirek I have flexed the power of the tweet: https://twitter.com/stevvooe/status/644448877978845184. We'll see if they help.
@VogonogoV Do you have a link to the forum post?

Got it, sorry for the delay!
https://forums.aws.amazon.com/thread.jspa?threadID=215090

@robertfirek I have contacted aws support for this and so far this is the "answer" (though they said they are still investigating it):

Within VPC, there's an extra layer of network infrastructure that lives above the EC2 instances. This isn't visible to you as a customer, but it handles all of the Public IP addressing and mapping to the instances within the VPC.

Now, as a connection is made from one EC2 instance to another over the public IP, the traffic needs to go "out" to the public internet, route to the dest.EC2.IP, and then traverse the network infrastructure in reverse. While this may work some of the time, it's not considered best practice. It's the functional equivalent of "out and back again" with Cisco equipment.

With that said, if you're connecting two EC2 instances together within a VPC, we highly recommend that you use the private IP addresses. If you need to add a DNS name for portability, then we highly encourage the use of Route53 Private Zones

I still have not gotten around to playing with this some more, however this is basically the workaround and I see no reason why amazon should have trouble with public IP addressing withing their own network, so hopefully a better solution will come out of this.

Thanks for response :)
Maybe its an idea to add some information to some docker docs: Pushing an image from one EC2 machine to another (a registry on EC2) might not work when using the external address of the EC2 machine.

@jasonf20 Thank you for poking them:) I agree with you. I don't see any good explanation why I should not use my public address/IP to connect to another instance (especially when they are in different VPCs and can be expose by proxy).
I use my registry both from my computer and from other VPCs. I think it is an error prone solution to create two separate addresses for the same registry.

@jasonf20 Thank you for adding your response.
@robertfirek While this is an unfortunate workaround, this explanation is actually very good, from a networking perspective. Albeit, it is hard to understand why this particular case is so pathological, since the network is opaque from our perspective. I would expect for them to populate a route for the VPC to avoid this extra traversal since that is the standard way to mitigate "out and back again". There may be mitigating factors for which we don't have visibility.

@jasonf20 I've tried solution with Private DNS zone and it doesn't solve the problem. When registry is in another VPC and you connected through peering connection from another VPC you have the same issue. When registry and client is in the same VPC the problem disappears.
Even if we don't "out and back again" case we still can get the same error.

@robertfirek That's good info, thanks. I will update the AWS query I have later with this info.

I work with @robertfirek and we narrowed the problem down to the TCP conversation in the exchange below, that shows the connection being reset. In our case this happened after ~73MBs pushed. "10.1.x" is the client and "172.31.x" is the Docker registry.
This looks to me like a strictly end-to-end TCP problem, somewhere between Go & Linux, and not directly AWS networking related (unless they a huge number of packets - and we do see loads of "out-of-order" and "dup-acks" before this happens - or do some serious TCP MITM shenanigans...)
Transfer-Encoding: chunked
Accept-Encoding: gzip```

<img width="1388" alt="capture d ecran 2015-09-18 a 16 11 11" src="https://cloud.githubusercontent.com/assets/1047312/9962875/087e4ed4-5e20-11e5-88e2-bb7ed962b916.png">

+1
I'm running docker registry v2 on an EC2 instance (Ubuntu 14.04) with a self-signed SSL certificate and a public IP address, with no intermediate proxy, using filesystem storage. I'm trying to push to it from a docker client outside EC2 hosted at a non-Amazon datacenter (also Ubuntu 14.04). I see the same symptoms everyone else is seeing here: intermittent connection resets when pushing large layers. I also see Wireshark reporting TCP out-of-order, dupe ACKs, finally followed by the registry sending a TCP reset that breaks the connection.

+1 confirm problems with running registry in ec2 behind nginx, read tcp xx.xx.xx.xx:5000: connection reset by peer occurs when pushing large layers

@ndelitski @gabenell @renaudguerin How would you like us to proceed on this issue? I am not sure if there is a specific fix we can do in just the registry that will address these problems.
Here are few possibilities that we may be able to address:

Perhaps there is a timeout that could be adjusted somewhere. Unlikely, since we are seeing resets.
Go's network stack could be at fault.

Without deep network analysis, I am not sure if we can identify the root cause.
 I've tweeted in their direction again: https://twitter.com/stevvooe/status/656910931474018305.

@stevvooe Someone on https://forums.aws.amazon.com/thread.jspa?threadID=215090 asked for some instance to check. Does anyone have spare one?

Hello, the AWS team is currently working with Docker to try and replicate this reported issue.
@gabenell - you mentioned that you had a wireshark capture; would you be willing to share that with us to help troubleshoot this issue? You can share this in the AWS Forum post tracking this report if you perfer: https://forums.aws.amazon.com/thread.jspa?threadID=215090
Thank you.

I don't have an instance with the not working setup anymore, because I need to have it working. So I'm using my previously mentioned workaround. I can recreate the issue and send a network capture if I get some instructions on what is needed and how to do it.

Thank you @VogonogoV. Here are the steps:

From the client, sudo tcpdump -i eth0 -vnnlp -s0 -w client.pcap
From the server (registry host), sudo tcpdump -i eth0 -vnnlp -s0 -w server.pcap
Attempt to replicate the problem by pushing a docker image from the client.
Once replicated, ctrl+c on both client and server to end the captures.

The files will likely be several hundred megabytes in size. If you can upload the captures to an S3 bucket, then generate a signed URLs if you prefer. You can share those URLs with me directly (suarea at amazon dot com). Thank you.

I'm seeing the same issue whereby the connection is dropping on Amazon AWS. Doesn't matter if you use the local disk or s3 and only seems to happen for layers that around 50MB or larger.

@JBodkin We are talking with AWS on this issue. They are blocked on needing a network trace. It would be great if you could contact them through https://forums.aws.amazon.com/thread.jspa?threadID=215090. Please let me know if you have success.

I'll see if I can grab an network trace the next time, for now I've started using VPC peering to resolve the issue.

I have this issue. Do you still need diagnostics - I can reproduce at will

@gtmtech I see the linked issue is still open; https://forums.aws.amazon.com/thread.jspa?threadID=215090 (and also contains instructions on how to get a network trace)

@gtmtech that would be extremely helpful

@JBodkin @gtmtech Reach out to AWS on that support issue. They are looking for a solid reproduction before moving forward. Let me know if you need anything.

I will try and do some diagnostics tomorrow - hopefully I can report back in 24h

Hi, sorry for the delay I did some testing this morning. Our setup is registry running in a docker container with an nginx SSL proxy setup running in another docker container, with docker pull/push happening sometimes from within the same VPC, sometimes from a different VPC, and sometimes outside.
I ran some tcpdumps, but it has to be said at the weekend I was seeing roughly 30% failure rate, and today its virtually nil which I cant account for as the configuration hasnt changed.
The particular error I was seeing at the weekend I have now diagnosed as an intermittent SSL handshake error coming from the nginx proxy, resulting in a bad or reset connection so it might be a red herring for this issue - I'll let you know if we discover more, in our case we obviously need to look into our SSL handshaking first.

Thanks for reporting back, @gtmtech!

@gtmtech Any luck contacting AWS support?

Any news on this issue?
I can reproduce it as well.

I was having this issue and thought it might be related to timeouts based on an earlier comment.  I thought this might have something to do with the block write size being too large and occasionally timing out so I changed (shortened? it's in KB, right?) it and explicitly set to the below and the problems seemed to resolve afterwards.
storage:        chunksize: 5242880

@13josh Users experiencing this issue are having connection resets while pushing and pulling over some VPC configuration. If you are experiencing this issue, changing the buffer size may help, for certain configurations, as it can change the connection envelope for a single given chunk request. Albeit, we am not sure that changing buffer size having an effect is indicative of this problem. In fact, if that worked, it may be a different issue.
@EliasGabrielsson @13josh I'd recommend relating your experience to https://forums.aws.amazon.com/thread.jspa?threadID=215090.

I get this issue as well, Im not using ec2 but instead a private network. Registry version 2 behind nginx proxy.

@dolphyvn This issue covers "connection reset" with AWS. If you're having this while not using AWS, you're likely having another, unrelated problem.

FYI, after spending a few hours on this with a sysadmin, we figured out that when you use --net host instead of --net bridge the whole problem disappear.
After a bunch of tcpdump and stuff, it seems that a RST packet are being sent by the host (and not the docker container), resulting in "connection reset by peer".
When you ps auxf on the EC2 running the registry container, you can see that the container is running a docker-proxy in front of the container. When using host network, the proxy disappears and the problem with it.
Hope it helps, I'm copying this on the AWS issue too
Bye

Well this is unfortunate because this means you can't reliably host distribution on AWS using ECS since ECS doesn't support --net host : https://github.com/aws/amazon-ecs-agent/issues/185

@kepeket That is an interesting data point. It might be worthwhile relating your experience on https://forums.aws.amazon.com/thread.jspa?threadID=215090.

Hi
Yes i tried but the login page of Amazon was infinite looping I don't why
I'll retry a bit later

à 21:05, Stephen Day<EMAIL_ADDRESS>a écrit :
@kepeket That is an interesting data point. It might be worthwhile relating your experience on https://forums.aws.amazon.com/thread.jspa?threadID=215090.
—
Reply to this email directly or view it on GitHub.

I hit the issue when I did a docker push on an EC2 instance to a registry running on the same instance, using the external-facing IP address. Using the internal IP did not trigger the problem. A large upload would generally fail after about 8 seconds. It was very easy to reproduce.
I collected a tcpdump to see what was happening. At the moment the upload failed, the EC2 instance was receiving a packet that was very far out of sequence. Its sequence number and timestamp were several seconds behind the actual TCP stream. Interestingly, this did not seem to be a retransmit of a previously-sent packet. Presumably this packet is generated within AWS' infrastructure.
Normally a packet like this should be treated as a spurious retransmit and ignored, but for some reason it was causing the local host to generate a RST packet and kill the connection. Given the anecdote in https://github.com/docker/distribution/issues/785#issuecomment-183338454 that running the registry container in host networking mode works around the issue, I suspected this had something to do with how Docker bridge networking works.
When operating in bridged mode, Docker creates some iptables rules to perform NAT between the exposed address/port and the container's internal address/port. I had a look at Linux's NAT implementation, which builds on top of nf_conntrack for connection tracking. nf_conntrack has a state machine that tracks connection state. If nf_conntrack believes its state is out of sync with the actual connection, it treats incoming packets as invalid. One of the checks is the tcp_window function, which rejects packets outside the TCP window. I believe this is the check that is failing.
nf_conntrack has a "be liberal" flag that accepts these packets as valid. Sure enough, after running:
echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal

...I haven't been able to trigger the issue anymore.
If this is indeed a successful workaround, should we consider having Docker Engine switch on that flag by default?
cc @tonyhb @dmp42 @stevvooe @mrjana

@aaronlehmann Are you sure this workaround fixes the problem? Because conntrack will simply accept this packet if it's outside the window http://lxr.free-electrons.com/source/net/netfilter/nf_conntrack_proto_tcp.c#L1036. Just that it won't update ct state but since this is really an unexpected packet it makes sense not to update ct state. This doesn't seem to be the direct cause of RST packet.

@mrjana: I'll do more testing tomorrow to confirm the workaround.
I'm not sure what causes the RST packet to get generated. Perhaps there's an iptables rule that's rejecting INVALID packets, or similar implicit behavior in the kernel.
FWIW, I found this post, which describes a similar problem that was resolved by switching on the "be liberal" flag: https://www.pitt-pladdy.com/blog/_20091125-185551_0000_Linux_Netfilter_and_Window_Scaling/

@mrjana: I can confirm that the setting makes a difference. I set ip_conntrack_tcp_be_liberal to 1, and pushed a huge image three times. The pushes succeeded without incident. Then I set the flag back to 0, and repeated the test. Each of three attempts had a severed connection within seconds.

Thanks for confirming that @aaronlehmann.
I took a look at all the places in the kernel where an RST could be generated and they are mainly generated when disconnect or close happened with unread data or if there was memory pressure in the host. So looks like RST is happening due to some indirect sequence of events after receiving the unexpected sequence number packer and I think we need to connect some missing dots there.
I will try to take a look at the packet capture that you sent me and see if I could see when exactly RST is getting generated in relation to the packet with the unexpected sequence number.

@mrjana: I think I get it now. When conntrack treats a packet like this as "invalid", it doesn't associate it with the flow that its tuple corresponds to. Thus, the packet doesn't get rewritten by the NAT rule, and ends up being handled as if it was part of a connection to the host's actual IP address. The host sees that it doesn't have a matching flow, and (correctly) sends a RST packet.
I found I can also work around this by adding a rule to the INPUT chain that drops invalid packets:
iptables -I INPUT -m conntrack --ctstate INVALID -j DROP

This prevents the packet from being interpreted as destined to the pre-NAT IP address, and prevents the RST from being generated.

@aaronlehmann Yes I agree with your prognosis. That is indeed what is happening here. And having to drop if ctstate is invalid is a much better fix than enable be_liberal flag. Although it probably should be inserted in DOCKER chain in the PREROUTING hook. Can you please file an issue in docker/libnetwork to get it fixed although it has very limited chance of making it into 1.11?

@aaronlehmann @mrjana Bravo!

Filed https://github.com/docker/libnetwork/issues/1090. Will also reach out to AWS with our conclusions.

I reported the findings to AWS. Some testing showed this affected instances in Ireland, but not Northern Virginia. They were able to reproduce the problem, and seem to have fixed it in the Ireland region. I'm going to close this ticket since we now know it's not a distribution-specific problem, and the AWS issue appears to be resolved.

I recently hit the same issue. It was caused by a problem affecting my network card. In my host (Windows Server 2016) when Hyper-V is enabled for docker, i have created an external virtual switch using the Network Interface card, and since that i hit the issue. It was resolved when i removed the Virtual Switch. Long story short, check your network interfaces (drivers, firmware...)

My 3G internet network disconnects when I try to docker push

@ikr0m then it's likely an issue with your 3G provider, right?

@dubo-dubon-duponey I'm not sure why it should be issue with provider. After executing docker push ... on terminal 3G connection disconnects. It happened twice. I was trying Getting started section of docker docs and it was simple image push to docker hub. I achieved to push finally with other workaround. I didn't dig into details of previous disconnection. That's why I found this thread to find some solution.

I'm not sure why it should be issue with provider. After executing docker push ... on terminal 3G connection disconnects.

It's hard to tell without more debugging information, but what you describe here looks like your 3G connection is not stable - which is why I suggested it might be an issue with your provider.
If that's the case, then it's unlikely there is anything docker can do here. "docker push" does not make you lose your uplink (3G or otherwise) by itself...
Suggestion: try to HTTP POST random content using curl, and see if it triggers the same problem with your connection.


Should fire the TileLayer events.

I don't think it fires the TileLayer events, such as tileloadstart.
I'll try figure out how to add them.

Hi @master5o1. Sorry for the terribly long delay. Feel free to re-open this with a pull request if you implement it. Thanks!


Add validateDomainName to ValidatesAttributes

There were no validation rules for domain names in the framework. This simple method validates an input against rfc1035.

I think the easier answer here might be using filter_var, which validates specifically against RFC 1035.
return filter_var($value, FILTER_VALIDATE_DOMAIN) !== false;

Thanks for your pull request to Laravel!
Unfortunately, I'm going to delay merging this code for now. To preserve our ability to adequately maintain the framework, we need to be very careful regarding the amount of code we include.
If possible, please consider releasing your code as a package so that the community can still take advantage of your contributions!
If you feel absolutely certain that this code corrects a bug in the framework, please "@" mention me in a follow-up comment with further explanation so that GitHub will send me a notification of your response.

Without FILTER_FLAG_HOSTNAME flag,filter_var doesn't work as expected. It recognizes com, -1.com, 1-.com and 1_1.com as valid domains!
So, we should write:
return filter_var($value, FILTER_VALIDATE_DOMAIN, FILTER_FLAG_HOSTNAME) !== false;

But, as the ToDo shows, it will be more robust to check the TLD against the list of TLDs that IANA provided.


Address tech debt

Alaska Airlines Pull Request
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Resolves: # (issue, if applicable)
Summary:
Please summarize the scope of the changes you have submitted, what the intent of the work is and anything that describes the before/after state of the project.
Type of change:
Please delete options that are not relevant.

[x] New capability
[x] Revision of an existing capability
[ ] Infrastructure change (automation, etc.)
[x] Other (please elaborate)

Checklist:

[x] My update follows the CONTRIBUTING guidelines of this project
[x] I have performed a self-review of my own update

By submitting this Pull Request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Pull Requests will be evaluated by their quality of update and whether it is consistent with the goals and values of this project. Any submission is to be considered a conversation between the submitter and the maintainers of this project and may require changes to your submission.
Thank you for your submission!
-- Auro Design System Team

Summary by Sourcery
Refactor the component registration process to utilize RuntimeUtils, update import paths, and introduce runtime utilities for component tag management. Update documentation to reflect these changes and clean up unused scripts. Add a new ESLint configuration file.
Enhancements:

Refactor the component registration process to use RuntimeUtils for registering custom elements.
Update import paths for buttonVersion and iconVersion to include file extensions.
Introduce AuroLibraryRuntimeUtils to handle component tag renaming.

Documentation:

Update documentation to reflect changes in the component registration process using RuntimeUtils.

Chores:

Remove unused scripts generateDocs.mjs and staticStyles-template.js.
Add a new ESLint configuration file eslint.config.mjs.

:tada: This PR is included in version 2.0.0-beta.1 :tada:
The release is available on:

npm package (@beta dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Formatting issues with python module on default settings for function, return, & new object declaration/values

There is an issue with the python import version of this where the function declaration, return values, as well as "new" objects are stringed together. These do not have a space preventing the code from being ran or compiling.
Both the functions, return values, as well as "new" objects declaration are stringed together. See below.
The only version (https://beautifier.io/) does not have this issue. This was tested with a fresh installation directly from installing with pip according to the instructions outlined for python.
Example:
if (N == 6) {         s = GetObject(y[0]).InstancesOf(y[1]);         var en = newEnumerator(s);         for (; !en.atEnd(); en.moveNext()) {             var it = en.item();             returnit.volumeserialnumber;             break;         }     }
or
functionCr(N) {     returnnewActiveXObject(j[N]); }
Should be:
if (N == 6) {         s = GetObject(y[0]).InstancesOf(y[1]);         var en = new Enumerator(s);         for (; !en.atEnd(); en.moveNext()) {             var it = en.item();             return it.volumeserialnumber;             break;         }     }
function Cr(N) {     return new ActiveXObject(j[N]); }

Issue non-existent. There was another beautifier that I had auto-running causing a conflict. I completely forgot that I had it up. Please disregard


Add constants of socket options

Signed-off-by: Kenta Tada<EMAIL_ADDRESS>

r? @JohnTitor
(rust_highfive has picked a reviewer for you, use r? to override)

@bors r+

:pushpin: Commit 1f97fa1db0454e4d2a5bf4dea157042c41de37ba has been approved by JohnTitor

:hourglass: Testing commit 1f97fa1db0454e4d2a5bf4dea157042c41de37ba with merge f18c583f19e22d6e497b5b33b45925862e318d51...

:sunny: Test successful - checks-actions, checks-cirrus-freebsd-11, checks-cirrus-freebsd-12, checks-cirrus-freebsd-13
Approved by: JohnTitor
Pushing f18c583f19e22d6e497b5b33b45925862e318d51 to master...


Added generic description based on device generic/specific code

Also fixed a couple of nil pointer exceptions

Codecov Report

Merging #29 into master will decrease coverage by 0.1%.
The diff coverage is 31.25%.

@@            Coverage Diff             @@
##           master      #29      +/-   ##
==========================================
- Coverage   59.83%   59.73%   -0.11%
==========================================
  Files          36       36
  Lines        1845     1855      +10
==========================================
+ Hits         1104     1108       +4
- Misses        715      721       +6
  Partials       26       26

Impacted Files
Coverage Δ

generators/commandclasses/commandclasses.go
0% <0%> (ø)
:arrow_up:

connection.go
66.66% <16.66%> (-0.35%)
:arrow_down:

nodes/node.go
23.49% <57.14%> (+1.61%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c389b24...aa08baf. Read the comment docs.

@stamp, are all these commits based on the original PR?

This was ages ago :p
It looks like its based on c389b2433780d904b85271ef2dc646ad368dd097

2017-09-17 19:02 GMT+02:00 Luke Scott<EMAIL_ADDRESS>
@stamp https://github.com/stamp, are all these commits based on the
original PR?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stampzilla/gozwave/pull/29#issuecomment-330062564,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABF-DfhnASdWkm9VlCZj9zwx-5M6C9yQks5sjVClgaJpZM4NkT65
.

@stamp, ignore me, I responded to the wrong PR.


Blob path pattern

Hi,
Is it possible to use regex or some kind of expression in Blob Path ? I have a pipeline which has a Copy activity, and I want this pipeline to be triggered when a file is uploaded on my Blob Storage, so its data are copied to the right table in the database.
But the filename of the file is different from one file to another, depending on the name of the destination table and on the version of the file : {MyTableName}_{MyCounter}.csv (where 001 is my counter for the file version). Is it possible to do that ?
Thank you !

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b7ee5166-ff40-f4ed-ce07-5312f935e388
Version Independent ID: 68f01b43-e3b2-003b-24cb-446a08aa0c1b
Content: Create event-based triggers in Azure Data Factory
Content Source: articles/data-factory/how-to-create-event-trigger.md
Service: data-factory
GitHub Login: @douglaslMS
Microsoft Alias: douglasl

@franelj
Thanks for the feedback! We are currently investigating and will update you shortly.

@franelj no, this is not a supported feature at this time, but please leave your feedback here:
https://feedback.azure.com/forums/270578-data-factory

Hello jason,
whether this feature is available now ?
Thank you
Dyanesh