close connections

Hello,
in the moment we testing your really nice package on our application but we had a problem with the open connections to the elasticserver. After benchmarks test or in our testing environment we had a lot open (zombie) connections with TIME_WAIT.
lsof -p say:
server  24086 root  276u  sock      0,7      0t0 63140932 can't identify protocol
netstat -an -p tcp
tcp        0      0 XXXXX:59824    XXX:9200         TIME_WAIT   -
Our connection func is:
func NewElasticSession() *elastic.Client {
httpClient := &http.Client{
    Transport: &BasicAuthTransport{
        username:  ReadIniString("elastic user"),
        password:  ReadIniString("elastic pass"),
    },
}

    elastic.SetTraceLog(log.New(os.Stdout, "", log.LstdFlags))
    elastic.SetInfoLog(log.New(os.Stdout, "", log.LstdFlags))
    client, err := elastic.NewClient(
            elastic.SetHttpClient(httpClient),
            elastic.SetURL(elasticServer),
            elastic.SetSniff(true),
    elastic.SetHealthcheckInterval(10*time.Second),
    elastic.SetMaxRetries(5),
    elastic.SetErrorLog(log.New(os.Stderr, "ELASTIC ", log.LstdFlags)),
    elastic.SetInfoLog(log.New(os.Stdout, "", log.LstdFlags)))
if err != nil {
    panic(err)
}

exists, err := client.IndexExists(index).Do()
if err != nil {
    panic(err)
}
if !exists {
    _, err := client.CreateIndex(index).Do()
    if err != nil {
        panic(err)
    }
}

return client

}

Hi there! Hmm, I've been using Elastic on server processes for months and I've never come up with that. But your method name got my curious... Do you perhaps create one connection per session? You should really create just one (or a handful) client per app and share it.

Yes, that´s correct. I will try to set one connection on startup and shared ist with the functions

nope. Same result. Tested on three different servers. Do you have a example for how do use the client -> server call?

This should do the trick.

Thank you olivere, looks better.;-)

hey, after testing same problem again. Sorry... We are testing with screaming frog and ab
If there a option to close the connection manually from your package?
 This seems odd. To be honest I'm using Elastic in production for over a year now and haven't seen this, so I'm not sure what causes this.
I know this typically happens when you open a HTTP connection, read the body but forgot to close it e.g. with defer resp.Body.Close(). I'm not sure if closing connections manually is the root cause of the bug; it may hide a problem, but my experience is that they typically come back and hit you even harder.
Can you extract a small test program that illustrates the behaviour? Also: What OS (uname -a), Go (go version), and Elasticsearch (curl <IP_ADDRESS>:9200) version are you using? Which version of Elastic are you using (and what's the import path)?

3.18.0-13-generic #14-Ubuntu SMP Fri Feb 6 09:55:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
go version go1.3.3 linux/amd64
Elasticsearch 1.7.0
For the go application we use gorilla as framwork. I show what a part i can extract and send you this code.

An example of your code would be perfect.
In the meantime can you check if a simple infinite loop like this increases the connection count? It doesn't do so here.

Just for reference: There was an issue with not closing the response body on errors with Go 1.3. But it should have been fixed. Any reason not to upgrade to 1.4 (and 1.5 is just around the corner)?

no, the ist our test environment. on the production server we use at the moment 1.4.2.
The interesting part is with low traffic we had no problems with the close of the connections, its only a problem high traffic (more then 10-20 req/sec)

is fixed, fault on my side

baxterbln, I have a similar problem. How did you solve it? Thanks!

@baxterbln how did you solve it?

@lelvisl Do you perhaps create one client per request? With elastic.NewClient, you should only create one client per application. It's thread-safe and you can safely use one client with multiple goroutines.

@olivere I already solve this problem by custom http client
	cli, err = elastic.NewSimpleClient(
		elastic.SetURL(c.Elastic.Url),
		elastic.SetHttpClient(&http.Client{Transport: &http.Transport{
			MaxIdleConns:       10,
			IdleConnTimeout:    30 * time.Second,
			DisableCompression: true,
			Dial: (&net.Dialer{
				Timeout:   30 * time.Second,
				KeepAlive: 30 * time.Second,
			}).Dial,
		}}),
	)

solve this problem by disableKeepAlives
		client, err := elastic.NewClient( 			elastic.SetURL(elasticHost), 			elastic.SetBasicAuth(elasticUser, elasticPass), 			elastic.SetSniff(false), 			elastic.SetHttpClient(&http.Client{Transport: &http.Transport{ 				MaxIdleConns:       10, 				IdleConnTimeout:    30 * time.Second, 				DisableCompression: true, 				DisableKeepAlives: true, 				DialContext: (&net.Dialer{ 					Timeout:   30 * time.Second, 					KeepAlive: 30 * time.Second, 				}).DialContext, 			}}))

So in our load test case, we start 1 server pod and send 300 concurrency requests(total 100000 requests) using apache bench. And TIME_WAIT socket count keep increasing. I think the solution is just increasing MaxIdleConnsPerHost(default 2) to match your concurrency level per pod. This is how Go http works link.


ls -l

ls -l does not work in PowerShell, and ls will probably give what we want anyway.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 8fd398c1-ed8b-911b-f11a-3739d80b3f32
Version Independent ID: dc3d55ff-3b55-a686-81f9-f4188054e381
Content: Restore a SQL Server database in Docker - SQL Server
Content Source: docs/linux/tutorial-restore-backup-in-sql-server-container.md
Product: sql
Technology: linux
GitHub Login: @VanMSFT
Microsoft Alias: vanto

Thanks @brettrowberry!
Hey @vin-yu - Can you confirm this?

Hello @brettrowberry! Thanks for your patience! I have fixed the typo in the doc, and it should be updated later today. We appreciate you bringing this to our attention!


Enable CephFS encryption for CephFS subvolumes

Describe the feature you'd like to have
CephFS make use of fscrypt based encryption or it will be available soon.
Ref# https://lwn.net/Articles/829448/
We have to think about the possibilities of integration for the subvolumes we provision and manage  from CSI.
CSI requirements can be summarized to below:
*) Each PVC is mapped to a subvolume in the backend
*) Each PVC has to be encrypted with a different key
     I believe in the `fscrypt` case, we will have a custom protector for “CSI” which is common for all the PVCs but will have a “seperate” protector key for each subvolume

*) The keys will be stored in a KMS engine and retrieved at time of mounting the volume to a workload/POD
*) The key retrieved from the KMS engine will be used for encrypting and decrypting the volume

*) Operations like “expansion”, “snapshot” , “Cloning to a new subvolume” on an existing PVC ..etc has to work without hiccups.
*) Once the snapshot or clone is in place the parent/source volume can be deleted at any time. IOW, no dependency exist to the source/original volume used for snapshot/clone .

I would like to capture the thoughts or would like to discuss on  fscrypt+ cephfs  implmentation against satisfying above requirements from Ceph CSI side.
@batrick @dillaman @vasyl-purchel @ShyamsundarR @johnStrunk

Thanks Madhu for the prompt reply and reopening this issue.
We are at a point of making critical design choices for our application. We will be going with CepfFS hoping that KMS encryption support will be added soon to the CephFS provisioning and storageclass.
regards

@Madhu-1 @jtlayton - What is the bug number for this issue on tracker.ceph.com
thanks

https://tracker.ceph.com/issues/46690 is the one, lets get confirmation from Jeff.

Hello!
I looked into how CephFS and FSCrypt could be integrated a bit and believe much of what is already there for RBD could be made to work for CephFS+FSCrypt as well. Biggest exception would be DEK storage.
I'd love to get started on a patch and like to hear if there is already something planned?
What I have in mind for a first version would be:

Enable FSCrypt always on the volume root level.
Focus on the 'metadata' KSM. Use Xattrs as DEK storage
Either leverage https://github.com/google/fscrypt or https://github.com/google/fscryptctl to set keys and policies
Where encryption is set up during CreateVolume, do this in NodeStateVolume to allow easy access to Xattrs

Hello!
I looked into how CephFS and FSCrypt could be integrated a bit and believe much of what is already there for RBD could be made to work for CephFS+FSCrypt as well. Biggest exception would be DEK storage.
I'd love to get started on a patch and like to hear if there is already something planned?
What I have in mind for a first version would be:
* Enable FSCrypt always on the volume root level.

* Focus on the 'metadata' KSM. Use Xattrs as DEK storage

* Either leverage https://github.com/google/fscrypt or https://github.com/google/fscryptctl to set keys and policies

* Where encryption is set up during CreateVolume, do this in NodeStateVolume to allow easy access to Xattrs

@irq0 That would be awesome :+1: , please give a try, we can also target this feature's first version in release 3.6. :+1:
Please let us know if you need any help on this.

Hello!
I looked into how CephFS and FSCrypt could be integrated a bit and believe much of what is already there for RBD could be made to work for CephFS+FSCrypt as well. Biggest exception would be DEK storage.
I'd love to get started on a patch and like to hear if there is already something planned?

@irq0 That would be awesome +1 , please give a try, we can also target this feature's first version in release 3.6. +1

My WIP branch is here: https://github.com/irq0/ceph-csi/commits/wip/fscrypt. Needs k8s config from my dev env repo https://github.com/irq0/dev-ceph-csi-fscrypt-config plus custom kernel and MDS as the feature isn't merged yet.
It is currently more or less a PoC. The main feature, unlocking volumes, is there and stores encrypted DEKs in an xattr.
It shells out to fscryptctl quite a bit. I'll polish it up and replace with the fscrypt go library next. I'll open a PR once that is done.

What I have in mind for a first version would be:
* Enable FSCrypt always on the volume root level.

Do you mean, subvolume or subvolumegroup in this context ?

Subvolume / k8s PV

* Focus on the 'metadata' KSM. Use Xattrs as DEK storage

That looks reasonable.

I'm not sure if there is a good alternative. I don't think there is something akin to the RBD volume metadata for subvolumes, or did I miss anything?

* Either leverage https://github.com/google/fscrypt or https://github.com/google/fscryptctl to set keys and policies

I would prefer fscrypt library than cli though or was exploring on that more than CLI.

Me too. I started with fscryptctl though, because it is way simpler to use.

* Where encryption is set up during CreateVolume, do this in NodeStateVolume to allow easy access to Xattrs

Yep.
Also, please feel free to have a design doc along with first POC implementation, that helps.
Please let us know if you need any help on this.

Thanks! I'm a bit unsure on how to best test this, as most of the functionality crosses the system border towards the OS. Any ideas?

Thanks a lot @irq0 for the revert and sharing the gist of what you have been experimenting. We got to discuss about this in 3.6 release triage call yesterday and few thoughts on how to proceed further.  It would be good to start with a design doc of this feature and get opinions/suggestions about the approaches from CSI and Ceph Team . It was also told that, we may have to wait for CephFS merge and kernel releases to make sure we are taking right approach towards whats going to land finally as a cephfs encryption solution. With all that, would it be possible to propose a design doc, so that we can discuss further and validate the possibilities of this feature in 3.6.
@nixpanic anything else to add which I missed here?

Will do. Should be ready mid next week.
I'll also want to explore integration possibilities with https://github.com/google/fscrypt some more. google/fscrypt and Ceph-CSI share quite a bit functionality around key management and there is probably some merit to making them compatible somehow. (The PoC code I wrote would require Ceph-CSI key management to unlock; google/fscrypt could allow multiple ways to unlock (e.g PAM, passwords, Ceph-CSI))
There aren't actually any hard CephFS dependencies here, since all the integration would do is key management. The nice thing about fscrypt is that this is independent of the filesystem below (see https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html). (So, what Ceph-CSI would do to unlock fscrypt on CephFS would work on ext4 as well)

Considering we have to wait for the dependency support in place and also some discussions on the design has to be concluded, removing this from the release 3.6 tracker.  Thanks for the great attempt to take it forward.. we will continue the effort and will try to get this in release 3.7..

@irq0 Thanks for all the meetings or disussions on this topic and continuously making progress on the same.  We will continue reviewing the new PR in place https://github.com/ceph/ceph-csi/pull/3310 and track this for upcoming Ceph CSI release. I am untracking this feature from release 3.7 . :+1:


Update for python3 and little more

Hi Eric,
I re-used your code for experimenting with LSTMs and Kaparthy's initial idea.
In my context I shall use python3 so I switch your code to python3 (which was a straigh-forward 2to3).
Then I added some more data inputs and small changes.
Feel free to take those changes if you are interested.

Thanks much for the contributions & much-needed update to Python 3!


SMP: Split Analysis of Expected Delta Utility from Logging

A new procedure calcUtils to call probEduChlg inside a triple loop. It computes & logs all combos of h,k,i,j except for what are needed for bestChallenge. This procedure would run in a separate thread.
A new procedure bestChallengeUtils to call probEduChlg for only those h,k,i,j combos needed to select the challenger. The results would get stored in a memory map as well. This procedure would run synchronously so that the data is ready for consumption by bestChallenge method.
Modified bestChallenge doesn't call probEduChlg. Instead, it takes data from the memory map used in bestChallengeUtils to find the best challenger.

Amit - it failed the build test, so I cannot approve it.
Also, you may notice comments in the code of SMPState::doBCN that the main top-level loops will be converted to use KBase::groupThreads. Is your change consistent with this? Did you use groupThreads?
Finally, did you verify that none of the numbers in the output log change when your new version of SMPC is run over SOE-Pol-Comp.csv and dummy3Dim.csv?


Removed Operations API references from two SQL swagger files

All 3 SQL swagger files currently reference the same Operations API, which includes the operations from all 3 files.
Updating to only reference this in one place to avoid generating 3 references to this in client SDK.
PR information

[ ] The title of the PR is clear and informative.
[ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
[ ] Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
[ ] If applicable, the PR references the bug/issue that it fixes.
[ ] Swagger files are correctly named (e.g. the api-version in the path should match the api-version in the spec).

Quality of Swagger

[ ] I have read the contribution guidelines.
[ ] My spec meets the review criteria:

[ ] The spec conforms to the Swagger 2.0 specification.
[ ] Validation errors from the Linter extension for VS Code have all been fixed for this spec. (Note: for large, previously checked in specs, there will likely be many errors shown. Please contact our team so we can set a timeframe for fixing these errors if your PR is not going to address them).
[ ] The spec follows the patterns described in the Swagger good patterns document unless the service API makes this impossible.

The commit history is looking weird due to your merge from azure/master to nathannfan/master_examples. Let's look at how to fix this together.

No modification for NodeJS

 https://github.com/Azure/azure-sdk-for-python/pull/1009

 https://github.com/Azure/azure-sdk-for-ruby/pull/699


POP windows shwoing up with warning

ADD ISSUE DESCRIPTION HERE
Version: 1.74.3
Commit: 97dec172d3256f8ca4bfb2143f3f76b503ca0534
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0
Embedder: codespaces

/needsMoreInfo

Thanks for creating this issue! We figured it's missing some basic information or in some other way doesn't follow our issue reporting guidelines. Please take the time to review these and update the issue.
Happy Coding!

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


useAnimateGroup doesn't work as expected after the second time the play function gets called

Describe the bug
useAnimateGroup's play function only works for the first two times it's called. After that the value of isPlaying doesn't get updated correctly and remains true no matter what is passed to play
To Reproduce
You can test this behavior in the demo provided in the docs for this hook : https://react-simple-animate.vercel.app/hooks
Expected behavior
Calling the play function should update the isPlaying state correctly
Screenshots

Additional context
the cause of this issue is how the play function is internally defined.
playRef.current = playRef.current
    ? playRef.current
    : (isPlay: boolean) => {
        ...
        setStyles(isPlay ? styles : [...styles].reverse());
        setPlaying(!isPlaying);
      };

As you can see, the function assigned to playRef.current is defined only on the first render and creates a closure on the value of isPlaying, which is initially false. Then, when we call play(!isPlaying), isPlaying gets set to true. But after the first render, because playRef.current keeps its old reference which has closed on the initial value of isPlaying, being false, calling play would always call setPlaying(true).
The solution would be to use the parameter isPlay and replace setPlaying(!isPlaying) with setPlaying(isPlay)
Also there is no test for this use case.
If you agree with the diagnosis, I'd be happy to create a PR

Send us a PR, that would be lovely.