UNKNOWN s3 error (100) in transfer manager as of Release 1.4.46

As of release 1.4.46 I am getting random UNKNOWN errors when transferring a number of files through TransferManager.
The exact same test application has no issues with earlier SDK releases.
Call stack:

aws-cpp-sdk-core.dll!GuessBodylessErrorType(Aws::Http::HttpResponseCode responseCode) Line 71	C++	Symbols loaded.
aws-cpp-sdk-core.dll!Aws::Client::AWSXMLClient::BuildAWSError(const std::shared_ptrAws::Http::HttpResponse & httpResponse) Line 743	C++	Symbols loaded.
aws-cpp-sdk-core.dll!Aws::Client::AWSClient::AttemptOneRequest(const Aws::Http::URI & uri, const Aws::AmazonWebServiceRequest & request, Aws::Http::HttpMethod method, const char * signerName) Line 298	C++	Symbols loaded.
aws-cpp-sdk-core.dll!Aws::Client::AWSClient::AttemptExhaustively(const Aws::Http::URI & uri, const Aws::AmazonWebServiceRequest & request, Aws::Http::HttpMethod method, const char * signerName) Line 174	C++	Symbols loaded.
aws-cpp-sdk-core.dll!Aws::Client::AWSXMLClient::MakeRequest(const Aws::Http::URI & uri, const Aws::AmazonWebServiceRequest & request, Aws::Http::HttpMethod method, const char * signerName) Line 687	C++	Symbols loaded.
aws-cpp-sdk-s3.dll!Aws::S3::S3Client::UploadPart(const Aws::S3::Model::UploadPartRequest & request) Line 2684	C++	Symbols loaded.
aws-cpp-sdk-s3.dll!Aws::S3::S3Client::UploadPartAsyncHelper(const Aws::S3::Model::UploadPartRequest & request, const std::function<void __cdecl(Aws::S3::S3Client const *,Aws::S3::Model::UploadPartRequest const &,Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult,Aws::Client::AWSError > const &,std::shared_ptr<Aws::Client::AsyncCallerContext const > const &)> & handler, const std::shared_ptr<Aws::Client::AsyncCallerContext const > & context) Line 2710	C++	Symbols loaded.
aws-cpp-sdk-s3.dll!Aws::S3::S3Client::UploadPartAsync::__l2::() Line 2705	C++	Symbols loaded.
aws-cpp-sdk-s3.dll!std::_Invoker_functor::_Call<void (void) & __ptr64>(Aws::S3::S3Client::UploadPartAsync::__l2::void (void) & _Obj)	C++	Non-user code. Symbols loaded.
aws-cpp-sdk-s3.dll!std::invoke<void (void) & __ptr64>(Aws::S3::S3Client::UploadPartAsync::__l2::void (void) & _Obj)	C++	Non-user code. Symbols loaded.
aws-cpp-sdk-s3.dll!std::_Invoker_retstd::_Unforced,0::_Call<void (void) & __ptr64>(Aws::S3::S3Client::UploadPartAsync::__l2::void (void) & <_Vals_0>)	C++	Non-user code. Symbols loaded.
aws-cpp-sdk-s3.dll!std::_Call_binder<std::_Unforced,void (void),std::tuple<>,std::tuple<> >(std::_Invoker_retstd::_Unforced,0 __formal, std::integer_sequence __formal, Aws::S3::S3Client::UploadPartAsync::__l2::void (void) & _Obj, std::tuple<> & _Tpl, std::tuple<> && _Ut) Line 1030	C++	Non-user code. Symbols loaded.
aws-cpp-sdk-s3.dll!std::_Binder<std::_Unforced,void (void) >::operator()<>() Line 1085	C++	Non-user code. Symbols loaded.
aws-cpp-sdk-s3.dll!std::_Invoker_functor::_Call<std::_Binder<std::_Unforced,void (void) > & __ptr64>(std::_Binder<std::_Unforced,void (void) > & _Obj)	C++	Non-user code. Symbols loaded.
aws-cpp-sdk-s3.dll!std::invoke<std::_Binder<std::_Unforced,void (void) > & __ptr64>(std::_Binder<std::_Unforced,void (void) > & _Obj)	C++	Non-user code. Symbols loaded.
aws-cpp-sdk-s3.dll!std::_Invoker_ret<void,1>::_Call<std::_Binder<std::_Unforced,void (void) > & __ptr64>(std::_Binder<std::_Unforced,void (void) > & <_Vals_0>)	C++	Non-user code. Symbols loaded.
aws-cpp-sdk-s3.dll!std::_Func_impl_no_alloc<std::_Binder<std::_Unforced,void (void) >,void>::_Do_call()	C++	Non-user code. Symbols loaded.
aws-cpp-sdk-core.dll!std::_Func_class::operator()()	C++	Non-user code. Symbols loaded.
aws-cpp-sdk-core.dll!Aws::Utils::Threading::DefaultExecutor::SubmitToThread::__l2::() Line 29	C++	Symbols loaded.

What platform/OS are you using? Windows 8/10
What compiler are you using? what version? Visual Studio 15 2017 Win64
What's your CMake arguments?
Can you provide a TRACE level log? (sanitize any sensitive information) yes

It's caused by our changes in core when making http request. Before 1.4.46, there is a high change that  http response could be null, so you might get different error as "can't connect to endpoint". Taking a look at #905

Hi Andrew!
Thank you for your response.
The problem is we don't get any errors at all using the previous releases.
I am running the exact same test application and there are no issues with
it until I upgrade the sdk.
So, I don't think the error is valid. It is always random, happening on a
different file every time.
If you suggest this is expected behaviour, then please advise how to avoid
this error.
Thank you,
Anna
On Mon, Jul 16, 2018 at 1:10 PM, Andrew Tang<EMAIL_ADDRESS>wrote:

It's caused by our changes in core when making http request. Before
1.4.46, there is a high change that http response could be null, so you
might get different error as "can't connect to endpoint". Taking a look at
#905 https://github.com/aws/aws-sdk-cpp/issues/905
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-405318337,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AnQi-54d0S5BjfouVs7B6NXMhAP1itAlks5uHMjrgaJpZM4VNh2v
.[image: Web Bug from
https://github.com/notifications/beacon/AnQi-woP95rfru4c-nVGi5FPIOsrX7EKks5uHMjrgaJpZM4VNh2v.gif]
{"api_version":"1.0","publisher":{"api_key":"
05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":
{"external_key":"github/aws/aws-sdk-cpp","title":"aws/aws-sdk-cpp","subtitle":"GitHub
repository","main_image_url":"https://assets-cdn.github.com/
images/email/message_cards/header.png","avatar_image_url":"
https://assets-cdn.github.com/images/email/message_cards/avatar.png
","action":{"name":"Open in GitHub","url":"https://github.
com/aws/aws-sdk-cpp"}},"updates":{"snippets":[{"icon":"PERSON","message":"@singku
in #910: It's caused by our changes in core when making http request.
Before 1.4.46, there is a high change that http response could be null, so
you might get different error as "can't connect to endpoint". Taking a
look at #905 "}],"action":{"name":"View Issue","url":"https://github.
com/aws/aws-sdk-cpp/issues/910#issuecomment-405318337"}}} [ { "@context":
"http://schema.org", "@type": "EmailMessage", "potentialAction": {
"@type": "ViewAction", "target": "https://github.com/aws/aws-
sdk-cpp/issues/910#issuecomment-405318337", "url": "
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-405318337",
"name": "View Issue" }, "description": "View this Issue on GitHub",
"publisher": { "@type": "Organization", "name": "GitHub", "url": "
https://github.com" } }, { "@type": "MessageCard", "@context": "
http://schema.org/extensions", "hideOriginalBody": "false", "originator":
"AF6C5A86-E920-430C-9C59-A73278B5EFEB", "title": "Re: [aws/aws-sdk-cpp]
UNKNOWN s3 error (100) in transfer manager as of Release 1.4.46- BUG
(#910)", "sections": [ { "text": "", "activityTitle": "Andrew Tang",
"activityImage": "https://assets-cdn.github.com/images/email/message_
cards/avatar.png", "activitySubtitle": "@singku", "facts": [ ] } ],
"potentialAction": [ { "name": "Add a comment", "@type": "ActionCard",
"inputs": [ { "isMultiLine": true, "@type": "TextInput", "id":
"IssueComment", "isRequired": false } ], "actions": [ { "name": "Comment",
"@type": "HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueComment",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910,\n"IssueComment":
"{{IssueComment.value}}"\n}" } ] }, { "targets": [ { "os": "default",
"uri": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-405318337" } ], "@type": "OpenUri", "name": "View on GitHub"
}, { "name": "Unsubscribe", "@type": "HttpPOST", "target": "
https://api.github.com", "body": "{\n"commandName":
"MuteNotification",\n"threadId": 355868079\n}" } ], "themeColor":
"26292E" } ]
Click here
http://esva5.caseware.com/cgi-bin/learn-msg.cgi?id=7F36328F73.ACB45 to
report this message as spam to Caseware E-Mail Security Server ESVA.

--
Anna Siniavina
Working Papers Developer
T: 416·867·9504 x1210 | F: 416·867·1906
<EMAIL_ADDRESS>https://www.caseware.com/
https://linkedin.com/company/caseware-international-inc
https://facebook.com/CaseWare-International-Inc-54418196590/
https://twitter.com/caseware
https://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg

Yes, there is an issue. The files in question never really get uploaded, even after a few retries. And the biggest problem is that the transfer manager receives TransferStatus::COMPLETED on the transferStatusUpdatedCallback every time it happens, so my test client now receives multiple completed hits for the same file, and none of them ever succeeds. Again, if this is a valid scenario, then the Transfer Manager should come back with a different status.
Thank you,
Anna

@pavian Right now our transfer tests against S3 never fails. Is that possible for you to show us the log file with failed S3 upload? Or even better, the code sample so that we can reproduce it.

I could e-mail you my whole test application in a private message.
Also, can you advise where to find SDK logs so I could send you the log?
Thanks
Anna
On Tue, Jul 17, 2018 at 12:40 PM, Andrew Tang<EMAIL_ADDRESS>wrote:

@pavian https://github.com/pavian Right now our transfer tests against
S3 never fails. Is that possible for you to show us the log file with
failed S3 upload? Or even better, the code sample so that we can reproduce
it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-405647837,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AnQi-8lCsJpKHn93R9FbTec5BHBpMVyaks5uHhNmgaJpZM4VNh2v
.[image: Web Bug from
https://github.com/notifications/beacon/AnQi-0T-fSk0lwsB7r07UkK5IK1rDox_ks5uHhNmgaJpZM4VNh2v.gif]
{"api_version":"1.0","publisher":{"api_key":"
05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":
{"external_key":"github/aws/aws-sdk-cpp","title":"aws/aws-sdk-cpp","subtitle":"GitHub
repository","main_image_url":"https://assets-cdn.github.com/
images/email/message_cards/header.png","avatar_image_url":"
https://assets-cdn.github.com/images/email/message_cards/avatar.png
","action":{"name":"Open in GitHub","url":"https://github.
com/aws/aws-sdk-cpp"}},"updates":{"snippets":[{"icon":"PERSON","message":"@singku
in #910: @pavian Right now our transfer tests against S3 never fails. Is
that possible for you to show us the log file with failed S3 upload? Or
even better, the code sample so that we can reproduce it.
"}],"action":{"name":"View Issue","url":"https://github.
com/aws/aws-sdk-cpp/issues/910#issuecomment-405647837"}}} [ { "@context":
"http://schema.org", "@type": "EmailMessage", "potentialAction": {
"@type": "ViewAction", "target": "https://github.com/aws/aws-
sdk-cpp/issues/910#issuecomment-405647837", "url": "
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-405647837",
"name": "View Issue" }, "description": "View this Issue on GitHub",
"publisher": { "@type": "Organization", "name": "GitHub", "url": "
https://github.com" } }, { "@type": "MessageCard", "@context": "
http://schema.org/extensions", "hideOriginalBody": "false", "originator":
"AF6C5A86-E920-430C-9C59-A73278B5EFEB", "title": "Re: [aws/aws-sdk-cpp]
UNKNOWN s3 error (100) in transfer manager as of Release 1.4.46- BUG
(#910)", "sections": [ { "text": "", "activityTitle": "Andrew Tang",
"activityImage": "https://assets-cdn.github.com/images/email/message_
cards/avatar.png", "activitySubtitle": "@singku", "facts": [ ] } ],
"potentialAction": [ { "name": "Add a comment", "@type": "ActionCard",
"inputs": [ { "isMultiLine": true, "@type": "TextInput", "id":
"IssueComment", "isRequired": false } ], "actions": [ { "name": "Comment",
"@type": "HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueComment",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910,\n"IssueComment":
"{{IssueComment.value}}"\n}" } ] }, { "name": "Close issue", "@type":
"HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueClose",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910\n}" }, { "targets": [ { "os":
"default", "uri": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-405647837" } ], "@type": "OpenUri", "name": "View on GitHub"
}, { "name": "Unsubscribe", "@type": "HttpPOST", "target": "
https://api.github.com", "body": "{\n"commandName":
"MuteNotification",\n"threadId": 355868079\n}" } ], "themeColor":
"26292E" } ]
Click here
http://esva5.caseware.com/cgi-bin/learn-msg.cgi?id=A0C6A27DCF.ADA3E to
report this message as spam to Caseware E-Mail Security Server ESVA.

--
Anna Siniavina
Working Papers Developer
T: 416·867·9504 x1210 | F: 416·867·1906
<EMAIL_ADDRESS>https://www.caseware.com/
https://linkedin.com/company/caseware-international-inc
https://facebook.com/CaseWare-International-Inc-54418196590/
https://twitter.com/caseware
https://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg

It should be at the same directory with the binary, named as aws_sdk_xxxx_xx_xx_xx.log.
you should call
    Aws::SDKOptions options;
    options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;

before call Aws::InitAPI().
My personal email is<EMAIL_ADDRESS>

Hi @pavian I took a look at both of the logs you sent to me.
In your own log, line 35:
 : 7ae1d65c-4414-4adf-badb-d1f288b24c67: UPLOAD FAILED: 4 Error code: 100 Error: No response body. Response code:<PHONE_NUMBER>

It actually says upload failed and the error code from SDK is 100, which is UNKNOWN. So I don't know why you said:

And the biggest problem is that the transfer manager receives TransferStatus::COMPLETED

For the failed reason inside SDK, take look at SDK log, line 2611:
[WARN] 2018-07-18 07:54:44 WinHttpSyncHttpClient [23976] Send request failed: The connection with the server was terminated abnormally

There is a connection error between your host and remote aws server. So that the request was never made to wire, so you got the -1<PHONE_NUMBER>) RESPONSE_NOT_MADE http response code.
BTW, these two log line are related because I searched the s3 key 7ae1d65c-4414-4adf-badb-d1f288b24c67 in sdk's log and the related thread id who is processing this key is 23976.
Anyway based on this, I think SDK do express the error correctly to application level. You could debug if there is any thing missing in your code. Please let me know if you have any further question.

Hi Andrew,
Can you explain why these connection issues are happening so much?
I believe that with an older sdk there might have been similar connection
issues that were suppressed and hidden for me. What I care about is the end
result. In the previous versions I was getting a successful upload at some
point in time. Now I never get the failed file to complete. Is it supposed
to retry until succeeded? I've tried waiting for that but it never
happened. All I care about is how to get to the successful the end result
Also, if I do get FAILED status from TM in this case, how do I know that it
is still retrying on the background and that I should continue waiting for
it to finish? All the other failed cases are sort of final and I know the
file will not be processed. In this particular case the TM will actually
continue working with this file, right? I need to know that somehow. In
fact, with the latest versions of the sdk I get multiple TM callback hits
for the same file. All seem to be related to different part numbers though.
But again, it never actually completes.
Please advise what I should do to see the file successfully uploaded in
this new SDK world?
Thank you!
Anna
On Wed, Jul 18, 2018 at 7:09 PM, Andrew Tang<EMAIL_ADDRESS>wrote:

Hi @pavian https://github.com/pavian I took a look at both of the logs
you sent to me.
In your own log, line 35:
: 7ae1d65c-4414-4adf-badb-d1f288b24c67: UPLOAD FAILED: 4 Error code: 100 Error: No response body. Response code:<PHONE_NUMBER>
It actually says upload failed and the error code from SDK is 100, which
is UNKNOWN. So I don't know why you said:
And the biggest problem is that the transfer manager receives
TransferStatus::COMPLETED
For the failed reason inside SDK, take look at SDK log, line 2611:
[WARN] 2018-07-18 07:54:44 WinHttpSyncHttpClient [23976] Send request failed: The connection with the server was terminated abnormally
There is a connection error between your host and remote aws server. So
that the request was never made to wire, so you got the -1<PHONE_NUMBER>)
RESPONSE_NOT_MADE http response code.
BTW, these two log line are related because I searched the s3 key
7ae1d65c-4414-4adf-badb-d1f288b24c67 in sdk's log and the related thread
id who is processing this key is 23976.
Anyway based on this, I think SDK do express the error correctly to
application level. You could debug if there is any thing missing in your
code. Please let me know if you have any further question.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-406101807,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AnQi-9qiHF9U5-1h_JXQVa-RPsm4aZxpks5uH8ATgaJpZM4VNh2v
.[image: Web Bug from
https://github.com/notifications/beacon/AnQi-5jlhAUAPyy-cDk38lJv6HGGHZU1ks5uH8ATgaJpZM4VNh2v.gif]
{"api_version":"1.0","publisher":{"api_key":"
05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":
{"external_key":"github/aws/aws-sdk-cpp","title":"aws/aws-sdk-cpp","subtitle":"GitHub
repository","main_image_url":"https://assets-cdn.github.com/
images/email/message_cards/header.png","avatar_image_url":"
https://assets-cdn.github.com/images/email/message_cards/avatar.png
","action":{"name":"Open in GitHub","url":"https://github.
com/aws/aws-sdk-cpp"}},"updates":{"snippets":[{"icon":"PERSON","message":"@singku
in #910: Hi @pavian I took a look at both of the logs you sent to me.\r\nIn
your own log, line 35:\r\n\r\n : 7ae1d65c-4414-4adf-badb-d1f288b24c67: UPLOAD FAILED: 4 Error code: 100 Error: No response body. Response code:<PHONE_NUMBER>\r\n\r\nIt actually says upload failed and the error code from
SDK is 100, which is UNKNOWN. So I don't know why you
said:\r\n\u003eAnd the biggest problem is that the transfer manager
receives TransferStatus::COMPLETED\r\n\r\nFor the failed reason inside
SDK, take look at SDK log, line 2611:\r\n\r\n[WARN] 2018-07-18 07:54:44 WinHttpSyncHttpClient [23976] Send request failed: The connection with the server was terminated abnormally\r\n\r\n\r\nThere is a connection error
between your host and remote aws server. So that the request was never made
to wire, so you got the -1<PHONE_NUMBER>) RESPONSE_NOT_MADE http response
code.\r\n\r\nBTW, these two log line are related because I searched the s3
key 7ae1d65c-4414-4adf-badb-d1f288b24c67 in sdk's log and the related
thread id who is processing this key is 23976.\r\n\r\n\r\nAnyway based on
this, I think SDK do express the error correctly to application level. You
could debug if there is any thing missing in your code. Please let me know
if you have any further question.\r\n"}],"action":{"name":"View
Issue","url":"https://github.com/aws/aws-sdk-cpp/issues/
910#issuecomment-406101807"}}} [ { "@context": "http://schema.org",
"@type": "EmailMessage", "potentialAction": { "@type": "ViewAction",
"target": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-406101807", "url": "https://github.com/aws/aws-
sdk-cpp/issues/910#issuecomment-406101807", "name": "View Issue" },
"description": "View this Issue on GitHub", "publisher": { "@type":
"Organization", "name": "GitHub", "url": "https://github.com" } }, {
"@type": "MessageCard", "@context": "http://schema.org/extensions",
"hideOriginalBody": "false", "originator": "AF6C5A86-E920-430C-9C59-A73278B5EFEB",
"title": "Re: [aws/aws-sdk-cpp] UNKNOWN s3 error (100) in transfer manager
as of Release 1.4.46- BUG (#910)", "sections": [ { "text": "",
"activityTitle": "Andrew Tang", "activityImage": "
https://assets-cdn.github.com/images/email/message_cards/avatar.png",
"activitySubtitle": "@singku", "facts": [ ] } ], "potentialAction": [ {
"name": "Add a comment", "@type": "ActionCard", "inputs": [ {
"isMultiLine": true, "@type": "TextInput", "id": "IssueComment",
"isRequired": false } ], "actions": [ { "name": "Comment", "@type":
"HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueComment",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910,\n"IssueComment":
"{{IssueComment.value}}"\n}" } ] }, { "name": "Close issue", "@type":
"HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueClose",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910\n}" }, { "targets": [ { "os":
"default", "uri": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-406101807" } ], "@type": "OpenUri", "name": "View on GitHub"
}, { "name": "Unsubscribe", "@type": "HttpPOST", "target": "
https://api.github.com", "body": "{\n"commandName":
"MuteNotification",\n"threadId": 355868079\n}" } ], "themeColor":
"26292E" } ]
Click here
http://esva4.caseware.com/cgi-bin/learn-msg.cgi?id=E0ED027E09.AE5CE to
report this message as spam to Caseware E-Mail Security Server ESVA.

--
Anna Siniavina
Working Papers Developer
T: 416·867·9504 x1210 | F: 416·867·1906
<EMAIL_ADDRESS>https://www.caseware.com/
https://linkedin.com/company/caseware-international-inc
https://facebook.com/CaseWare-International-Inc-54418196590/
https://twitter.com/caseware
https://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg

Hi Anna, I am taking a look at how didn't it retried. At the same time could you please attach the sample project so as to for me to reproduce it?

It turns out that SDK would retry when there is connection error before 1.4.46, but will not after 1.4.46.
Details:

Before 1.4.46, when connection error happens, response will be nullptr, then BuildAWSError will generate retry-able error code.
After 1.4.46, when connection error happens, when generating error, retry-able or not is based on the Http response code, since the request is never made , the Http response code is REQUEST_NOT_MADE, and this code is not in the retry-able code list.

I think it's doable to add REQUST_NOT_MADE to the list because all other http codes in that list are related to connection error but we will need to review it first. Will update to you later.

Hi Andrew?
Any update on this issue?
I see a lot of important changes in the transfer Manager code after that
release and I really wanted to test the latest SDK for performance but I
can't do that until this problem is fixed.
Also, since you already have our sample project, I was wondering if you
could answer one more question for me. We are trying to play with a couple
of parameters to get the best performance (number of execution threads,
http connections, buffer size) while not using up the whole 100% of the
CPU. When compared to CLI we now seem to get a better performance on
uploads, but not on the downloads. Is there any other parameters you can
recommend we could play with to achieve the best download performance and
to avoid overloading the CPU?
Thank you very much,
Anna
On Mon, Jul 23, 2018 at 5:44 PM, Andrew Tang<EMAIL_ADDRESS>wrote:

It turns out that SDK would retry when there is connection error before
1.4.46, but will not after 1.4.46.
Details:
Before 1.4.46, when connection error happens, response will be nullptr,
then BuildAWSError will generate retry-able error code.
After 1.4.46, when connection error happens, when generating error,
retry-able or not is based on the Http response code, since the request is
never made , the Http response code is REQUEST_NOT_MADE, and this code is
not in the retry-able code list.
I think it's doable to add REQUST_NOT_MADE to the list because all other
http codes in that list are related to connection error but we will need to
review it first. Will update to you later.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-407211165,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AnQi-_Ir1na-3tOuWs1fScw5uNvgfyFJks5uJkPRgaJpZM4VNh2v
.[image: Web Bug from
https://github.com/notifications/beacon/AnQi-44wIwDkbPzINlYDSXfQmnM2CBH5ks5uJkPRgaJpZM4VNh2v.gif]
{"api_version":"1.0","publisher":{"api_key":"
05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":
{"external_key":"github/aws/aws-sdk-cpp","title":"aws/aws-sdk-cpp","subtitle":"GitHub
repository","main_image_url":"https://assets-cdn.github.com/
images/email/message_cards/header.png","avatar_image_url":"
https://assets-cdn.github.com/images/email/message_cards/avatar.png
","action":{"name":"Open in GitHub","url":"https://github.
com/aws/aws-sdk-cpp"}},"updates":{"snippets":[{"icon":"PERSON","message":"@singku
in #910: It turns out that SDK would retry when there is connection error
before 1.4.46, but will not after 1.4.46.\r\nDetails:\r\n\u003eBefore
1.4.46, when connection error happens, response will be nullptr, then
BuildAWSError will generate retry-able error code.\r\n\u003eAfter 1.4.46,
when connection error happens, when generating error, retry-able or not is
based on the Http response code, since the request is never made , the Http
response code is REQUEST_NOT_MADE, and this code is not in the retry-able
code list. \r\n\r\nI think it's doable to add REQUST_NOT_MADE to the list
because all other http codes in that list are related to connection error
but we will need to review it first. Will update to you
later."}],"action":{"name":"View Issue","url":"https://github.
com/aws/aws-sdk-cpp/issues/910#issuecomment-407211165"}}} [ { "@context":
"http://schema.org", "@type": "EmailMessage", "potentialAction": {
"@type": "ViewAction", "target": "https://github.com/aws/aws-
sdk-cpp/issues/910#issuecomment-407211165", "url": "
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-407211165",
"name": "View Issue" }, "description": "View this Issue on GitHub",
"publisher": { "@type": "Organization", "name": "GitHub", "url": "
https://github.com" } }, { "@type": "MessageCard", "@context": "
http://schema.org/extensions", "hideOriginalBody": "false", "originator":
"AF6C5A86-E920-430C-9C59-A73278B5EFEB", "title": "Re: [aws/aws-sdk-cpp]
UNKNOWN s3 error (100) in transfer manager as of Release 1.4.46- BUG
(#910)", "sections": [ { "text": "", "activityTitle": "Andrew Tang",
"activityImage": "https://assets-cdn.github.com/images/email/message_
cards/avatar.png", "activitySubtitle": "@singku", "facts": [ ] } ],
"potentialAction": [ { "name": "Add a comment", "@type": "ActionCard",
"inputs": [ { "isMultiLine": true, "@type": "TextInput", "id":
"IssueComment", "isRequired": false } ], "actions": [ { "name": "Comment",
"@type": "HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueComment",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910,\n"IssueComment":
"{{IssueComment.value}}"\n}" } ] }, { "name": "Close issue", "@type":
"HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueClose",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910\n}" }, { "targets": [ { "os":
"default", "uri": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-407211165" } ], "@type": "OpenUri", "name": "View on GitHub"
}, { "name": "Unsubscribe", "@type": "HttpPOST", "target": "
https://api.github.com", "body": "{\n"commandName":
"MuteNotification",\n"threadId": 355868079\n}" } ], "themeColor":
"26292E" } ]
Click here
http://esva5.caseware.com/cgi-bin/learn-msg.cgi?id=EA81528867.ACF12 to
report this message as spam to Caseware E-Mail Security Server ESVA.

--
Anna Siniavina
Working Papers Developer
T: 416·867·9504 x1210 | F: 416·867·1906
<EMAIL_ADDRESS>https://www.caseware.com/
https://linkedin.com/company/caseware-international-inc
https://facebook.com/CaseWare-International-Inc-54418196590/
https://twitter.com/caseware
https://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg

Hi Anna, we have fixed the retry issue for connection problem for transferManager. You saw the performance bump because we also replaced Jsoncpp with cJson, which is faster on json deserialize. Not sure why you just observed efficiency bump on upload not download. We don't have a recommendation on threads used parallelly, it depends on the CPUs on your host. Another place you could play with is the buffer size set in transfer configuration. Take a look at the doc of https://github.com/aws/aws-sdk-cpp/blob/e40d691f239f068794c268ac3d75ff6b62ddb7db/aws-cpp-sdk-transfer/include/aws/transfer/TransferManager.h#L88

Thank you, Andrew!
Do yo know which SDK release contains the transferManager retry fix?
On Tue, Jul 31, 2018 at 4:09 PM, Andrew Tang<EMAIL_ADDRESS>wrote:

Hi Anna, we have fixed the retry issue for connection problem for
transferManager. You saw the performance bump because we also replaced
Jsoncpp with cJson, which is faster on json deserialize. Not sure why you
just observed efficiency bump on upload not download. We don't have a
recommendation on threads used parallelly, it depends on the CPUs on your
host. Another place you could play with is the buffer size set in transfer
configuration. Take a look at the doc of https://github.com/aws/aws-
sdk-cpp/blob/e40d691f239f068794c268ac3d75ff6b62ddb7db/aws-cpp-sdk-
transfer/include/aws/transfer/TransferManager.h#L88
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409351520,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AnQi-8vE6dTHD1ZG9p2kmMnTVeuPQmdyks5uMLmOgaJpZM4VNh2v
.[image: Web Bug from
https://github.com/notifications/beacon/AnQi-0xOPBxu4_awln0sC0VZvHj-fdTqks5uMLmOgaJpZM4VNh2v.gif]
{"api_version":"1.0","publisher":{"api_key":"
05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":
{"external_key":"github/aws/aws-sdk-cpp","title":"aws/aws-sdk-cpp","subtitle":"GitHub
repository","main_image_url":"https://assets-cdn.github.com/
images/email/message_cards/header.png","avatar_image_url":"
https://assets-cdn.github.com/images/email/message_cards/avatar.png
","action":{"name":"Open in GitHub","url":"https://github.
com/aws/aws-sdk-cpp"}},"updates":{"snippets":[{"icon":"PERSON","message":"@singku
in #910: Hi Anna, we have fixed the retry issue for connection problem for
transferManager. You saw the performance bump because we also replaced
Jsoncpp with cJson, which is faster on json deserialize. Not sure why you
just observed efficiency bump on upload not download. We don't have a
recommendation on threads used parallelly, it depends on the CPUs on your
host. Another place you could play with is the buffer size set in transfer
configuration. Take a look at the doc of https://github.com/aws/aws-
sdk-cpp/blob/e40d691f239f068794c268ac3d75ff6b62ddb7db/aws-cpp-sdk-
transfer/include/aws/transfer/TransferManager.h#L88"}],"action":{"name":"View
Issue","url":"https://github.com/aws/aws-sdk-cpp/issues/
910#issuecomment-409351520"}}} [ { "@context": "http://schema.org",
"@type": "EmailMessage", "potentialAction": { "@type": "ViewAction",
"target": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-409351520", "url": "https://github.com/aws/aws-
sdk-cpp/issues/910#issuecomment-409351520", "name": "View Issue" },
"description": "View this Issue on GitHub", "publisher": { "@type":
"Organization", "name": "GitHub", "url": "https://github.com" } }, {
"@type": "MessageCard", "@context": "http://schema.org/extensions",
"hideOriginalBody": "false", "originator": "AF6C5A86-E920-430C-9C59-A73278B5EFEB",
"title": "Re: [aws/aws-sdk-cpp] UNKNOWN s3 error (100) in transfer manager
as of Release 1.4.46- BUG (#910)", "sections": [ { "text": "",
"activityTitle": "Andrew Tang", "activityImage": "
https://assets-cdn.github.com/images/email/message_cards/avatar.png",
"activitySubtitle": "@singku", "facts": [ ] } ], "potentialAction": [ {
"name": "Add a comment", "@type": "ActionCard", "inputs": [ {
"isMultiLine": true, "@type": "TextInput", "id": "IssueComment",
"isRequired": false } ], "actions": [ { "name": "Comment", "@type":
"HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueComment",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910,\n"IssueComment":
"{{IssueComment.value}}"\n}" } ] }, { "name": "Close issue", "@type":
"HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueClose",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910\n}" }, { "targets": [ { "os":
"default", "uri": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-409351520" } ], "@type": "OpenUri", "name": "View on GitHub"
}, { "name": "Unsubscribe", "@type": "HttpPOST", "target": "
https://api.github.com", "body": "{\n"commandName":
"MuteNotification",\n"threadId": 355868079\n}" } ], "themeColor":
"26292E" } ]
Click here
http://esva5.caseware.com/cgi-bin/learn-msg.cgi?id=A202927DF2.A8B1A to
report this message as spam to Caseware E-Mail Security Server ESVA.

--
Anna Siniavina
Working Papers Developer
T: 416·867·9504 x1210 | F: 416·867·1906
<EMAIL_ADDRESS>https://www.caseware.com/
https://linkedin.com/company/caseware-international-inc
https://facebook.com/CaseWare-International-Inc-54418196590/
https://twitter.com/caseware
https://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg

96187c4
On Tue, Jul 31, 2018 at 10:17 AM pavian<EMAIL_ADDRESS>wrote:

Hi Andrew?
Any update on this issue?
I see a lot of important changes in the transfer Manager code after that
release and I really wanted to test the latest SDK for performance but I
can't do that until this problem is fixed.
Also, since you already have our sample project, I was wondering if you
could answer one more question for me. We are trying to play with a couple
of parameters to get the best performance (number of execution threads,
http connections, buffer size) while not using up the whole 100% of the
CPU. When compared to CLI we now seem to get a better performance on
uploads, but not on the downloads. Is there any other parameters you can
recommend we could play with to achieve the best download performance and
to avoid overloading the CPU?
Thank you very much,
Anna
On Mon, Jul 23, 2018 at 5:44 PM, Andrew Tang<EMAIL_ADDRESS>wrote:

It turns out that SDK would retry when there is connection error before
1.4.46, but will not after 1.4.46.
Details:
Before 1.4.46, when connection error happens, response will be nullptr,
then BuildAWSError will generate retry-able error code.
After 1.4.46, when connection error happens, when generating error,
retry-able or not is based on the Http response code, since the request
is
never made , the Http response code is REQUEST_NOT_MADE, and this code is
not in the retry-able code list.
I think it's doable to add REQUST_NOT_MADE to the list because all other
http codes in that list are related to connection error but we will need
to
review it first. Will update to you later.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-407211165,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AnQi-_Ir1na-3tOuWs1fScw5uNvgfyFJks5uJkPRgaJpZM4VNh2v
.[image: Web Bug from

https://github.com/notifications/beacon/AnQi-44wIwDkbPzINlYDSXfQmnM2CBH5ks5uJkPRgaJpZM4VNh2v.gif
]

{"api_version":"1.0","publisher":{"api_key":"
05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":

{"external_key":"github/aws/aws-sdk-cpp","title":"aws/aws-sdk-cpp","subtitle":"GitHub

repository","main_image_url":"https://assets-cdn.github.com/
images/email/message_cards/header.png","avatar_image_url":"
https://assets-cdn.github.com/images/email/message_cards/avatar.png
","action":{"name":"Open in GitHub","url":"https://github.

com/aws/aws-sdk-cpp"}},"updates":{"snippets":[{"icon":"PERSON","message":"@singku

in #910: It turns out that SDK would retry when there is connection error
before 1.4.46, but will not after 1.4.46.\r\nDetails:\r\n\u003eBefore
1.4.46, when connection error happens, response will be nullptr, then
BuildAWSError will generate retry-able error code.\r\n\u003eAfter
1.4.46,
when connection error happens, when generating error, retry-able or not
is
based on the Http response code, since the request is never made , the
Http
response code is REQUEST_NOT_MADE, and this code is not in the retry-able
code list. \r\n\r\nI think it's doable to add REQUST_NOT_MADE to the list
because all other http codes in that list are related to connection error
but we will need to review it first. Will update to you
later."}],"action":{"name":"View Issue","url":"https://github.
com/aws/aws-sdk-cpp/issues/910#issuecomment-407211165"}}} [ { "@context":
"http://schema.org", "@type": "EmailMessage", "potentialAction": {
"@type": "ViewAction", "target": "https://github.com/aws/aws-
sdk-cpp/issues/910#issuecomment-407211165", "url": "
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-407211165",
"name": "View Issue" }, "description": "View this Issue on GitHub",
"publisher": { "@type": "Organization", "name": "GitHub", "url": "
https://github.com" } }, { "@type": "MessageCard", "@context": "
http://schema.org/extensions", "hideOriginalBody": "false",
"originator":
"AF6C5A86-E920-430C-9C59-A73278B5EFEB", "title": "Re: [aws/aws-sdk-cpp]
UNKNOWN s3 error (100) in transfer manager as of Release 1.4.46- BUG
(#910)", "sections": [ { "text": "", "activityTitle": "Andrew Tang",
"activityImage": "https://assets-cdn.github.com/images/email/message_
cards/avatar.png", "activitySubtitle": "@singku", "facts": [ ] } ],
"potentialAction": [ { "name": "Add a comment", "@type": "ActionCard",
"inputs": [ { "isMultiLine": true, "@type": "TextInput", "id":
"IssueComment", "isRequired": false } ], "actions": [ { "name":
"Comment",
"@type": "HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueComment",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910,\n"IssueComment":
"{{IssueComment.value}}"\n}" } ] }, { "name": "Close issue", "@type":
"HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueClose",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910\n}" }, { "targets": [ { "os":
"default", "uri": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-407211165" } ], "@type": "OpenUri", "name": "View on GitHub"
}, { "name": "Unsubscribe", "@type": "HttpPOST", "target": "
https://api.github.com", "body": "{\n"commandName":
"MuteNotification",\n"threadId": 355868079\n}" } ], "themeColor":
"26292E" } ]
Click here
http://esva5.caseware.com/cgi-bin/learn-msg.cgi?id=EA81528867.ACF12 to
report this message as spam to Caseware E-Mail Security Server ESVA.

--
Anna Siniavina
Working Papers Developer
T: 416·867·9504 x1210 | F: 416·867·1906
<EMAIL_ADDRESS>https://www.caseware.com/
https://linkedin.com/company/caseware-international-inc
https://facebook.com/CaseWare-International-Inc-54418196590/
https://twitter.com/caseware
https://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409299049,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADTsCEyI8cLGx08IjQjTSY70al869wioks5uMJEWgaJpZM4VNh2v
.

--
Best Regards!
Liang.
Software Engineer
Game Developer

I see. it was committed 5 days ago. Does it mean both 1.5.3 and 1.5.4
releases will already have it then? It is not in their release descriptions.
On Wed, Aug 1, 2018 at 9:13 AM, Andrew Tang<EMAIL_ADDRESS>wrote:

96187c4
On Tue, Jul 31, 2018 at 10:17 AM pavian<EMAIL_ADDRESS>wrote:

Hi Andrew?
Any update on this issue?
I see a lot of important changes in the transfer Manager code after that
release and I really wanted to test the latest SDK for performance but I
can't do that until this problem is fixed.
Also, since you already have our sample project, I was wondering if you
could answer one more question for me. We are trying to play with a
couple
of parameters to get the best performance (number of execution threads,
http connections, buffer size) while not using up the whole 100% of the
CPU. When compared to CLI we now seem to get a better performance on
uploads, but not on the downloads. Is there any other parameters you can
recommend we could play with to achieve the best download performance and
to avoid overloading the CPU?
Thank you very much,
Anna
On Mon, Jul 23, 2018 at 5:44 PM, Andrew Tang<EMAIL_ADDRESS>wrote:

It turns out that SDK would retry when there is connection error before
1.4.46, but will not after 1.4.46.
Details:
Before 1.4.46, when connection error happens, response will be nullptr,
then BuildAWSError will generate retry-able error code.
After 1.4.46, when connection error happens, when generating error,
retry-able or not is based on the Http response code, since the request
is
never made , the Http response code is REQUEST_NOT_MADE, and this code
is
not in the retry-able code list.
I think it's doable to add REQUST_NOT_MADE to the list because all
other
http codes in that list are related to connection error but we will
need
to
review it first. Will update to you later.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-407211165
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AnQi-_Ir1na-
3tOuWs1fScw5uNvgfyFJks5uJkPRgaJpZM4VNh2v
.[image: Web Bug from

https://github.com/notifications/beacon/AnQi-
44wIwDkbPzINlYDSXfQmnM2CBH5ks5uJkPRgaJpZM4VNh2v.gif
]

{"api_version":"1.0","publisher":{"api_key":"
05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":

{"external_key":"github/aws/aws-sdk-cpp","title":"aws/aws-
sdk-cpp","subtitle":"GitHub

repository","main_image_url":"https://assets-cdn.github.com/
images/email/message_cards/header.png","avatar_image_url":"
https://assets-cdn.github.com/images/email/message_cards/avatar.png
","action":{"name":"Open in GitHub","url":"https://github.

com/aws/aws-sdk-cpp"}},"updates":{"snippets":[{"icon":
"PERSON","message":"@singku

in #910: It turns out that SDK would retry when there is connection
error
before 1.4.46, but will not after 1.4.46.\r\nDetails:\r\n\u003eBefore
1.4.46, when connection error happens, response will be nullptr, then
BuildAWSError will generate retry-able error code.\r\n\u003eAfter
1.4.46,
when connection error happens, when generating error, retry-able or not
is
based on the Http response code, since the request is never made , the
Http
response code is REQUEST_NOT_MADE, and this code is not in the
retry-able
code list. \r\n\r\nI think it's doable to add REQUST_NOT_MADE to the
list
because all other http codes in that list are related to connection
error
but we will need to review it first. Will update to you
later."}],"action":{"name":"View Issue","url":"https://github.
com/aws/aws-sdk-cpp/issues/910#issuecomment-407211165"}}} [ {
"@context":
"http://schema.org", "@type": "EmailMessage", "potentialAction": {
"@type": "ViewAction", "target": "https://github.com/aws/aws-
sdk-cpp/issues/910#issuecomment-407211165", "url": "
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-407211165",
"name": "View Issue" }, "description": "View this Issue on GitHub",
"publisher": { "@type": "Organization", "name": "GitHub", "url": "
https://github.com" } }, { "@type": "MessageCard", "@context": "
http://schema.org/extensions", "hideOriginalBody": "false",
"originator":
"AF6C5A86-E920-430C-9C59-A73278B5EFEB", "title": "Re:
[aws/aws-sdk-cpp]
UNKNOWN s3 error (100) in transfer manager as of Release 1.4.46- BUG
(#910)", "sections": [ { "text": "", "activityTitle": "Andrew
Tang",
"activityImage": "https://assets-cdn.github.com/images/email/message_
cards/avatar.png", "activitySubtitle": "@singku", "facts": [ ] } ],
"potentialAction": [ { "name": "Add a comment", "@type": "ActionCard",
"inputs": [ { "isMultiLine": true, "@type": "TextInput", "id":
"IssueComment", "isRequired": false } ], "actions": [ { "name":
"Comment",
"@type": "HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueComment",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910,\n"IssueComment":
"{{IssueComment.value}}"\n}" } ] }, { "name": "Close issue", "@type":
"HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueClose",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910\n}" }, { "targets": [ { "os":
"default", "uri": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-407211165" } ], "@type": "OpenUri", "name": "View on
GitHub"
}, { "name": "Unsubscribe", "@type": "HttpPOST", "target": "
https://api.github.com", "body": "{\n"commandName":
"MuteNotification",\n"threadId": 355868079\n}" } ], "themeColor":
"26292E" } ]
Click here
http://esva5.caseware.com/cgi-bin/learn-msg.cgi?id=EA81528867.ACF12
to
report this message as spam to Caseware E-Mail Security Server ESVA.

--
Anna Siniavina
Working Papers Developer
T: 416·867·9504 x1210 | F: 416·867·1906
<EMAIL_ADDRESS>https://www.caseware.com/
https://linkedin.com/company/caseware-international-inc
https://facebook.com/CaseWare-International-Inc-54418196590/
https://twitter.com/caseware
https://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409299049,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/
ADTsCEyI8cLGx08IjQjTSY70al869wioks5uMJEWgaJpZM4VNh2v>
.

--
Best Regards!
Liang.
Software Engineer
Game Developer
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409569816,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AnQi-18Qi-BLob1UX6wKzIXU2lWD0OxNks5uMal9gaJpZM4VNh2v
.[image: Web Bug from
https://github.com/notifications/beacon/AnQi-xLyDr0KQuurwZgsjo0_mLtdYEXHks5uMal9gaJpZM4VNh2v.gif]
{"api_version":"1.0","publisher":{"api_key":"
05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":
{"external_key":"github/aws/aws-sdk-cpp","title":"aws/aws-sdk-cpp","subtitle":"GitHub
repository","main_image_url":"https://assets-cdn.github.com/
images/email/message_cards/header.png","avatar_image_url":"
https://assets-cdn.github.com/images/email/message_cards/avatar.png
","action":{"name":"Open in GitHub","url":"https://github.
com/aws/aws-sdk-cpp"}},"updates":{"snippets":[{"icon":"PERSON","message":"@singku
in #910: 96187c4\n\nOn Tue, Jul 31, 2018 at 10:17 AM pavian
<EMAIL_ADDRESS>wrote:\n\n\u003e Hi Andrew?\n\u003e
Any update on this issue?\n\u003e I see a lot of important changes in the
transfer Manager code after that\n\u003e release and I really wanted to
test the latest SDK for performance but I\n\u003e can't do that until this
problem is fixed.\n\u003e\n\u003e Also, since you already have our sample
project, I was wondering if you\n\u003e could answer one more question for
me. We are trying to play with a couple\n\u003e of parameters to get the
best performance (number of execution threads,\n\u003e http connections,
buffer size) while not using up the whole 100% of the\n\u003e CPU. When
compared to CLI we now seem to get a better performance on\n\u003e uploads,
but not on the downloads. Is there any other parameters you can\n\u003e
recommend we could play with to achieve the best download performance
and\n\u003e to avoid overloading the CPU?\n\u003e\n\u003e Thank you very
much,\n\u003e Anna\n\u003e\n\u003e On Mon, Jul 23, 2018 at 5:44 PM, Andrew
Tang<EMAIL_ADDRESS>wrote:\n\u003e\n\u003e
\u003e It turns out that SDK would retry when there is connection error
before\n\u003e \u003e 1.4.46, but will not after 1.4.46.\n\u003e \u003e
Details:\n\u003e \u003e\n\u003e \u003e Before 1.4.46, when connection error
happens, response will be nullptr,\n\u003e \u003e then BuildAWSError will
generate retry-able error code.\n\u003e \u003e After 1.4.46, when
connection error happens, when generating error,\n\u003e \u003e retry-able
or not is based on the Http response code, since the request\n\u003e
is\n\u003e \u003e never made , the Http response code is REQUEST_NOT_MADE,
and this code is\n\u003e \u003e not in the retry-able code list.\n\u003e
\u003e\n\u003e \u003e I think it's doable to add REQUST_NOT_MADE to the
list because all other\n\u003e \u003e http codes in that list are related
to connection error but we will need\n\u003e to\n\u003e \u003e review it
first. Will update to you later.\n\u003e \u003e\n\u003e \u003e —\n\u003e
\u003e You are receiving this because you were mentioned.\n\u003e \u003e
Reply to this email directly, view it on GitHub\n\u003e \u003e
\u003chttps://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-407211165\u003e,\n\u003e
http://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-407211165\u003e,\n\u003e
\u003e or mute the thread\n\u003e \u003e \u003c\n\u003e
https://github.com/notifications/unsubscribe-auth/AnQi-_Ir1na-
3tOuWs1fScw5uNvgfyFJks5uJkPRgaJpZM4VNh2v\n\u003e \u003e\n\u003e \u003e
.[image: Web Bug from\n\u003e \u003e\n\u003e https://github.com/
notifications/beacon/AnQi-44wIwDkbPzINlYDSXfQmnM2CBH5ks5
uJkPRgaJpZM4VNh2v.gif\n\u003e ]\n\u003e \u003e {"api_version":"1.0","
publisher":{"api_key":"\n\u003e \u003e 05dde50f1d1a384dd78767c55493e4
bb","name":"GitHub"},"entity":\n\u003e \u003e\n\u003e
{"external_key":"github/aws/aws-sdk-cpp","title":"
aws/aws-sdk-cpp","subtitle":"GitHub\n\u003e \u003e
repository","main_image_url":"https://assets-cdn.github.com/\n\u003e
\u003e images/email/message_cards/header.png","avatar_image_url":"\n\u003e
\u003e https://assets-cdn.github.com/images/email/message_cards/
avatar.png\n\u003e \u003e ","action":{"name":"Open in
GitHub","url":"https://github.\n\u003e \u003e\n\u003e
com/aws/aws-sdk-cpp"}},"updates":{"snippets":[{"
icon":"PERSON","message":"@singku\n\u003e \u003e in #910: It turns
out that SDK would retry when there is connection error\n\u003e \u003e
before 1.4.46, but will not after 1.4.46.\r\nDetails:\r\n\u003eBefore\n\u003e
\u003e 1.4.46, when connection error happens, response will be nullptr,
then\n\u003e \u003e BuildAWSError will generate retry-able error
code.\r\n\u003eAfter\n\u003e 1.4.46,\n\u003e \u003e when connection
error happens, when generating error, retry-able or not\n\u003e is\n\u003e
\u003e based on the Http response code, since the request is never made ,
the\n\u003e Http\n\u003e \u003e response code is REQUEST_NOT_MADE, and this
code is not in the retry-able\n\u003e \u003e code list. \r\n\r\nI think
it's doable to add REQUST_NOT_MADE to the list\n\u003e \u003e because all
other http codes in that list are related to connection error\n\u003e
\u003e but we will need to review it first. Will update to you\n\u003e
\u003e later."}],"action":{"name":"View Issue","url":"https://
github.\n\u003e \u003e com/aws/aws-sdk-cpp/issues/
910#issuecomment-407211165"}}} [ { "@context":\n\u003e \u003e "
http://schema.org", "@type": "EmailMessage", "potentialAction":
{\n\u003e \u003e "@type": "ViewAction", "target": "
https://github.com/aws/aws-\n\u003e \u003e sdk-cpp/issues/910#issuecomment-407211165",
"url": "\n\u003e \u003e https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-407211165",\n\u003e \u003e "name": "View Issue" },
"description": "View this Issue on GitHub",\n\u003e \u003e
"publisher": { "@type": "Organization", "name": "GitHub",
"url": "\n\u003e \u003e https://github.com" } }, { "@type":
"MessageCard", "@context": "\n\u003e \u003e
http://schema.org/extensions\", "hideOriginalBody": "false",\n\u003e
"originator":\n\u003e \u003e "AF6C5A86-E920-430C-9C59-A73278B5EFEB",
"title": "Re: [aws/aws-sdk-cpp]\n\u003e \u003e UNKNOWN s3 error (100) in
transfer manager as of Release 1.4.46- BUG\n\u003e \u003e (#910)",
"sections": [ { "text": "", "activityTitle": "Andrew
Tang",\n\u003e \u003e "activityImage": "https://assets-cdn.github.
com/images/email/message_\n\u003e \u003e cards/avatar.png",
"activitySubtitle": "@singku", "facts": [ ] } ],\n\u003e \u003e
"potentialAction": [ { "name": "Add a comment", "@type":
"ActionCard",\n\u003e \u003e "inputs": [ { "isMultiLine": true,
"@type": "TextInput", "id":\n\u003e \u003e "IssueComment",
"isRequired": false } ], "actions": [ { "name":\n\u003e
"Comment",\n\u003e \u003e "@type": "HttpPOST", "target": "
https://api.github.com", "body":\n\u003e \u003e
"{\n\"commandName\": \"IssueComment\",\n\"
repositoryFullName\":\n\u003e \u003e \"aws/aws-sdk-cpp\",\n\"issueId\":
910,\n\"IssueComment\":\n\u003e \u003e \"{{IssueComment.value}}\"\n}"
} ] }, { "name": "Close issue", "@type":\n\u003e \u003e "HttpPOST",
"target": "https://api.github.com", "body":\n\u003e \u003e
"{\n\"commandName\": \"IssueClose\",\n\"
repositoryFullName\":\n\u003e \u003e \"aws/aws-sdk-cpp\",\n\"issueId\":
910\n}" }, { "targets": [ { "os":\n\u003e \u003e "default",
"uri": "https://github.com/aws/aws-sdk-cpp/issues/910#\n\u003e \u003e
issuecomment-407211165" } ], "@type": "OpenUri", "name": "View on
GitHub"\n\u003e \u003e }, { "name": "Unsubscribe", "@type":
"HttpPOST", "target": "\n\u003e \u003e https://api.github.com",
"body": "{\n\"commandName\":\n\u003e \u003e
\"MuteNotification\",\n\"threadId\": 355868079\n}" } ],
"themeColor":\n\u003e \u003e "26292E" } ]\n\u003e \u003e --\n\u003e
\u003e Click here\n\u003e \u003e \u003chttp://esva5.caseware.
com/cgi-bin/learn-msg.cgi?id=EA81528867.ACF12\u003e
http://esva5.caseware.com/cgi-bin/learn-msg.cgi?id=EA81528867.ACF12\u003e
to\n\u003e \u003e report this message as spam to Caseware E-Mail Security
Server ESVA.\n\u003e\n\u003e\n\u003e\n\u003e\n\u003e --\n\u003e\n\u003e
Anna Siniavina\n\u003e Working Papers Developer\n\u003e T: 416·867·9504
x1210 | F: 416·867·1906\n\u003e<EMAIL_ADDRESS>\u003chttps://www.caseware.com/\u003e\n\u003e\n\u003e
http://www.caseware.com/\u003e\n\u003e\n\u003e \u003chttps://
linkedin.com/company/caseware-international-inc\u003e\n\u003e
http://linkedin.com/company/caseware-international-inc\u003e\n\u003e
\u003chttps://facebook.com/CaseWare-International-Inc-
54418196590/\u003e\n\u003e
http://facebook.com/CaseWare-International-Inc-54418196590/\u003e\n\u003e
\u003chttps://twitter.com/caseware\u003e\n\u003e
http://twitter.com/caseware\u003e\n\u003e \u003chttps://
youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg\u003e\n\u003e\n\u003e
http://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg\u003e\n\u003e\n\u003e
—\n\u003e You are receiving this because you modified the open/close
state.\n\u003e Reply to this email directly, view it on GitHub\n\u003e
\u003chttps://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-409299049\u003e,\n\u003e
http://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409299049\u003e,\n\u003e
or mute the thread\n\u003e \u003chttps://github.com/
notifications/unsubscribe-auth/ADTsCEyI8cLGx08IjQjTSY70al869w
ioks5uMJEWgaJpZM4VNh2v\u003e\n\u003e
http://github.com/notifications/unsubscribe-auth/ADTsCEyI8cLGx08IjQjTSY70al869wioks5uMJEWgaJpZM4VNh2v\u003e\n\u003e
.\n\u003e\n-- \n\nBest Regards!\n\nLiang.\n------------------\nSoftware
Engineer\nGame Developer\n"}],"action":{"name":"View Issue","url":"
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409569816"}}} [
{ "@context": "http://schema.org", "@type": "EmailMessage",
"potentialAction": { "@type": "ViewAction", "target": "
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409569816",
"url": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-409569816", "name": "View Issue" }, "description": "View
this Issue on GitHub", "publisher": { "@type": "Organization", "name":
"GitHub", "url": "https://github.com" } }, { "@type": "MessageCard",
"@context": "http://schema.org/extensions", "hideOriginalBody": "false",
"originator": "AF6C5A86-E920-430C-9C59-A73278B5EFEB", "title": "Re:
[aws/aws-sdk-cpp] UNKNOWN s3 error (100) in transfer manager as of Release
1.4.46 (#910)", "sections": [ { "text": "", "activityTitle": "Andrew
Tang", "activityImage": "https://assets-cdn.github.
com/images/email/message_cards/avatar.png", "activitySubtitle":
"@singku", "facts": [ ] } ], "potentialAction": [ { "name": "Add a
comment", "@type": "ActionCard", "inputs": [ { "isMultiLine": true,
"@type": "TextInput", "id": "IssueComment", "isRequired": false } ],
"actions": [ { "name": "Comment", "@type": "HttpPOST", "target": "
https://api.github.com", "body": "{\n"commandName":
"IssueComment",\n"repositoryFullName": "aws/aws-sdk-cpp",\n"issueId":
910,\n"IssueComment": "{{IssueComment.value}}"\n}" } ] }, { "name":
"Close issue", "@type": "HttpPOST", "target": "https://api.github.com",
"body": "{\n"commandName": "IssueClose",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910\n}" }, { "targets": [ { "os":
"default", "uri": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-409569816" } ], "@type": "OpenUri", "name": "View on GitHub"
}, { "name": "Unsubscribe", "@type": "HttpPOST", "target": "
https://api.github.com", "body": "{\n"commandName":
"MuteNotification",\n"threadId": 355868079\n}" } ], "themeColor":
"26292E" } ]
Click here
http://esva4.caseware.com/cgi-bin/learn-msg.cgi?id=4CB8F28FBE.AA5CF to
report this message as spam to Caseware E-Mail Security Server ESVA.

--
Anna Siniavina
Working Papers Developer
T: 416·867·9504 x1210 | F: 416·867·1906
<EMAIL_ADDRESS>https://www.caseware.com/
https://linkedin.com/company/caseware-international-inc
https://facebook.com/CaseWare-International-Inc-54418196590/
https://twitter.com/caseware
https://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg

Right and we usually tag with the latest commit description.
On Wed, Aug 1, 2018 at 6:25 AM pavian<EMAIL_ADDRESS>wrote:

I see. it was committed 5 days ago. Does it mean both 1.5.3 and 1.5.4
releases will already have it then? It is not in their release
descriptions.
On Wed, Aug 1, 2018 at 9:13 AM, Andrew Tang<EMAIL_ADDRESS>wrote:

96187c4
On Tue, Jul 31, 2018 at 10:17 AM pavian<EMAIL_ADDRESS>wrote:

Hi Andrew?
Any update on this issue?
I see a lot of important changes in the transfer Manager code after
that
release and I really wanted to test the latest SDK for performance but
I
can't do that until this problem is fixed.
Also, since you already have our sample project, I was wondering if you
could answer one more question for me. We are trying to play with a
couple
of parameters to get the best performance (number of execution threads,
http connections, buffer size) while not using up the whole 100% of the
CPU. When compared to CLI we now seem to get a better performance on
uploads, but not on the downloads. Is there any other parameters you
can
recommend we could play with to achieve the best download performance
and
to avoid overloading the CPU?
Thank you very much,
Anna
On Mon, Jul 23, 2018 at 5:44 PM, Andrew Tang<EMAIL_ADDRESS>

wrote:

It turns out that SDK would retry when there is connection error
before
1.4.46, but will not after 1.4.46.
Details:
Before 1.4.46, when connection error happens, response will be
nullptr,
then BuildAWSError will generate retry-able error code.
After 1.4.46, when connection error happens, when generating error,
retry-able or not is based on the Http response code, since the
request
is
never made , the Http response code is REQUEST_NOT_MADE, and this
code
is
not in the retry-able code list.
I think it's doable to add REQUST_NOT_MADE to the list because all
other
http codes in that list are related to connection error but we will
need
to
review it first. Will update to you later.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-407211165
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AnQi-_Ir1na-
3tOuWs1fScw5uNvgfyFJks5uJkPRgaJpZM4VNh2v
.[image: Web Bug from

https://github.com/notifications/beacon/AnQi-
44wIwDkbPzINlYDSXfQmnM2CBH5ks5uJkPRgaJpZM4VNh2v.gif
]

{"api_version":"1.0","publisher":{"api_key":"
05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":

{"external_key":"github/aws/aws-sdk-cpp","title":"aws/aws-
sdk-cpp","subtitle":"GitHub

repository","main_image_url":"https://assets-cdn.github.com/
images/email/message_cards/header.png","avatar_image_url":"
https://assets-cdn.github.com/images/email/message_cards/avatar.png
","action":{"name":"Open in GitHub","url":"https://github.

com/aws/aws-sdk-cpp"}},"updates":{"snippets":[{"icon":
"PERSON","message":"@singku

in #910: It turns out that SDK would retry when there is connection
error
before 1.4.46, but will not after 1.4.46.\r\nDetails:\r\n\u003eBefore
1.4.46, when connection error happens, response will be nullptr, then
BuildAWSError will generate retry-able error code.\r\n\u003eAfter
1.4.46,
when connection error happens, when generating error, retry-able or
not
is
based on the Http response code, since the request is never made ,
the
Http
response code is REQUEST_NOT_MADE, and this code is not in the
retry-able
code list. \r\n\r\nI think it's doable to add REQUST_NOT_MADE to the
list
because all other http codes in that list are related to connection
error
but we will need to review it first. Will update to you
later."}],"action":{"name":"View Issue","url":"https://github.
com/aws/aws-sdk-cpp/issues/910#issuecomment-407211165"}}} [ {
"@context":
"http://schema.org", "@type": "EmailMessage", "potentialAction": {
"@type": "ViewAction", "target": "https://github.com/aws/aws-
sdk-cpp/issues/910#issuecomment-407211165", "url": "
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-407211165
",
"name": "View Issue" }, "description": "View this Issue on GitHub",
"publisher": { "@type": "Organization", "name": "GitHub", "url": "
https://github.com" } }, { "@type": "MessageCard", "@context": "
http://schema.org/extensions", "hideOriginalBody": "false",
"originator":
"AF6C5A86-E920-430C-9C59-A73278B5EFEB", "title": "Re:
[aws/aws-sdk-cpp]
UNKNOWN s3 error (100) in transfer manager as of Release 1.4.46- BUG
(#910)", "sections": [ { "text": "", "activityTitle": "Andrew
Tang",
"activityImage": "
https://assets-cdn.github.com/images/email/message_
cards/avatar.png", "activitySubtitle": "@singku", "facts": [ ] } ],
"potentialAction": [ { "name": "Add a comment", "@type":
"ActionCard",
"inputs": [ { "isMultiLine": true, "@type": "TextInput", "id":
"IssueComment", "isRequired": false } ], "actions": [ { "name":
"Comment",
"@type": "HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueComment",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910,\n"IssueComment":
"{{IssueComment.value}}"\n}" } ] }, { "name": "Close issue",
"@type":
"HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueClose",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910\n}" }, { "targets": [ { "os":
"default", "uri": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-407211165" } ], "@type": "OpenUri", "name": "View on
GitHub"
}, { "name": "Unsubscribe", "@type": "HttpPOST", "target": "
https://api.github.com", "body": "{\n"commandName":
"MuteNotification",\n"threadId": 355868079\n}" } ], "themeColor":
"26292E" } ]
Click here
<http://esva5.caseware.com/cgi-bin/learn-msg.cgi?id=EA81528867.ACF12

to

report this message as spam to Caseware E-Mail Security Server ESVA.

--
Anna Siniavina
Working Papers Developer
T: 416·867·9504 x1210 | F: 416·867·1906
<EMAIL_ADDRESS>https://www.caseware.com/
https://linkedin.com/company/caseware-international-inc
https://facebook.com/CaseWare-International-Inc-54418196590/
https://twitter.com/caseware
https://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409299049
,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/
ADTsCEyI8cLGx08IjQjTSY70al869wioks5uMJEWgaJpZM4VNh2v>
.

--
Best Regards!
Liang.
Software Engineer
Game Developer
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409569816,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AnQi-18Qi-BLob1UX6wKzIXU2lWD0OxNks5uMal9gaJpZM4VNh2v
.[image: Web Bug from

https://github.com/notifications/beacon/AnQi-xLyDr0KQuurwZgsjo0_mLtdYEXHks5uMal9gaJpZM4VNh2v.gif
]

{"api_version":"1.0","publisher":{"api_key":"
05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":

{"external_key":"github/aws/aws-sdk-cpp","title":"aws/aws-sdk-cpp","subtitle":"GitHub

repository","main_image_url":"https://assets-cdn.github.com/
images/email/message_cards/header.png","avatar_image_url":"
https://assets-cdn.github.com/images/email/message_cards/avatar.png
","action":{"name":"Open in GitHub","url":"https://github.

com/aws/aws-sdk-cpp"}},"updates":{"snippets":[{"icon":"PERSON","message":"@singku

in #910: 96187c4\n\nOn Tue, Jul 31, 2018 at 10:17 AM pavian
<EMAIL_ADDRESS>wrote:\n\n\u003e Hi Andrew?\n\u003e
Any update on this issue?\n\u003e I see a lot of important changes in the
transfer Manager code after that\n\u003e release and I really wanted to
test the latest SDK for performance but I\n\u003e can't do that until
this
problem is fixed.\n\u003e\n\u003e Also, since you already have our sample
project, I was wondering if you\n\u003e could answer one more question
for
me. We are trying to play with a couple\n\u003e of parameters to get the
best performance (number of execution threads,\n\u003e http connections,
buffer size) while not using up the whole 100% of the\n\u003e CPU. When
compared to CLI we now seem to get a better performance on\n\u003e
uploads,
but not on the downloads. Is there any other parameters you can\n\u003e
recommend we could play with to achieve the best download performance
and\n\u003e to avoid overloading the CPU?\n\u003e\n\u003e Thank you very
much,\n\u003e Anna\n\u003e\n\u003e On Mon, Jul 23, 2018 at 5:44 PM,
Andrew
Tang<EMAIL_ADDRESS>wrote:\n\u003e\n\u003e
\u003e It turns out that SDK would retry when there is connection error
before\n\u003e \u003e 1.4.46, but will not after 1.4.46.\n\u003e \u003e
Details:\n\u003e \u003e\n\u003e \u003e Before 1.4.46, when connection
error
happens, response will be nullptr,\n\u003e \u003e then BuildAWSError will
generate retry-able error code.\n\u003e \u003e After 1.4.46, when
connection error happens, when generating error,\n\u003e \u003e
retry-able
or not is based on the Http response code, since the request\n\u003e
is\n\u003e \u003e never made , the Http response code is
REQUEST_NOT_MADE,
and this code is\n\u003e \u003e not in the retry-able code list.\n\u003e
\u003e\n\u003e \u003e I think it's doable to add REQUST_NOT_MADE to the
list because all other\n\u003e \u003e http codes in that list are related
to connection error but we will need\n\u003e to\n\u003e \u003e review it
first. Will update to you later.\n\u003e \u003e\n\u003e \u003e —\n\u003e
\u003e You are receiving this because you were mentioned.\n\u003e \u003e
Reply to this email directly, view it on GitHub\n\u003e \u003e
\u003chttps://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-407211165\u003e,\n\u003e
<
http://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-407211165\u003e,\n\u003e
\u003e or mute the thread\n\u003e \u003e \u003c\n\u003e
https://github.com/notifications/unsubscribe-auth/AnQi-_Ir1na-
3tOuWs1fScw5uNvgfyFJks5uJkPRgaJpZM4VNh2v\n\u003e \u003e\n\u003e \u003e
.[image: Web Bug from\n\u003e \u003e\n\u003e https://github.com/
notifications/beacon/AnQi-44wIwDkbPzINlYDSXfQmnM2CBH5ks5
uJkPRgaJpZM4VNh2v.gif\n\u003e ]\n\u003e \u003e
{"api_version":"1.0","
publisher":{"api_key":"\n\u003e \u003e 05dde50f1d1a384dd78767c55493e4
bb","name":"GitHub"},"entity":\n\u003e \u003e\n\u003e
{"external_key":"github/aws/aws-sdk-cpp","title":"
aws/aws-sdk-cpp","subtitle":"GitHub\n\u003e \u003e
repository","main_image_url":"https://assets-cdn.github.com/\n\u003e
\u003e
images/email/message_cards/header.png","avatar_image_url":"\n\u003e
\u003e https://assets-cdn.github.com/images/email/message_cards/
avatar.png\n\u003e \u003e ","action":{"name":"Open in
GitHub","url":"https://github.\n\u003e \u003e\n\u003e
com/aws/aws-sdk-cpp"}},"updates":{"snippets":[{"
icon":"PERSON","message":"@singku\n\u003e \u003e in #910: It turns
out that SDK would retry when there is connection error\n\u003e \u003e
before 1.4.46, but will not after
1.4.46.\r\nDetails:\r\n\u003eBefore\n\u003e
\u003e 1.4.46, when connection error happens, response will be nullptr,
then\n\u003e \u003e BuildAWSError will generate retry-able error
code.\r\n\u003eAfter\n\u003e 1.4.46,\n\u003e \u003e when connection
error happens, when generating error, retry-able or not\n\u003e
is\n\u003e
\u003e based on the Http response code, since the request is never made ,
the\n\u003e Http\n\u003e \u003e response code is REQUEST_NOT_MADE, and
this
code is not in the retry-able\n\u003e \u003e code list. \r\n\r\nI
think
it's doable to add REQUST_NOT_MADE to the list\n\u003e \u003e because all
other http codes in that list are related to connection error\n\u003e
\u003e but we will need to review it first. Will update to you\n\u003e
\u003e later."}],"action":{"name":"View Issue","url":"https://
github.\n\u003e \u003e com/aws/aws-sdk-cpp/issues/
910#issuecomment-407211165"}}} [ { "@context":\n\u003e \u003e "
http://schema.org", "@type": "EmailMessage", "potentialAction":
{\n\u003e \u003e "@type": "ViewAction", "target": "
https://github.com/aws/aws-\n\u003e \u003e
sdk-cpp/issues/910#issuecomment-407211165",
"url": "\n\u003e \u003e
https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-407211165",\n\u003e \u003e "name": "View Issue" },
"description": "View this Issue on GitHub",\n\u003e \u003e
"publisher": { "@type": "Organization", "name": "GitHub",
"url": "\n\u003e \u003e https://github.com" } }, { "@type":
"MessageCard", "@context": "\n\u003e \u003e
http://schema.org/extensions\", "hideOriginalBody": "false",\n\u003e
"originator":\n\u003e \u003e "AF6C5A86-E920-430C-9C59-A73278B5EFEB",
"title": "Re: [aws/aws-sdk-cpp]\n\u003e \u003e UNKNOWN s3 error (100)
in
transfer manager as of Release 1.4.46- BUG\n\u003e \u003e (#910)",
"sections": [ { "text": "", "activityTitle": "Andrew
Tang",\n\u003e \u003e "activityImage": "https://assets-cdn.github.
com/images/email/message_\n\u003e \u003e cards/avatar.png",
"activitySubtitle": "@singku", "facts": [ ] } ],\n\u003e \u003e
"potentialAction": [ { "name": "Add a comment", "@type":
"ActionCard",\n\u003e \u003e "inputs": [ { "isMultiLine": true,
"@type": "TextInput", "id":\n\u003e \u003e "IssueComment",
"isRequired": false } ], "actions": [ { "name":\n\u003e
"Comment",\n\u003e \u003e "@type": "HttpPOST", "target": "
https://api.github.com", "body":\n\u003e \u003e
"{\n\"commandName\": \"IssueComment\",\n\"
repositoryFullName\":\n\u003e \u003e
\"aws/aws-sdk-cpp\",\n\"issueId\":
910,\n\"IssueComment\":\n\u003e \u003e
\"{{IssueComment.value}}\"\n}"
} ] }, { "name": "Close issue", "@type":\n\u003e \u003e
"HttpPOST",
"target": "https://api.github.com", "body":\n\u003e \u003e
"{\n\"commandName\": \"IssueClose\",\n\"
repositoryFullName\":\n\u003e \u003e
\"aws/aws-sdk-cpp\",\n\"issueId\":
910\n}" }, { "targets": [ { "os":\n\u003e \u003e "default",
"uri": "https://github.com/aws/aws-sdk-cpp/issues/910#\n\u003e \u003e
issuecomment-407211165" } ], "@type": "OpenUri", "name": "View on
GitHub"\n\u003e \u003e }, { "name": "Unsubscribe", "@type":
"HttpPOST", "target": "\n\u003e \u003e https://api.github.com",
"body": "{\n\"commandName\":\n\u003e \u003e
\"MuteNotification\",\n\"threadId\": 355868079\n}" } ],
"themeColor":\n\u003e \u003e "26292E" } ]\n\u003e \u003e --\n\u003e
\u003e Click here\n\u003e \u003e \u003chttp://esva5.caseware.
com/cgi-bin/learn-msg.cgi?id=EA81528867.ACF12\u003e
<
http://esva5.caseware.com/cgi-bin/learn-msg.cgi?id=EA81528867.ACF12\u003e
to\n\u003e \u003e report this message as spam to Caseware E-Mail Security
Server ESVA.\n\u003e\n\u003e\n\u003e\n\u003e\n\u003e --\n\u003e\n\u003e
Anna Siniavina\n\u003e Working Papers Developer\n\u003e T: 416·867·9504
x1210 | F: 416·867·1906\n\u003e<EMAIL_ADDRESS>\n\u003e\n\u003e
\u003chttps://www.caseware.com/\u003e\n\u003e\n\u003e
http://www.caseware.com/\u003e\n\u003e\n\u003e
http://www.caseware.com/\u003e\n\u003e\n\u003e
\u003chttps://
linkedin.com/company/caseware-international-inc\u003e\n\u003e
http://linkedin.com/company/caseware-international-inc\u003e\n\u003e
<
http://linkedin.com/company/caseware-international-inc\u003e\n\u003e
\u003chttps://facebook.com/CaseWare-International-Inc-
54418196590/\u003e\n\u003e
<
http://facebook.com/CaseWare-International-Inc-54418196590/\u003e\n\u003e
\u003chttps://twitter.com/caseware\u003e\n\u003e
http://twitter.com/caseware\u003e\n\u003e
http://twitter.com/caseware\u003e\n\u003e \u003chttps://
youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg\u003e\n\u003e\n\u003e
http://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg\u003e\n\u003e\n\u003e
<
http://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg\u003e\n\u003e\n\u003e
—\n\u003e You are receiving this because you modified the open/close
state.\n\u003e Reply to this email directly, view it on GitHub\n\u003e
\u003chttps://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-409299049\u003e,\n\u003e
<
http://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409299049\u003e,\n\u003e
or mute the thread\n\u003e \u003chttps://github.com/
notifications/unsubscribe-auth/ADTsCEyI8cLGx08IjQjTSY70al869w
ioks5uMJEWgaJpZM4VNh2v\u003e\n\u003e
<
http://github.com/notifications/unsubscribe-auth/ADTsCEyI8cLGx08IjQjTSY70al869wioks5uMJEWgaJpZM4VNh2v\u003e\n\u003e
.\n\u003e\n-- \n\nBest Regards!\n\nLiang.\n------------------\nSoftware
Engineer\nGame Developer\n"}],"action":{"name":"View Issue","url":"
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409569816"}}}
[
{ "@context": "http://schema.org", "@type": "EmailMessage",
"potentialAction": { "@type": "ViewAction", "target": "
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409569816",
"url": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-409569816", "name": "View Issue" }, "description": "View
this Issue on GitHub", "publisher": { "@type": "Organization", "name":
"GitHub", "url": "https://github.com" } }, { "@type": "MessageCard",
"@context": "http://schema.org/extensions", "hideOriginalBody": "false",
"originator": "AF6C5A86-E920-430C-9C59-A73278B5EFEB", "title": "Re:
[aws/aws-sdk-cpp] UNKNOWN s3 error (100) in transfer manager as of
Release
1.4.46 (#910)", "sections": [ { "text": "", "activityTitle": "Andrew
Tang", "activityImage": "https://assets-cdn.github.
com/images/email/message_cards/avatar.png", "activitySubtitle":
"@singku", "facts": [ ] } ], "potentialAction": [ { "name": "Add a
comment", "@type": "ActionCard", "inputs": [ { "isMultiLine": true,
"@type": "TextInput", "id": "IssueComment", "isRequired": false } ],
"actions": [ { "name": "Comment", "@type": "HttpPOST", "target": "
https://api.github.com", "body": "{\n"commandName":
"IssueComment",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId":
910,\n"IssueComment": "{{IssueComment.value}}"\n}" } ] }, { "name":
"Close issue", "@type": "HttpPOST", "target": "https://api.github.com",
"body": "{\n"commandName": "IssueClose",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910\n}" }, { "targets": [ { "os":
"default", "uri": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-409569816" } ], "@type": "OpenUri", "name": "View on GitHub"
}, { "name": "Unsubscribe", "@type": "HttpPOST", "target": "
https://api.github.com", "body": "{\n"commandName":
"MuteNotification",\n"threadId": 355868079\n}" } ], "themeColor":
"26292E" } ]
Click here
http://esva4.caseware.com/cgi-bin/learn-msg.cgi?id=4CB8F28FBE.AA5CF to
report this message as spam to Caseware E-Mail Security Server ESVA.

--
Anna Siniavina
Working Papers Developer
T: 416·867·9504 x1210 | F: 416·867·1906
<EMAIL_ADDRESS>https://www.caseware.com/
https://linkedin.com/company/caseware-international-inc
https://facebook.com/CaseWare-International-Inc-54418196590/
https://twitter.com/caseware
https://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409573320,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADTsCJ95IjtX3WZVvx-6IG5aHW3hPnrJks5uMaw1gaJpZM4VNh2v
.

--
Best Regards!
Liang.
Software Engineer
Game Developer

Hi Andrew.
There is still something strange about the way the SDK downloads. I am
running my test app with one huge 8 gb file, trying to play with the buffer
size.
It does make a huge difference on the performance in this case, but the
downloads still take 3 times longer than uploads, and I can see it in the
process monitor. Like, uploads go up to 300-350 mbps, downloads stay below
a hundred with the same parameter values. This can't be right.
Here are the results:
Version 2 maxConnection = 25 executor threads = 4 bufferFactor =  1 (def
5MB buffer)
8-1-2018 - 13:12:40 : UPLOAD START : c7487ddb-be84-43d2-97f4-9059d99c9e7f -
D:\PerformanceTest\Src\8gigFile.ac_
8-1-2018 - 13:19:49 : UPLOAD END c7487ddb-be84-43d2-97f4-9059d99c9e7f
8-1-2018 - 13:19:51 : DOWNLOAD START c7487ddb-be84-43d2-97f4-9059d99c9e7f -
D:\PerformanceTest\Download\8gigFile.ac_
8-1-2018 - 13:38:17: DOWNLOAD END c7487ddb-be84-43d2-97f4-9059d99c9e7f
Version 2 maxConnection = 25 executor threads = 5 bufferFactor =  5 (25 MB
buffer )
8-1-2018 - 13:48:39 : UPLOAD START : 096f9289-c5f7-4f74-8c73-9d78886b26cb -
D:\PerformanceTest\Src\8gigFile.ac_
8-1-2018 - 13:52:55 : UPLOAD END 096f9289-c5f7-4f74-8c73-9d78886b26cb
8-1-2018 - 13:52:55 : DOWNLOAD START 096f9289-c5f7-4f74-8c73-9d78886b26cb -
D:\PerformanceTest\Download\8gigFile.ac_
8-1-2018 - 14:05:11: DOWNLOAD END 096f9289-c5f7-4f74-8c73-9d78886b26cb
Can you please see if there is an issue with the way downloads are
implemented?
Thank you
P.S. I can send you an 8 gb file if necessary
Anna
On Tue, Jul 31, 2018 at 4:09 PM, Andrew Tang<EMAIL_ADDRESS>wrote:

Hi Anna, we have fixed the retry issue for connection problem for
transferManager. You saw the performance bump because we also replaced
Jsoncpp with cJson, which is faster on json deserialize. Not sure why you
just observed efficiency bump on upload not download. We don't have a
recommendation on threads used parallelly, it depends on the CPUs on your
host. Another place you could play with is the buffer size set in transfer
configuration. Take a look at the doc of https://github.com/aws/aws-
sdk-cpp/blob/e40d691f239f068794c268ac3d75ff6b62ddb7db/aws-cpp-sdk-
transfer/include/aws/transfer/TransferManager.h#L88
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409351520,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AnQi-8vE6dTHD1ZG9p2kmMnTVeuPQmdyks5uMLmOgaJpZM4VNh2v
.[image: Web Bug from
https://github.com/notifications/beacon/AnQi-0xOPBxu4_awln0sC0VZvHj-fdTqks5uMLmOgaJpZM4VNh2v.gif]
{"api_version":"1.0","publisher":{"api_key":"
05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":
{"external_key":"github/aws/aws-sdk-cpp","title":"aws/aws-sdk-cpp","subtitle":"GitHub
repository","main_image_url":"https://assets-cdn.github.com/
images/email/message_cards/header.png","avatar_image_url":"
https://assets-cdn.github.com/images/email/message_cards/avatar.png
","action":{"name":"Open in GitHub","url":"https://github.
com/aws/aws-sdk-cpp"}},"updates":{"snippets":[{"icon":"PERSON","message":"@singku
in #910: Hi Anna, we have fixed the retry issue for connection problem for
transferManager. You saw the performance bump because we also replaced
Jsoncpp with cJson, which is faster on json deserialize. Not sure why you
just observed efficiency bump on upload not download. We don't have a
recommendation on threads used parallelly, it depends on the CPUs on your
host. Another place you could play with is the buffer size set in transfer
configuration. Take a look at the doc of https://github.com/aws/aws-
sdk-cpp/blob/e40d691f239f068794c268ac3d75ff6b62ddb7db/aws-cpp-sdk-
transfer/include/aws/transfer/TransferManager.h#L88"}],"action":{"name":"View
Issue","url":"https://github.com/aws/aws-sdk-cpp/issues/
910#issuecomment-409351520"}}} [ { "@context": "http://schema.org",
"@type": "EmailMessage", "potentialAction": { "@type": "ViewAction",
"target": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-409351520", "url": "https://github.com/aws/aws-
sdk-cpp/issues/910#issuecomment-409351520", "name": "View Issue" },
"description": "View this Issue on GitHub", "publisher": { "@type":
"Organization", "name": "GitHub", "url": "https://github.com" } }, {
"@type": "MessageCard", "@context": "http://schema.org/extensions",
"hideOriginalBody": "false", "originator": "AF6C5A86-E920-430C-9C59-A73278B5EFEB",
"title": "Re: [aws/aws-sdk-cpp] UNKNOWN s3 error (100) in transfer manager
as of Release 1.4.46- BUG (#910)", "sections": [ { "text": "",
"activityTitle": "Andrew Tang", "activityImage": "
https://assets-cdn.github.com/images/email/message_cards/avatar.png",
"activitySubtitle": "@singku", "facts": [ ] } ], "potentialAction": [ {
"name": "Add a comment", "@type": "ActionCard", "inputs": [ {
"isMultiLine": true, "@type": "TextInput", "id": "IssueComment",
"isRequired": false } ], "actions": [ { "name": "Comment", "@type":
"HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueComment",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910,\n"IssueComment":
"{{IssueComment.value}}"\n}" } ] }, { "name": "Close issue", "@type":
"HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueClose",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910\n}" }, { "targets": [ { "os":
"default", "uri": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-409351520" } ], "@type": "OpenUri", "name": "View on GitHub"
}, { "name": "Unsubscribe", "@type": "HttpPOST", "target": "
https://api.github.com", "body": "{\n"commandName":
"MuteNotification",\n"threadId": 355868079\n}" } ], "themeColor":
"26292E" } ]
Click here
http://esva5.caseware.com/cgi-bin/learn-msg.cgi?id=A202927DF2.A8B1A to
report this message as spam to Caseware E-Mail Security Server ESVA.

--
Anna Siniavina
Working Papers Developer
T: 416·867·9504 x1210 | F: 416·867·1906
<EMAIL_ADDRESS>https://www.caseware.com/
https://linkedin.com/company/caseware-international-inc
https://facebook.com/CaseWare-International-Inc-54418196590/
https://twitter.com/caseware
https://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg

Building it myself. had some issues integrating nugget version with our
main projects.
Our app sets transferBufferMaxHeapSize = transferManagerConfig.bufferSize *
number of executor threads. Does this sound right? And it does cut the
times almost in half in both cases, so I can see it is working. However the
download is still a lot slower than the upload.
Thanks for looking into it!
Anna
On Wed, Aug 1, 2018 at 2:19 PM, Andrew Tang<EMAIL_ADDRESS>wrote:

Note: by setting the buffersize, you also need to set the
transferBufferMaxHeapSize, see the doc.
Question: Are you using the libraries from Nuget or build by yourself?
Anyway I will try your application with above settings to upload/download
large file. Will get back to you later.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409672360,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AnQi-8T5tdHlhFHGVV7Vr4I8W6oIXCXjks5uMfE1gaJpZM4VNh2v
.[image: Web Bug from
https://github.com/notifications/beacon/AnQi-wxTmt2i-IDagYK0lg-1NTiHtyQ6ks5uMfE1gaJpZM4VNh2v.gif]
{"api_version":"1.0","publisher":{"api_key":"
05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":
{"external_key":"github/aws/aws-sdk-cpp","title":"aws/aws-sdk-cpp","subtitle":"GitHub
repository","main_image_url":"https://assets-cdn.github.com/
images/email/message_cards/header.png","avatar_image_url":"
https://assets-cdn.github.com/images/email/message_cards/avatar.png
","action":{"name":"Open in GitHub","url":"https://github.
com/aws/aws-sdk-cpp"}},"updates":{"snippets":[{"icon":"PERSON","message":"@singku
in #910: Note: by setting the buffersize, you also need to set the
transferBufferMaxHeapSize, see the doc.\r\n\r\nQuestion: Are you using
the libraries from Nuget or build by yourself? \r\n\r\nAnyway I will try
your application with above settings to upload/download large file. Will
get back to you later. "}],"action":{"name":"View Issue","url":"
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409672360"}}} [
{ "@context": "http://schema.org", "@type": "EmailMessage",
"potentialAction": { "@type": "ViewAction", "target": "
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-409672360",
"url": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-409672360", "name": "View Issue" }, "description": "View
this Issue on GitHub", "publisher": { "@type": "Organization", "name":
"GitHub", "url": "https://github.com" } }, { "@type": "MessageCard",
"@context": "http://schema.org/extensions", "hideOriginalBody": "false",
"originator": "AF6C5A86-E920-430C-9C59-A73278B5EFEB", "title": "Re:
[aws/aws-sdk-cpp] UNKNOWN s3 error (100) in transfer manager as of Release
1.4.46 (#910)", "sections": [ { "text": "", "activityTitle": "Andrew
Tang", "activityImage": "https://assets-cdn.github.
com/images/email/message_cards/avatar.png", "activitySubtitle":
"@singku", "facts": [ ] } ], "potentialAction": [ { "name": "Add a
comment", "@type": "ActionCard", "inputs": [ { "isMultiLine": true,
"@type": "TextInput", "id": "IssueComment", "isRequired": false } ],
"actions": [ { "name": "Comment", "@type": "HttpPOST", "target": "
https://api.github.com", "body": "{\n"commandName":
"IssueComment",\n"repositoryFullName": "aws/aws-sdk-cpp",\n"issueId":
910,\n"IssueComment": "{{IssueComment.value}}"\n}" } ] }, { "name":
"Close issue", "@type": "HttpPOST", "target": "https://api.github.com",
"body": "{\n"commandName": "IssueClose",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910\n}" }, { "targets": [ { "os":
"default", "uri": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-409672360" } ], "@type": "OpenUri", "name": "View on GitHub"
}, { "name": "Unsubscribe", "@type": "HttpPOST", "target": "
https://api.github.com", "body": "{\n"commandName":
"MuteNotification",\n"threadId": 355868079\n}" } ], "themeColor":
"26292E" } ]
Click here
http://esva4.caseware.com/cgi-bin/learn-msg.cgi?id=0DEBC28FB8.ACE8C to
report this message as spam to Caseware E-Mail Security Server ESVA.

--
Anna Siniavina
Working Papers Developer
T: 416·867·9504 x1210 | F: 416·867·1906
<EMAIL_ADDRESS>https://www.caseware.com/
https://linkedin.com/company/caseware-international-inc
https://facebook.com/CaseWare-International-Inc-54418196590/
https://twitter.com/caseware
https://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg

@pavian Hi Anna,  I can reproduce this download slower than upload behavior on my end, I will find out the reason.

@pavian After several tests with large and small files. I found in general, upload is faster than download. Also after a closer looking at the code in core and TransferManager. I think this is the expected behavior.
When uploading a part. You need to read from local file and send it to network, when finished sending to network, a part is considered uploaded. S3 will take care of combining of parts, and you can start sending another part.
When downloading a part. Server need to read it and send it to network, when finished sending to network, you will need to write it to local file before using this buffer to receiving another part. (Note that parallel write can improve the speed a little bit, but the main cost is still the additional work, and introduce parallel write will make the code more complicated, because you will have to maintain multiple file handlers.)
So when downloading, there is an additional work you need to do. That's why download is slower than upload.
In order to verify that, I commented out the this line in TransferManager
https://github.com/aws/aws-sdk-cpp/blob/29ff65a8b321edd7ae4574bc74f55e9d26374527/aws-cpp-sdk-transfer/source/transfer/TransferManager.cpp#L896
Then download and upload time of a 40GB file is about the same.
I also tried using Aws CLI to upload and download a 40GB file. Though upload and download time is about the same, but it's around 6 minutes. Whereas in C++ SDK, upload takes 3 minutes, and download takes like 6 minutes.
Hope this can help.

Hi Andrew,
Thank you very much for investigating this.
Just want to clarify something. When you say "there is an additional work
you need to do", do you mean we have to do some preparations in our code
before downloading this file? Or is it just stating the fact the Transfer
Manager needs to do more work?
If I read your comment properly, you are saying that you are getting cli
upload/downloads and c++ sdk downloads to be the same (6 minutes) but c++
uploads are much faster at 3 minutes?  We can't get our values even close
to the cli:
Download S3 to EBS 1.5GB file
CLI:
best cpu: 16%
best speed: 16.5s
overall best: 23% at 18.4s, 20% at 19.4s
transfer manager:
best cpu: 20%
best speed: 26s
overall best: 26% at 57s, 38% at 41s
How do we get them to be the same?
Thank you,
Anna
On Sat, Aug 4, 2018 at 1:32 AM, Andrew Tang<EMAIL_ADDRESS>wrote:

@pavian https://github.com/pavian After several tests with large and
small files. I found in general, upload is faster than download. Also after
a closer looking at the code in core and TransferManager. I think this is
the expected behavior.
When uploading a part. You need to read from local file and send it to
network, when finished sending to network, a part is considered uploaded.
S3 will take care of combining of parts, and you can start sending another
part.
When downloading a part. Server need to read it and send it to network,
when finished sending to network, you will need to write it to local file
before using this buffer to receiving another part. (Note that parallel
write can improve the speed a little bit, but the main cost is still the
additional work, and introduce parallel write will make the code more
complicated, because you will have to maintain multiple file handlers.)
So when downloading, there is an additional work you need to do. That's
why download is slower than upload.
In order to verify that, I commented out the this line in TransferManager
https://github.com/aws/aws-sdk-cpp/blob/29ff65a8b321edd7ae4574bc74f55e
9d26374527/aws-cpp-sdk-transfer/source/transfer/TransferManager.cpp#L896
Then download and upload time of a 40GB file is about the same.
I also tried using Aws CLI to upload and download a 40GB file. Though
upload and download time is about the same, but it's around 6 minutes.
Whereas in C++ SDK, upload takes 3 minutes, and download takes like 6
minutes.
Hope this can help.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-410425257,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AnQi-5LiZ9pRnF8sTMueeQZfO_Xg0fXYks5uNTHagaJpZM4VNh2v
.[image: Web Bug from
https://github.com/notifications/beacon/AnQi-9EyfVoSkO75oT1aljwTEKK19enoks5uNTHagaJpZM4VNh2v.gif]
{"api_version":"1.0","publisher":{"api_key":"
05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":
{"external_key":"github/aws/aws-sdk-cpp","title":"aws/aws-sdk-cpp","subtitle":"GitHub
repository","main_image_url":"https://assets-cdn.github.com/
images/email/message_cards/header.png","avatar_image_url":"
https://assets-cdn.github.com/images/email/message_cards/avatar.png
","action":{"name":"Open in GitHub","url":"https://github.
com/aws/aws-sdk-cpp"}},"updates":{"snippets":[{"icon":"PERSON","message":"@singku
in #910: @pavian After several tests with large and small files. I found in
general, upload is faster than download. Also after a closer looking at the
code in core and TransferManager. I think this is the expected
behavior.\r\n\r\nWhen uploading a part. You need to read from local file
and send it to network, when finished sending to network, a part is
considered uploaded. S3 will take care of combining of parts, and you can
start sending another part.\r\n\r\nWhen downloading a part. Server need to
read it and send it to network, when finished sending to network, you will
need to write it to local file before using this buffer to receiving
another part. (Note that parallel write can improve the speed a little bit,
but the main cost is still the additional work, and introduce parallel
write will make the code more complicated, because you will have to
maintain multiple file handlers.)\r\n\r\nSo when downloading, there is an
additional work you need to do. That's why download is slower than
upload.\r\n\r\nIn order to verify that, I commented out the this line in
TransferManager\r\nhttps://github.com/aws/aws-sdk-cpp/blob/
29ff65a8b321edd7ae4574bc74f55e9d26374527/aws-cpp-sdk-
transfer/source/transfer/TransferManager.cpp#L896\r\n\r\nThen
http://github.com/aws/aws-sdk-cpp/blob/29ff65a8b321edd7ae4574bc74f55e9d26374527/aws-cpp-sdk-transfer/source/transfer/TransferManager.cpp#L896\r\n\r\nThen
download and upload time of a 40GB file is about the same.\r\n\r\nI also
tried using Aws CLI to upload and download a 40GB file. Though upload and
download time is about the same, but it's around 6 minutes. Whereas in C++
SDK, upload takes 3 minutes, and download takes like 6 minutes.\r\n\r\nHope
this can help.\r\n"}],"action":{"name":"View Issue","url":"https://github.
com/aws/aws-sdk-cpp/issues/910#issuecomment-410425257"}}} [ { "@context":
"http://schema.org", "@type": "EmailMessage", "potentialAction": {
"@type": "ViewAction", "target": "https://github.com/aws/aws-
sdk-cpp/issues/910#issuecomment-410425257", "url": "
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-410425257",
"name": "View Issue" }, "description": "View this Issue on GitHub",
"publisher": { "@type": "Organization", "name": "GitHub", "url": "
https://github.com" } }, { "@type": "MessageCard", "@context": "
http://schema.org/extensions", "hideOriginalBody": "false", "originator":
"AF6C5A86-E920-430C-9C59-A73278B5EFEB", "title": "Re: [aws/aws-sdk-cpp]
UNKNOWN s3 error (100) in transfer manager as of Release 1.4.46 (#910)",
"sections": [ { "text": "", "activityTitle": "Andrew Tang",
"activityImage": "https://assets-cdn.github.com/images/email/message_
cards/avatar.png", "activitySubtitle": "@singku", "facts": [ ] } ],
"potentialAction": [ { "name": "Add a comment", "@type": "ActionCard",
"inputs": [ { "isMultiLine": true, "@type": "TextInput", "id":
"IssueComment", "isRequired": false } ], "actions": [ { "name": "Comment",
"@type": "HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueComment",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910,\n"IssueComment":
"{{IssueComment.value}}"\n}" } ] }, { "name": "Close issue", "@type":
"HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueClose",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910\n}" }, { "targets": [ { "os":
"default", "uri": "https://github.com/aws/aws-sdk-cpp/issues/910#
issuecomment-410425257" } ], "@type": "OpenUri", "name": "View on GitHub"
}, { "name": "Unsubscribe", "@type": "HttpPOST", "target": "
https://api.github.com", "body": "{\n"commandName":
"MuteNotification",\n"threadId": 355868079\n}" } ], "themeColor":
"26292E" } ]
Click here
http://esva5.caseware.com/cgi-bin/learn-msg.cgi?id=3EF4828867.ABE82 to
report this message as spam to Caseware E-Mail Security Server ESVA.

--
Anna Siniavina
Working Papers Developer
T: 416·867·9504 x1210 | F: 416·867·1906
<EMAIL_ADDRESS>https://www.caseware.com/
https://linkedin.com/company/caseware-international-inc
https://facebook.com/CaseWare-International-Inc-54418196590/
https://twitter.com/caseware
https://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg

do you mean we have to do some preparations in our code
before downloading this file? Or is it just stating the fact the Transfer
Manager needs to do more work?

TransferManager needs to do additional work to write the parts to local disk.
The test I did was on Linux, with BufferSize to be 50MB and ThreadPool size to be 25. Also I tried with a fairly large file, 40GB, so that the speed can be stabilized.

Hi Andrew,
Can I ask you another TransferManager performance question real quick?
Thanks,
Anna
On Fri, Aug 10, 2018 at 4:04 PM Andrew Tang<EMAIL_ADDRESS>wrote:

do you mean we have to do some preparations in our code
before downloading this file? Or is it just stating the fact the Transfer
Manager needs to do more work?
TransferManager needs to do additional work to write the parts to local
disk.
The test I did was on Linux, with BufferSize to be 50MB and ThreadPool
size to be 25. Also I tried with a fairly large file, 40GB, so that the
speed can be stabilized.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-412190838,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AnQi-zv_jinu8tkt2ZtFy04laUnwRokGks5uPecigaJpZM4VNh2v
.[image: Web Bug from
https://github.com/notifications/beacon/AnQi-9gsRNSIuW2M-ekg7cZenaolQSalks5uPecigaJpZM4VNh2v.gif]
{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/aws/aws-sdk-cpp","title":"aws/aws-sdk-cpp","subtitle":"GitHub
repository","main_image_url":"
https://assets-cdn.github.com/images/email/message_cards/header.png
","avatar_image_url":"
https://assets-cdn.github.com/images/email/message_cards/avatar.png","action":{"name":"Open
in GitHub","url":"https://github.com/aws/aws-sdk-cpp"}},"updates":{"snippets":[{"icon":"PERSON","message":"@singku
in #910: \u003edo you mean we have to do some preparations in our
code\r\nbefore downloading this file? Or is it just stating the fact the
Transfer\r\nManager needs to do more work?\r\n\r\nTransferManager needs to
do additional work to write the parts to local disk.\r\n\r\nThe test I did
was on Linux, with BufferSize to be 50MB and ThreadPool size to be 25. Also
I tried with a fairly large file, 40GB, so that the speed can be
stabilized."}],"action":{"name":"View Issue","url":"
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-412190838"}}} [
{ "@context": "http://schema.org", "@type": "EmailMessage",
"potentialAction": { "@type": "ViewAction", "target": "
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-412190838",
"url": "
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-412190838",
"name": "View Issue" }, "description": "View this Issue on GitHub",
"publisher": { "@type": "Organization", "name": "GitHub", "url": "
https://github.com" } }, { "@type": "MessageCard", "@context": "
http://schema.org/extensions", "hideOriginalBody": "false", "originator":
"AF6C5A86-E920-430C-9C59-A73278B5EFEB", "title": "Re: [aws/aws-sdk-cpp]
UNKNOWN s3 error (100) in transfer manager as of Release 1.4.46 (#910)",
"sections": [ { "text": "", "activityTitle": "Andrew Tang",
"activityImage": "
https://assets-cdn.github.com/images/email/message_cards/avatar.png",
"activitySubtitle": "@singku", "facts": [ ] } ], "potentialAction": [ {
"name": "Add a comment", "@type": "ActionCard", "inputs": [ {
"isMultiLine": true, "@type": "TextInput", "id": "IssueComment",
"isRequired": false } ], "actions": [ { "name": "Comment", "@type":
"HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "IssueComment",\n"repositoryFullName":
"aws/aws-sdk-cpp",\n"issueId": 910,\n"IssueComment":
"{{IssueComment.value}}"\n}" } ] }, { "targets": [ { "os": "default",
"uri": "
https://github.com/aws/aws-sdk-cpp/issues/910#issuecomment-412190838" }
], "@type": "OpenUri", "name": "View on GitHub" }, { "name": "Unsubscribe",
"@type": "HttpPOST", "target": "https://api.github.com", "body":
"{\n"commandName": "MuteNotification",\n"threadId": 355868079\n}" }
], "themeColor": "26292E" } ]
Click here
http://esva4.caseware.com/cgi-bin/learn-msg.cgi?id=2CACE28FB5.AC175 to
report this message as spam to Caseware E-Mail Security Server ESVA.

--
Anna Siniavina
Working Papers Developer
T: 416·867·9504 x1210 | F: 416·867·1906
<EMAIL_ADDRESS>https://www.caseware.com/
https://linkedin.com/company/caseware-international-inc
https://facebook.com/CaseWare-International-Inc-54418196590/
https://twitter.com/caseware
https://youtube.com/channel/UCRlosKtcCJVii9rsOvIB5gg