array_sort should fail if array contains NULL

Bug description
In Presto, array_sort function doesn't allow input arrays that contain nulls.
com.facebook.presto.spi.PrestoException: Array contains elements not supported for comparison
	at com.facebook.presto.operator.scalar.ArraySortFunction$2.compare(ArraySortFunction.java:101)
	at com.facebook.presto.operator.scalar.ArraySortFunction$2.compare(ArraySortFunction.java:91)
...
Caused by: com.facebook.presto.common.NotSupportedException: ARRAY comparison not supported for arrays with null elements
	at com.facebook.presto.common.type.TypeUtils.checkElementNotNull(TypeUtils.java:180)
	at com.facebook.presto.common.type.ArrayType.compareTo(ArrayType.java:124)
	at com.facebook.presto.operator.scalar.ArraySortFunction$2.compare(ArraySortFunction.java:97)

This issue was discovered by running AggregationFuzzer with Presto as source of truth.
CC: @laithsakka @duanmeng
System information
n/a
Relevant logs
No response

@mbasmanova This should be the same issue as we discussed in https://github.com/facebookincubator/velox/issues/6314#issuecomment-1725021636. I will fix this issue after that.

@duanmeng Thanks. This is similar issue. The only difference is array_sort is a scalar function, but the other discussion was about aggregate functions.

This happens in the case of complex types. For example,
Scalar Type
presto> SELECT array_sort(col0) FROM ( VALUES (array [2, null, 1])) AS t(col0);
    _col0
--------------
 [1, 2, null]

Complex Type (The partial success and failure issue should be fixed too)
presto> SELECT array_sort(col0) FROM ( VALUES (array [array [1, 2, 3, 4], array [2, null, 3]])) AS t(col0);
            _col0
------------------------------
 [[1, 2, 3, 4], [2, null, 3]]

presto> SELECT array_sort(col0) FROM ( VALUES (array [array [1, 2, 3, 4], array [1, null, 3]])) AS t(col0);

Query 20230925_113531_00074_6r5h7 failed: Array contains elements not supported for comparison

@duanmeng Presto semantics are a bit challenging. array_sort applied to arrays of complex types with nested nulls may or may not fail depending on whether the sorting logic needs to compare the nulls to decide the order. In Velox, one option would be to (1) check all array elements before sorting and throw if any element contains null; (2) check for contains-nulls only during comparison (Presto's implementation). Option (2) promises to match Presto's semantics (however weird these are), but I'm not sure how to convince myself that the results won't depend on the specific sorting algorithm being used. We could start with option (2) and if we find that this produces inconsistent results, switch to option (1).
presto:di> select array_sort(array[array[1, null], array[2, null]]);
         _col0
------------------------
 [[1, null], [2, null]]
(1 row)

Query 20230926_061441_03625_sq8kq, FINISHED, 1 node
Splits: 5 total, 5 done (100.00%)
174ms [0 rows, 0B] [0 rows/s, 0B/s]

presto:di> select array_sort(array[array[1, null], array[1, 2]]);
Query 20230926_061447_03626_sq8kq failed: Array contains elements not supported for comparison

Opened an issue in prestodb repo to clarify whether existing semantics are intentional: https://github.com/prestodb/presto/issues/20965

@duanmeng Are you still interested to work on this? Anything blocking?

@mbasmanova Yes, I will keep working on it.


[FLINK-32225][Runtime] Merge some task deployment related fields to auxiliary class.

What is the purpose of the change
TaskDeploymentDescriptorFactory#fromExecution needs to retrieve several fields from the ExecutionGraphAccessor. We could introduce a new TaskDeploymentDescriptorHelper to merge all these fields to make the ExecutionGraph/TaskDeploymentDescriptor more readable.
Brief change log

Merge all task deployment related fields to TaskDeploymentDescriptorHelper

Verifying this change
This change is code refactoring without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)

@Thesharing @reswqa, could you please take a look at this PR?

Thanks for inviting me, also cc @wanglijie95.

Thanks @huwh for creating this PR. I have a different idea:
We can create the TaskDeploymentDescriptorFactory outside ExecutionGraph and pass it in through the ctor. TheTaskDeploymentDescriptorFactory provides a createDeploymentDescriptor(Execution execution, ...) method to create the deployment descriptor for an execution.
In this way, the job/graph level configurations such as partitionLocationConstraint and nonFinishedHybridPartitionShouldBeUnknown should be passed into the TaskDeploymentDescriptorFactory instead of ExecutionGraph.
WDYT ? @Thesharing @reswqa

Thanks @huwh for creating this PR. I have a different idea:
We can create the TaskDeploymentDescriptorFactory outside ExecutionGraph and pass it in through the ctor of ExecutionGraph. TheTaskDeploymentDescriptorFactory provides a createDeploymentDescriptor(Execution execution, ...) method to create the deployment descriptor for an execution.
In this way, the job/graph level configurations such as partitionLocationConstraint and nonFinishedHybridPartitionShouldBeUnknown should be passed into the TaskDeploymentDescriptorFactory instead of ExecutionGraph.
WDYT ? @Thesharing @reswqa

It's a great idea. Currently TaskDeploymentDescriptorFactory is created for each execution. But it involves several job-level configurations. This is a bit confusing.
However, this change would introduce a lot of code changes. Maybe we'll need a small design doc to show the scope affected by this refactoring job.

@Thesharing I had a quick look, and found that the ctor and createDeploymentDescriptor are only used in 2 places: Execution and TaskDeploymentDescriptorFactoryTest, it seems the code changes are not quite complicated. Can you help to confirm it @huwh , thanks.

@Thesharing I had a quick look, and found that the ctor and createDeploymentDescriptor are only used in 2 places: Execution and TaskDeploymentDescriptorFactoryTest, it seems the code changes are not quite complicated. Can you help to confirm it @huwh , thanks.

Yes, I overcomplicate this. Thank you for pointing this out. Aside from the initialization of TaskDeploymentDescriptorFactory, the remaining code changes are acceptable.

Thanks @wanglijie95 and @Thesharing  for the analysis.
After making a quick code changes, I think it makes sense to change TaskDeploymentDescriptorFactory to Job Level, so it only contains job level immutable configurations. We create the TaskDeploymentDescriptorFactory in the DefaultExecutionGraphBuilder and pass it to the ExecutionGraph. Executions can use TaskDeploymentDescriptorFactory#createDeploymentDescriptor(Execution execution, ...) to create the DeploymentDescriptor, and in this method, we can put all execution-level configurations, such as InputGateDeploymentDescriptors, into the DeploymentDescriptor.
I will take more tests to confirm this.

Thanks for your review, @wanglijie95 . comments addressed. Please take a look.
Since only "BlobWriter.serializeAndTryOffload" will throw an IOException, I'd like to remove the try catch of creating the DefaultExecutionGraph after adding a separate try catch block to create TaskDeploymentDescriptorFactory.

Thanks a lot, comments addressed and fixup commit squashed.

@Thesharing, do you have any recommendations or concerns?  I'd like to merge this so I can continue working on https://github.com/apache/flink/pull/22674.


How send to broadcast messages

Hi Mukulhase,
How send to broadcast message multiple number, This feature implemented this api. Please help me...

this API available for personal use only.. if you want to broadcast message multiple number, means you want to spamming... so forget about spaming


fix: updates dependencies

What
fix: updates dependencies
react-scripts                    ~4.0.3   →  ~5.0.0
@commitlint/cli                  ~15.0.0  →  ~16.0.0
@commitlint/config-conventional  ~15.0.0  →  ~16.0.0

:tada: This PR is included in version 0.13.5 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


error if BOI is open

if missinghud and isaac are open at the same time then missing hud has an error
'boost::interproccesscommunication error occurred. Check MHUD2 log for details'
how do i fix this?

log has nothing, but when i look at the MHUD2 file it says
boost::interprocess communication error occured: boost::interprocess_exception::library_error (Error Code: 0)


If you tried to load a PyTorch model from a TF 2.0 checkpoint,  please set from_tf=True.

Has anyone encountered this problem?
Unable to load weights from pytorch checkpoint file for './pretrained/pytorch_model-00002-of-00003.bin' at
'./pretrained/pytorch_model-00002-of-00003.bin'. If you tried to load a PyTorch model from a TF 2.0 checkpoint,
please set from_tf=True.

Same problem, not solved yet.


Flush stdout/stderr streams for line-by-line processing by external apps, adds -Wall, Werror compiler flags

Hi again Przemek,
As mentioned in #2, I am the developer of "Rescuezilla", an easy-to-use graphical drop-in replacement to Clonezilla licensed under GPLv3. I am currently implementing Rescuezilla's Image Explorer functionality.
To make sure my minor work so far on partclone-nbd doesn't become like partclone-utils (lots of independent forks) I am creating this pull request with my minor improvements so far.
Random notes:
I like partclone-nbd. It's lightweight and pretty well-written. I like that CMake makes out-of-tree builds trivial compared to partclone-utils' autoconf.
partclone-nbd definitely handles NTFS filesystems better than the current partclone-utils forks, so Rescuezilla v2.1 will use partclone-nbd. However I adapted a partclone-utils test script and found that partclone-nbd does appear to have issues around some filesystem types. partclone-utils happens to also support "ntfsclone" which is used by Clonezilla and so is of great interest to Rescuezilla. Because of partclone-utils' ntfsclone support, I hope to do the bulk of my future Image Explorer-related development on the partclone-utils rather than partclone-nbd.

OK, I've updated my usage of char* to u8*, rebased onto your most recent and repushed the changes.


Unable to select date/month/year from native date picker object

I am facing a problem where i am not able to select date/month/year from native date picker using selendroid
i tried this .  driver.findElement(By.id("month")).sendKeys("Oct");  But it's not working
from irc:
lukeis
charbrandeis: is this in a native component? with a DatePicker object? (not a webview)  if so… I don't think we've explicitly handled that code path in selendroid, I don't see any code handling DatePicker objects (nor calling the method needed to be called… updateDate)
charbrandeis:
yes, it's in a native component with a date picker object..
== chhsiung has quit
lukeis
ok, then I don't see that being handled in selendroid currently

Hi
I created a script using samsung galaxy S5 and the script works fine on
S5.But if i try to run the same script on Nexus 7,it couldn't interact with
a button on the alert.Can you please look into this?
i am attaching a screenshot from Nexus 7 .
Thanks
Charles

cause datePicker has other key reaction on specific mobile phone brands and android "versions". Better option to push "button1" is to use a tap() function.

This is pretty strange. I am able to select date picker date if I am using below capability instead of uiautomator2.
capability name="automationName" value="Appium"
removed below
capability name="automationName" value="uiautomator2"
But now for me Allow Permission is blocker.
Can you suggest?


Disable TCP Checksum Offloading(TCO) on RHEL for vSphere

Description of changes:
Recent updates with vmxnet3 to support udp offloading for Vxlan and geneve introduced bugs with TCP checksum offloading that computes incorrect checksum for packets when TCO is enabled. Multiple fixes and subsequent bugs from those fixes where introduced which lead to enabling TCO on RHEL when using vmware drops packets due to incorrectly computed checksums. This changes disables TCO explicitly for RHEL + vSphere combination.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/lgtm
/approve


remove batch/v1beta1 and policy/v1beta1 apis for k8s 1.25

What does this PR do?
Remove references to batch/v1beta1 and policy/v1beta1 in preparation for k8s 1.25 upgrade

Motivation
Blocks upgrade to k8s 1.25

Additional Notes

Possible Drawbacks / Trade-offs
Will not work on k8s 1.20 or earlier

Describe how to test/QA your changes

Reviewer's Checklist

[ ] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
[ ] Use the major_change label if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.
[ ] A release note has been added or the changelog/no-changelog label has been applied.
[ ] Changed code has automated tests for its functionality.
[ ] Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
[ ] At least one team/.. label has been applied, indicating the team(s) that should QA this change.
[ ] If applicable, docs team has been notified or an issue has been opened on the documentation repo.
[ ] If applicable, the need-change/operator and need-change/helm labels have been applied.
[x] If applicable, the k8s/1.21 label, indicating the lowest Kubernetes version compatible with this feature.
[ ] If applicable, the config template has been updated.

/label k8s/1.21

Hey @a7i,
Thanks for your contribution. Unfortunately we cannot drop support for older Kubernetes as-is, what we usually do is to implement support for both older and newer objects until we can't (because lib upgrade is necessary and it removes support for older objects), in which case we need to send a communication to customer to warn them early about upcoming breaking change.

Hey @a7i,
Thanks for your contribution. Unfortunately we cannot drop support for older Kubernetes as-is, what we usually do is to implement support for both older and newer objects until we can't (because lib upgrade is necessary and it removes support for older objects), in which case we need to send a communication to customer to warn them early about upcoming breaking change.

@vboulineau What do customers that need to upgrade to 1.25 do? Given that 1.27 is around the corner, it makes sense to drop APIs that are no longer supported on 1.25 and replace them with those that were added in 1.21 (which came out almost two years ago).
What should be the proper changes then? Happy to apply changes based on your feedback.

So we already have the logic to use the newer resources if available, but we fallback to previous version if not available, it allows us to support a wide range of versions (currently it's handled automatically in KSM, but through configuration in orchestrator check)
The code you're removing is handling the older versions and only used if newer versions is not available.
For 1.25+ users there's nothing to do, it should just work OOTB.

I see, thanks for the explanation. Going to close this PR then


add rdzv_backend parameter to DDPJobDefinition

This change allows the user to define the rdzv_backend to be used when creating a DDPJobDefinition.
This change is required because currently, the backend is hard coded to "static" in torchx dist.py. Howver, depending on whether or not we are using the Ray Scheduler or the MCAD scheduler, the required backend needs to change as well.
This PR will be paired with an upcoming change to torchx/OCP that accounts for the different backends submitted here.

@MichaelClifford Can you rebase on main?

@anishasthana rebased :)


Agent+config support together with Marta's imageVector support and other fixes (unitests)

Supersedes #27
More description in above PR and following issues:
Resolves #1, resolves #2, resolves #3, resolves #11 and possibly resolves #28
Changes/features:
configs:

deploy CRI-RM agent and it Adjustments CRDs - more docs here
dynamic configuration update by agent - docs here
configurations files (default/force/per node/per group) can be passed through ControllerDeployment as well as by providerConfig #11  (handle force.cfg - when provided will overwrite fallback and agent )

fixes:

fix issues with imageVector

other enhacmentes:

update gardener to 1.56.0 and cri-rm to 0.7.2
remove "remove daemonset" and replace it with trap for SIGTERM
rename targets in Makefile: build-images and push-images optimization (caching)
Makefile: replace VERSION with TAG
replace v2.isvimgreg.com with kind-local-provider localhost:5001
refactor code (restructure) - move actuator/configs/healthchecks to separate packages,
add some tests for configs/actuator

TODO (after Marta review)

[x] document default values from charts.yaml as examples to tested outside extension (e.g. helm template) no to be commited to repo

[x] move charts to internal

[x] consider healthcheck to pkg/controller/ and actuator to pkg/controller/lifecycle

[x] remove TODOs/comments

[x] enable logging for node-agent (need addtional annotation/label from Damin PR)

[x] add license headers

[x] disable zaplogger development option to False

Not done, moved out of this PR:

consider removing fallback.cfg.sample as default - because it enambles TopologyAware policy which is different from built-in
tests on SLES
test what happens when you provide broken config

@mmucek95 I addressed most of the comments in this commit - you may take a look: https://github.com/intel/gardener-extension-cri-resmgr/pull/33/commits/2cf9e056c353602b25831a3e1f9480161b258361

marking tests as done - will be done on master when merged before release
moving "default config" work out of this PR https://github.com/intel/gardener-extension-cri-resmgr/issues/44


math/big: fast path for Cmp if same

I noticed that math/big.Int Cmp method does not have a fast path for the case if x and y are the same. Wouldn't it be beneficial to have there something along the lines:
if x == y {
    r = 0
    return
}

Mul seems to do similar.
Same applies for big.Rat and big.Float.

Note that Mul has that check because if the two numbers are the same you can do squaring, which is faster than a fully general multiplication. This is a fast-path that cannot be easily implemented by the caller since a fast squaring routine in not trivial to implement. On the other hand, it's easy to avoid an Int.Cmp call when the pointers are the same, and you can do that in user code.

Seems reasonable, if it fires. You might try putting it in, adding some logging when it does/doesn’t fire, and then running make.bash. If it fires a non-trivial percentage of times it definitely seems worthwhile.

Spoke to @griesemer; go ahead, for Go 1.14.

Is anyone working on it? I'm looking to start contributing, and need an easy task for a warmup.


Add VLAN tags to PCAP meta data

This PR makes 802.1Q VLAN tag information queryable. VLAN parsing encompasses single tagging and double tagging. The first 802.1Q header after the MAC address is the outer VLAN tag and an optional, sub-sequent VLAN tag the inner VLAN tag:

To expose the tags, the packet schema now includes a new sub-record vlan with two fields outer and inner. For example, to query all non-zero outer VLAN tags, you can know write vlan.outer > 0.
:memo: Checklist

[x] How do I add a changelog entry for plugins? IIRC we have just one CHANGELOG.md file in the top-level plugin directory. There's no changelog directory from where the master changelog gets generated, as we do for VAST itself.
[x] Update docs.tenzir.com/vast.

💬 Discussion

[x] Should we remove --disable-community-id flag? We are unconditionally generating Community IDs now. The configuration flag only determines whether we add the extra string to the table slice.

:dart: Review Instructions
Commit-by-commit.
Appreciate that the decapsulation logic has now moved into a separate frame abstraction that provides a factory function frame::make. Note that detailed error handling is not important because we're simply skipping invalid frames, which is why the function only returns a std::optional<frame>.
The last refactoring commit only shuffles code around but doesn't make any changes to the big-picture logic. The change adds several bounds checks to make the overall parsing safer as well.

How do I add a changelog entry for plugins? IIRC we have just one CHANGELOG.md file in the top-level plugin directory. There's no changelog directory from where the master changelog gets generated, as we do for VAST itself.

Just modify the plugin's changelog file, and bump its version. In this case it's a breaking change due to the removal of the --disable-community-id flag.
We probably need two separate changelog entries here.

Update docs.tenzir.com/vast.

The graphic from this PR would probably also do nicely on there.

Should we remove --disable-community-id flag?

I think so, yes.

Just modify the plugin's changelog file, and bump its version. In this case it's a breaking change due to the removal of the --disable-community-id flag.

Actually, the behavior of the flag just changed in that it no longer changes the schema. I decided to keep it.

@dominiklohmann there's an issue with fmt 8.1 on macOS CI:
 VAST_DEBUG("skipping packet of type {:#0x}", frame->type);

This compiles fine locally, but the enum frame->type is apparently not auto-printable in CI. How should this be fixed?

This compiles fine locally, but the enum frame->type is apparently not auto-printable in CI. How should this be fixed?

It's an enum class, so you need to write a custom formatter for it. This has been the case since {fmt} 8.0, it's not a breaking change in 8.1.

@dominiklohmann does this look reasonable?
diff --git a/plugins/pcap/main.cpp b/plugins/pcap/main.cpp
index f36e6a2b9..fe74c8e0f 100644
--- a/plugins/pcap/main.cpp
+++ b/plugins/pcap/main.cpp
@@ -143,6 +143,28 @@ enum class frame_type : char {
   vxlan = '\x14'
 };

+} // namespace vast::plugins::pcap
+
+namespace fmt {
+
+template <>
+struct formatter<vast::plugins::pcap::frame_type> {
+  template <typename ParseContext>
+  constexpr auto parse(ParseContext& ctx) {
+    return ctx.begin();
+  }
+
+  template <typename FormatContext>
+  auto format(vast::plugins::pcap::frame_type value, FormatContext& ctx) {
+    // TODO(C++23): std::to_underlying instead of static_cast.
+    return format_to(ctx.out(), "{:#0x}", static_cast<char>(value));
+  }
+};
+
+} // namespace fmt
+
+namespace vast::plugins::pcap {
+
 /// An 802.3 Ethernet frame.
 struct frame {
   static std::optional<frame>
@@ -505,7 +527,7 @@ protected:
       auto packet = packet::make(frame->payload, frame->type);
       if (!packet) {
         ++discard_count_;
-        VAST_DEBUG("skipping packet of type {:#0x}", frame->type);
+        VAST_DEBUG("skipping packet of type {}", frame->type);
         continue;
       }
       // Parse layer 4.

Yes, except for the style nit s/typename/class in the templates. You can also inherit from the existing formatter for char, which makes everything even simpler and gains you support for {:x} for free.

@dominiklohmann can you explain to me the {fmt} error message?

@dominiklohmann can you explain to me the {fmt} error message?

I think your formatter shouldn't be a const member function. It can be constexpr, though, but the parent format function is definitely not marked const.

I think your formatter shouldn't be a const member function. It can be constexpr, though, but the parent format function is definitely not marked const.

Hm, I copied the example almost 1-to-1 from the {fmt} docs.

Now you're trading macOS for Debian CI. 😆


TST: Conda CI build not using latest wheels?

See: https://github.com/pydata/pandas-gbq/pull/247#discussion_r259496042
The Conda build is likely using an old version of pandas, even though it's supposed to be using pre-release wheels.

I do see pandas 0.25 nightly builds on https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a83.ssl.cf2.rackcdn.com/ so it's not as simple as changing the URL. Next steps to debug should probably be to print out the version of pandas in the https://github.com/pydata/pandas-gbq/blob/master/ci/run_conda.sh script.
Actually, I think I figured it out.
https://github.com/pydata/pandas-gbq/blob/8a263453e66b29dcfaed018bd0714d669bfa44e2/.circleci/config.yml#L53
should set PANDAS=MASTER if we want to use the nightlies on the conda build. Then again, maybe it's still useful to have some minimum version of pandas build. So if that's currently the oldest  supported, we should make sure one of the pip builds has that pandas version.


Update the README goals and scope for NU5 mainnet activation

Motivation
Update Zebra's readme with the latest release goals and scope.
Solution

[ ] Wait until all the NU5 validation tickets are done
[ ] Update the goals
[ ] Update the scope
[ ] Check the README for other outdated info

We did as part of the beta.10 release, though we might want to review the documentation before the release candidate (I'll create a another ticket for that)


[WIP] Drop retention policy

Fixes #5653.
This is done, but I'll wait for #5663 to be merged, then I'll pluck out the small changes to the query executor in this PR and put them into the query executor in its new location.

Adds methods on the tsdb.Store for deleting shards in a retention policy, and cleaning up said retention policy.
Refactors tsdb.Store.DeleteDatabase so that it does not require the ids of the shards within the database.
Wires up deleting retention policy with the query executor (for the moment).

Looks like this will fix #5394 as well.

One last item.  But :+1: after that.

Discussed defensive nil shard checks with @joelegasse  and @jwilder, and we decided to remove them as they should be redundant.


Make usage of the new stop semantics to properly shutdown the plugin

Implements stop to return from run according to https://github.com/elastic/logstash/issues/3210
Depends on https://github.com/elastic/logstash-devutils/pull/32 and https://github.com/elastic/logstash/pull/3812
Fixes #14
Depends on Stud::Task being properly setup to finish the task when requested (https://github.com/jordansissel/ruby-stud/issues/26)

doesn't run for me:
  1) LogStash::Inputs::Irc behaves like an interruptible input plugin #stop returns from run
     Failure/Error: expect(plugin_thread).to be_alive
       expected `#<Thread:0x72e34f77 dead>.alive?` to return true, got false
     Shared Example Group: "an interruptible input plugin" called from ./spec/inputs/irc_spec.rb:6
     # /Users/joaoduarte/projects/logstash-devutils/lib/logstash/devutils/rspec/shared_examples.rb:14:in `(root)'
     # /Users/joaoduarte/.rvm/gems/jruby-1.7.19/gems/rspec-wait-0.0.7/lib/rspec/wait.rb:46:in `(root)'

@jsvd yes! this happen with .22 Stud, but not with .21, working on make it pass again .

#<NameError: uninitialized constant Stud::Task>
org/jruby/RubyModule.java:2726:in `const_missing'
/Users/joaoduarte/all_logstash_plugins/logstash-input-irc/lib/logstash/inputs/irc.rb:91:in `run'
/Users/joaoduarte/projects/logstash-devutils/lib/logstash/devutils/rspec/shared_examples.rb:11:in `(root)'

@jsvd yeah! just found it now, I guess I was getting stud .21 from logstash-core or anyother place, but not the .22  :+1:

@jsvd should work as expected now.

LGTM, can you squash and merge?


Add a connection pool

This commit adds a basic connection pool implementation. It is still extremely WIP but includes breaking changes that must probably be taken care of before the 1.0.0 tag (see the comment below).
I'll add  as many comments as possible tonight to explain parts that need to.
Let me know what you think! 🙂
There are many things to fix before the PR is truly ready for review/merge:
Making HTTPBin ready for pooling tests:

[x] Add delayed responses (enables testing the creation of new connections by the connection providers when existing ones are in-use)
[x] Add connections that close themselves after they served a request
[x] Channels that close themselves after a certain delay (to test how the pool reacts when a stored channel closes)
[x] Simulate connection delays (useful for performance testing of pooled vs non-pooled)

Misc issues/improvements

[x] Fix issue where an existing ConnectionProvider will not add the proper SSL handler for https connections
[x] Avoiding acquiring Locks for as long as its currently the case

Testing

[x] Stress tests for the pool
[x] Check the respect of the maximum number of concurrent connections for HTTP/1.1

Is this needed?

Make a performance comparator between pooled and non-pooled client

Not in this commit:
HTTP/2:

Add HTTP/2 support to HTTPBin
Finish and test the implementation of the HTTP/2 connection provider
Correct detection of HTTP/2 when its available

@Lukasa

The general shape of this seems good, but I think we need a lot more commenting explaining what the strategy is here. There's a decent amount of code duplication too that could be cleaned up. However, it's a definite great start, well done!

Thank you! I'll add comments to explain everything.
@Lukasa @t089  I will fix the parts where you noticed issues, thank you for your feedback! There is also quite a lot of code that needs to be cleaned.
The main goal of doing this draft PR right now is to be ensure breaking changes can be added before the 1.0.0 tag, even if actual pooling was only postponed for after the release. From what I remember of previous discussions, being able to add pooling entirely transparently in a SemVer minor version was something that would be nice to have, on this subject, I think breaking changes might need to be merged before the release tag.
I think #102 already adresses what's needed for thread safety issues and or the pool to have enough informations to make the better decisions for performance, but there is another part to consider: retried requests.
Since a connection pool augments the chances of failure due to connections being kept alive for longer, having an automatic retry strategy is needed. Note that connection "failure" for non-leased connections already has a basic minimal support in the current implementation: if a connection is closed, it is directly removed from the pool.
When the implementation becomes better, the TaskHandler will probably want to detect when an in-execution task fails and retry it automatically  if possible as we had discussed with @Lukasa and @weissi. AFAICT this mechanism might need breaking changes to the HTTPClientResponseDelegate: when a request is re-tried, some delegates will need to reset their internal state, so a delegate method such as willRetryRequest(...) -> EventLoopFuture<Void> might be needed.
About this I have two ideas:

Make a breaking change before the 1.0.0 tag by requiring all delegate implementations to explicitly handle retries, this way a connection pool with retries can be added entirely transparently in the future.
Make no change to the current requirements of the delegate and add an an opt-in change in the future (by making a new type of delegate for retries or adding a default implementation): in this case, a connection pool could still be added, but no SemVer minor change could automatically add the retry mechanism for everyone except if the library user explicitly starts handling the retry.

I'm not really sure of what the best thing to do would be so I would appreciate to have your opinion about this, if needed I can also open it as a separate issue.

Make a breaking change before the 1.0.0 tag by requiring all delegate implementations to explicitly handle retries, this way a connection pool with retries can be added entirely transparently in the future.

I'm quite tempted by this approach, as it allows us to potentially shim a solution in by having an automatic-retry delegate that can be composed with a user-provided delegate in some way. In general, trying to do too much is likely to go badly for us. I think it may be a good idea to explicitly say that retries must be handled by delegates (and to add some API surface for doing so), as it allows us to punt on making the decision any further.
However, @weissi may be opposed to adding more API for this.

I think it may be a good idea to explicitly say that retries must be handled by delegates (and to add some API surface for doing so)

Just to be sure of what you mean exactly by "handled by delegates": do you mean something similar to willRetryRequest(...)? Or do you mean an even more "manual" handling?

The https://github.com/swift-server/async-http-client/pull/105/commits/0d33412525570bc70582fcf1eb343f1483799757 commit adds new options (passed as Connection: close and X-internal-delay: [delayInMilliseconds] HTTP headers to control how requests are served. Some new tests already take advantage of it, for instance testStressGetHttps and testResponseConnectionCloseGet. The commit also fixes misc issues and documentation issues.

I'm quite tempted by this approach, as it allows us to potentially shim a solution in by having an automatic-retry delegate that can be composed with a user-provided delegate in some way. In general, trying to do too much is likely to go badly for us. I think it may be a good idea to explicitly say that retries must be handled by delegates (and to add some API surface for doing so), as it allows us to punt on making the decision any further.
However, @weissi may be opposed to adding more API for this.

I think I'm down with that. I think we should make automatic retrying very easy for somebody to implement in their own delegate. That could work either with composition as @Lukasa proposed or possibly with a default implementation?

Im trying to bring this back to light because it speeds up AsyncHTTPRequest considerably.

Test Case '-[AthleticNetworkTests.PerformanceTests testURLSession]' started.

Im trying to bring this back to light because it speeds up AsyncHTTPRequest considerably.

The PR is now ready for review! 🙂
cc @weissi @Lukasa @t089 @tomerd @artemredkin
Thanks to the precious help of @kylebrowning this PR now includes the latest changes of master and all tests are passing, the benchmarks are really insightful too and let us ensure that these changes are beneficial to the performance.
What do you think about the connection pool implementation? What can be improved / made clearer?
Thank you!
A few things to check in all cases before merging:

[ ] Check there is no problematic reference cycle between connections and the pool
[ ] Remove HTTP/2 related pooling, this PR only focuses on HTTP/1.1

The latest PRs have fixed an issue that caused the pool to be unused, this was the main reason why there was so much little code coverage as you noticed @weissi, this is now much better but still improvable.
@tomerd Thank you for your feedback, I fixed multiple points in the latest pushes, still have to see what can be improved about locks.

Here are additional tests results now that some bugs have been solved, they are using these performance tests.
Test results (release, 250 requests capped at a maximum of 8 concurrent requests)

- Without pooling:
    - AsyncHTTPClient: 1.888 s (local), 13.4s (online)
    - URLSession : 0.516 s (local), 5s (online)
- Pooling:
    - AsyncHTTPClient : 0.39 s (local),  3.3s (online)
    - URLSession : 0.52 s (local), 4.4 s (online)

Results:
- AsyncHTTPClient got 4 to 5x faster (on an HTTPS connection)
- AsyncHTTPClient is now around 30% faster than URLSession on macOS

Hi guys, really excited by seeing this work going ahead. We have an issue on aws-sdk-swift reporting less than optimal speeds on AWS lambda when making api calls (which require an HTTP request). They were approx 5-6x slower then Node.js. When we looked at Node.js times. We found connection reuse improve http request times by approx 4x. So I am hoping that connection reuse in async-http-client will improve aws-sdk-swift times in a similar way.

I setup a lambda using this branch. Results are fantastic. After the first request, I'm seeing 8x improvement on subsequent requests. Great work.
I am getting the occasional crash though, this may be something I am though doing. I'll investigate

@adam-fowler Thank you for your interest in this PR and for trying this branch! 🙂
It's great to know it actually improves the performance, if you can, don't hesitate to post what is the line where it crashed in case this is an issue I'm not aware of yet.
A likely cause of bugs right now is that there is that there is still a few threading issues that need to be fixed, rewriting this synchronisation logic will hopefully also make the code clearer.

Not being able to get crash info from an AWS lambda, I replicated the code on mac. Here are details of one crash. It crashes with a EXC_BAD_ACCESS on line 356 of function closeIfNeeded()
self.parentPool._connectionProviders[key] = nil

This is the callstack
#0	0x00007fff72f80eb0 in swift_isUniquelyReferenced_nonNull_native ()
#1	0x00007fff72cd29b1 in Dictionary._Variant.removeValue(forKey:) ()
#2	0x00007fff72ebf223 in specialized Dictionary.subscript.setter ()
#3	0x00007fff72c9d922 in Dictionary.subscript.setter ()
#4	0x000000011314111a in ConnectionPool.HTTP1ConnectionProvider.closeIfNeeded() at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/async-http-client/Sources/AsyncHTTPClient/ConnectionPool.swift:356
#5	0x000000011313c814 in ConnectionPool.HTTP1ConnectionProvider.availableConnections.didset at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/async-http-client/Sources/AsyncHTTPClient/ConnectionPool.swift:241
#6	0x000000011313ca54 in ConnectionPool.HTTP1ConnectionProvider.availableConnections.setter ()
#7	0x000000011313cb90 in ConnectionPool.HTTP1ConnectionProvider.availableConnections.modify ()
#8	0x0000000113140e2d in closure #2 in closure #1 in ConnectionPool.HTTP1ConnectionProvider.closeAllConnections() at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/async-http-client/Sources/AsyncHTTPClient/ConnectionPool.swift:349
#9	0x0000000113140e5c in thunk for @escaping @callee_guaranteed () -> () ()
#10	0x0000000113142e21 in partial apply for thunk for @escaping @callee_guaranteed () -> () ()
#11	0x000000011323b06c in thunk for @escaping @callee_guaranteed (@in_guaranteed A) -> (@out ()) ()
#12	0x000000011323cb47 in closure #1 in EventLoopFuture.whenSuccess(_:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoopFuture.swift:704
#13	0x0000000113234db0 in thunk for @escaping @callee_guaranteed () -> (@owned CallbackList) ()
#14	0x00000001132360d1 in thunk for @escaping @callee_guaranteed () -> (@owned CallbackList)partial apply ()
#15	0x0000000113234793 in thunk for @escaping @callee_guaranteed () -> (@out CallbackList) ()
#16	0x0000000113235f74 in CallbackList._run() at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoopFuture.swift:109
#17	0x0000000113236938 in EventLoopPromise._resolve(value:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoopFuture.swift:230
#18	0x0000000113236821 in EventLoopPromise.succeed(_:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoopFuture.swift:175
#19	0x00000001132461b2 in processResult #1 <A><A1>(_:_:) in static EventLoopFuture._reduceCompletions0<A>(_:_:_:onResult:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoopFuture.swift:1262
#20	0x000000011324626c in closure #1 in static EventLoopFuture._reduceCompletions0<A>(_:_:_:onResult:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoopFuture.swift:1275
#21	0x0000000113247ea6 in partial apply for closure #1 in static EventLoopFuture._reduceCompletions0<A>(_:_:_:onResult:) ()
#22	0x000000011323d1a0 in closure #1 in EventLoopFuture.whenComplete(_:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoopFuture.swift:742
#23	0x00000001132355cf in CallbackList._run() at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoopFuture.swift:83
#24	0x0000000113236ec5 in closure #1 in EventLoopPromise._resolve(value:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoopFuture.swift:233
#25	0x00000001132291fc in thunk for @escaping @callee_guaranteed () -> () ()
#26	0x0000000113233821 in partial apply for thunk for @escaping @callee_guaranteed () -> () ()
#27	0x000000011322921c in thunk for @escaping @callee_guaranteed () -> (@out ()) ()
#28	0x0000000113229267 in closure #3 in SelectableEventLoop.run() at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoop.swift:943
#29	0x0000000113196f9f in thunk for @callee_guaranteed () -> (@error @owned Error) ()
#30	0x0000000113230234 in thunk for @callee_guaranteed () -> (@error @owned Error)partial apply ()
#31	0x0000000113224b82 in closure #1 in withAutoReleasePool<A>(_:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoop.swift:640
#32	0x000000011323384f in partial apply for closure #1 in withAutoReleasePool<A>(_:) ()
#33	0x00007fff734abf0e in autoreleasepool<A>(invoking:) ()
#34	0x0000000113224b29 in withAutoReleasePool<A>(_:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoop.swift:639
#35	0x0000000113227e3b in SelectableEventLoop.run() at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoop.swift:942
#36	0x000000011322b40b in closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:initializer:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoop.swift:1114
#37	0x000000011323358a in partial apply for closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:initializer:) ()
#38	0x000000011322ba5f in thunk for @escaping @callee_guaranteed (@guaranteed NIOThread) -> () ()
#39	0x00000001132db931 in partial apply for thunk for @escaping @callee_guaranteed (@guaranteed NIOThread) -> () ()
#40	0x00000001132dbceb in closure #1 in static NIOThread.spawnAndRun(name:detachThread:body:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/Thread.swift:110
#41	0x00000001132dbd69 in @objc closure #1 in static NIOThread.spawnAndRun(name:detachThread:body:) ()
#42	0x00007fff73a05d76 in _pthread_start ()
#43	0x00007fff73a025d7 in thread_start ()

Unfortunately the debugger died on me before I could get more info

Second crash:
Fatal error: Unexpectedly found nil while unwrapping an Optional value
Line 156, in CircularBuffer.subscript(position:)
return self._buffer[position.backingIndex]!

Callstack
#0	0x00007fff72f2f460 in _swift_runtime_on_report ()
#1	0x00007fff72fa7223 in _swift_stdlib_reportFatalErrorInFile ()
#2	0x00007fff72ebc18e in specialized closure #1 in closure #1 in closure #1 in _assertionFailure(_:_:file:line:flags:) ()
#3	0x00007fff72ebc307 in specialized closure #1 in closure #1 in _assertionFailure(_:_:file:line:flags:) ()
#4	0x00007fff72c8bf0c in specialized String.withUTF8<A>(_:) ()
#5	0x00007fff72e8e710 in specialized _assertionFailure(_:_:file:line:flags:) ()
#6	0x00007fff72c8b4c9 in _assertionFailure(_:_:file:line:flags:) ()
#7	0x00000001139fb83b in CircularBuffer.subscript.getter at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/CircularBuffer.swift:156
#8	0x00000001139fb95f in CircularBuffer.subscript.read ()
#9	0x00000001139fcb14 in protocol witness for Collection.subscript.read in conformance CircularBuffer<A> ()
#10	0x00007fff72c6dec5 in MutableCollection.swapAt(_:_:) ()
#11	0x00000001139462e9 in closure #1 in ConnectionPool.HTTP1ConnectionProvider.configureClose(of:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/async-http-client/Sources/AsyncHTTPClient/ConnectionPool.swift:301
#12	0x00000001139464e0 in thunk for @escaping @callee_guaranteed (@guaranteed Result<(), Error>) -> () ()
#13	0x0000000113a441c0 in closure #1 in EventLoopFuture.whenComplete(_:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoopFuture.swift:742
#14	0x0000000113a3c5ef in CallbackList._run() at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoopFuture.swift:83
#15	0x0000000113a3d958 in EventLoopPromise._resolve(value:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoopFuture.swift:230
#16	0x0000000113a3d841 in EventLoopPromise.succeed(_:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoopFuture.swift:175
#17	0x00000001139a377c in closure #4 in BaseSocketChannel.close0(error:mode:promise:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/BaseSocketChannel.swift:771
#18	0x0000000113a3021c in thunk for @escaping @callee_guaranteed () -> () ()
#19	0x0000000113a3a841 in partial apply for thunk for @escaping @callee_guaranteed () -> () ()
#20	0x0000000113a3023c in thunk for @escaping @callee_guaranteed () -> (@out ()) ()
#21	0x0000000113a30287 in closure #3 in SelectableEventLoop.run() at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoop.swift:943
#22	0x000000011399dfbf in thunk for @callee_guaranteed () -> (@error @owned Error) ()
#23	0x0000000113a37254 in thunk for @callee_guaranteed () -> (@error @owned Error)partial apply ()
#24	0x0000000113a2bba2 in closure #1 in withAutoReleasePool<A>(_:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoop.swift:640
#25	0x0000000113a3a86f in partial apply for closure #1 in withAutoReleasePool<A>(_:) ()
#26	0x00007fff734abf0e in autoreleasepool<A>(invoking:) ()
#27	0x0000000113a2bb49 in withAutoReleasePool<A>(_:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoop.swift:639
#28	0x0000000113a2ee5b in SelectableEventLoop.run() at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoop.swift:942
#29	0x0000000113a3242b in closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:initializer:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/EventLoop.swift:1114
#30	0x0000000113a3a5aa in partial apply for closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:initializer:) ()
#31	0x0000000113a32a7f in thunk for @escaping @callee_guaranteed (@guaranteed NIOThread) -> () ()
#32	0x0000000113ae2951 in partial apply for thunk for @escaping @callee_guaranteed (@guaranteed NIOThread) -> () ()
#33	0x0000000113ae2d0b in closure #1 in static NIOThread.spawnAndRun(name:detachThread:body:) at /Users/adamfowler/Library/Developer/Xcode/DerivedData/aws-sdk-swift-exwegbtchkficlgchleuoxvuyiig/SourcePackages/checkouts/swift-nio/Sources/NIO/Thread.swift:110
#34	0x0000000113ae2d89 in @objc closure #1 in static NIOThread.spawnAndRun(name:detachThread:body:) ()
#35	0x00007fff73a05d76 in _pthread_start ()
#36	0x00007fff73a025d7 in thread_start ()

The offending line in your code is 301 in HTTP1ConnectionProvider.configureClose()
self.availableConnections.swapAt(self.availableConnections.startIndex, indexToRemove)

I managed to replicate the above crashes with the following code. Repeat the flatMap approx 16 times. Given the crashes are most likely threading issues it doesn't crash 100% of the time, but it is pretty consistent.
        let eventLoopGroup = MultiThreadedEventLoopGroup(numberOfThreads: System.coreCount)
        let httpClient = HTTPClient(eventLoopGroupProvider: .shared(eventLoopGroup))

        _ = try httpClient.get(url: "https://httpbin.org/get")
            .flatMap { _ in
                return httpClient.get(url: "https://httpbin.org/get")
            }
            .flatMap { _ in
                return httpClient.get(url: "https://httpbin.org/get")
            }
...
            .flatMap { _ in
                return httpClient.get(url: "https://httpbin.org/get")
        }.wait()

        try httpClient.syncShutdown()

Thanks @adam-fowler , one problem that will lead to crashes is definitely  this: https://github.com/swift-server/async-http-client/pull/105/files#r354349720

@adam-fowler Thank you for the stack trace and for the reproduction code, it looks really close to the bugs I already had during some tests and indeed due to the existing threading issues.

@artemredkin Can we add this as a milestone for 1.1.0 since it did not make the cut for 1.0?
Theres a lot of people looking forward to this feature and it'll increase adoption, im sure of it.

@kylebrowning done

@swift-server-bot add to whitelist

@swift-server-bot test this please

The latest commit introduces a new HTTP1ConnectionProvider.State type that enables a better isolation between the "core pooling logic" (ie: given the current state, what should be done when a connection is requested/released/closed) and the code that takes care of creating and leasing connections through calls to SwiftNIO types.
This new approach should improve code clarity and fix the existing threading issues that led to crashes such as the one @adam-fowler encountered, fix the issue @tomerd mentioned about locks that were taken for too long which could even cause deadlocks as noticed by @weissi.
There are still some // TODO:s around that must be fixed plus some code reviews that haven't been addressed yet, but the stability should be much better now so let me know if it fixes all issues you encountered up until right now or if you have any feedback on the latest commit.
@weissi Thanks again for suggesting taking this approach! 🙂

great stuff @adtrevor !

@swift-server-bot test this please

@swift-server-bot test this please

@Lukasa Thank you! 🙂 Alongside the synchronisation and code duplication issues I also know of a few parts in the code where there might be a few missing stuff, but happy to see we are getting closer to being able to merge this !

@adtrevor this guy has a small merge conflict now. Mind rebasing to or merging master and fixing it?

@weissi Indeed, thank you.
Currently working on this in order to correctly merge the changes of #151 and trying to figure out the best approach to generating a Key from either kind of requests, also checking if there is any special handling that should be done regarding proxies and ssl.
The latest commits (diff) made a few changes to the client:

Correct implementation of syncShutdown(requiresCleanClose: Bool), a new method that throws if there were still active tasks when the method was called. Note that a call to syncShutdown(requiresCleanClose: true) will always close the client as best as it can regardless of wether or not it was a clean shutdown (ie: it will first close everything before actually throwing).
Now, syncShutdown cancels any remaining active tasks, this change might impact users who didn't explicitly wait on tasks completion to shutdown the client and still expected to get a successful result from their requests after client shutdown, now it will fail with HTTPClientError.cancelled.

@swift-server-bot test this please

Looks like there might be an issue with TSan on 5.1? It seems like a TSan issue is reported before the tests actually start?

@swift-server-bot add to whitelist

Thanks for the new test of https://github.com/swift-server/async-http-client/pull/105/commits/92aedbddebc85bd9e378e6438d73b93b958c9f45 @weissi

@artemredkin why not start with fixed and add config later in a separate PR?

@artemredkin Thanks!
Latest commit fixes the bugs found by @weissi

@Lukasa Thanks for the NITs, I'll check and fix your other reviews soon

Great, please @ me when you're ready for a re-review.

This PR is ready for a new round of reviews! 🙂
All FIXMEs and changes requests have now been addressed, and the latest commits contain fixes for all unhandled errors and synchronisation issues I knew of.
If you see no additional issues, I think the PR is ready to be merged. I'll squash the commits and make a correct commit message when the PR is approved.
Thanks

YAY!

Thank you all for your reviews and advices, I learned a lot doing this! 🙂
Special thanks to you too @kylebrowning for bringing this back up a few months ago!


Issue #40: Add aperture/bg range functionality

Pull request tackling issue #40
Am unsure if the file naming scheme is what we want. I kept the date up to the minutes (took out the seconds). Here's some ideas I had because the simplicity of the current one worries me (having to escape certain characters) :
Current One
S3_yyyy-mm-dd_hh-mm_ap=value_bg=value_eventname
Some Other Ideas
S3_yyyy-mm-dd_hh-mm_ap-10_bg-15_eventname
S3_yyyy-mm-dd_hh-mm_ap10_bg15_eventname
S3_yyyy-mm-dd_hh:mm_ap=10_bg=15_eventname
S3_yyyy-mm-dd_ap=10_bg=15_eventname_run1    (2, 3, etc...) incrementally
or any permutation of these single changes above at once

Regarding the Travis failure to run, it seems the error is being caused by a bug in newer versions of numpy:
In a forum I found:
"The most recent version of NumPy has a bug that means it generates that error if it's imported twice in the same process."
as well as:
RuntimeError: implement_array_function method already has a docstring
Numpy github issue
I played around for a bit by limiting the numpy version even down to 1.14.0 but nothing really worked. Then I took a closer look at exactly where the error is raised and the setup file that is failing seems to be from a dependency package, as it occurs in the following log message:
Running batman-package-2.4.8/setup.py

These are some good folder naming suggestions.  Personally, I like: S3_yyyy-mm-dd_eventname_ap10_bg15_runX, where "runX" only gets used if the folder already exists.  I guess you would need a recursive check for run1, run2, etc.

These are some good folder naming suggestions. Personally, I like: S3_yyyy-mm-dd_eventname_ap10_bg15_runX, where "runX" only gets used if the folder already exists. I guess you would need a recursive check for run1, run2, etc.

Yes this is also my personal favorite, I'll make it work!

I worked on the generalized routine for the plotting we talked about. The only worry I have with it is that if the user quits a run halfway and doesn't clear their directories they'll have some folders of run 1 (let's say 4 out of the total 10) and then if they run the code again, 6 out of the 10 directories will say run 1 while 4 of them will say the next run (run 2).

Good point on your concern about having inconsistent run numbers if someone quits midway.  A simple workaround would be the create all of the folders nearly simultaneously (using a separate double for loop near the beginning) before moving to the current double for loop that does all the work.

Got it done! Let me know what you think


for-of loop does not loop (declarations clash in generated js)

TypeScript Version:  2.1.4
Compiled with --target ES5.
Code
// A *self-contained* demonstration of the problem follows...
class Main {

	constructor() {
		this.register("a", "b", "c");
	}

	private register(...names: string[]): void {
		for(let name of names) {
			console.log(name);

			this.bar({
				[name + ".a"]: () => { this.foo(name); },
			});
		}
	}

	private bar(a: any): void {}

	private foo(name: string): void {}

}

new Main();

Expected behavior:
Code prints "a", "b", and "c" on three lines
Actual behavior:
Code prints only a single line containing "a".
The bug only occurs when using both a Rest parameter ("...names") and the computed property name ([name + ".a"]) and a for-of loop and the arrow function in the loop body.
When I look at the generated JS I can see that there is a clash in variable declaration in the for-loop and the loop-body that has been extracted into a function.
TypeScript 2.1.5 gives correct output, but I have not found any explicit bug that is supposed to fix this behavior. So I am afraid that it was fixed only as a side effect of some other change and might break again in the future.
Could you include a test case like this?

This is fixed by https://github.com/Microsoft/TypeScript/pull/12831. Test added in https://github.com/Microsoft/TypeScript/pull/13946


SendTransaction check for insufficient funds + translate errors

Purpose
Ensure that SendTransaction does not allow transaction to be sent, when they would fail due to insufficient funds.
Changes

Check available funds before sending transactions in sendTransaction.
Translate errors in SendTransaction.

Checklist

[x] My code follows the style of this project.
[x] The code compiles without warnings.
[x] I have performed a self-review of the changes.
[x] I have documented my code, in particular the intent of the
hard-to-understand areas.
[x] (If necessary) I have updated the CHANGELOG.

This was caused by an off-by-one error in the cost calculation.


The article about the color datatype results in a 404

This article about the color datatype (referenced from here) results in a 404 page not found.

Thanks for flagging that, @Luctins. I'll look into what's going wrong here. For now, you can refer to the Java Processing reference entry: https://processing.org/reference/color_datatype.html


NBA 2020 Rosters breaking on player nationality

Describe the bug
Unable to pull in roster data for the upcoming 2020 NBA season. Roster function returns an error ('ao') when parsing player nationality.
To Reproduce
atl = Roster('ATL',year=2020)
Expected behavior
Expected this function to return a number of data points for players on the current roster.
Screenshots
https://imgur.com/q5m5IZE
Desktop (please complete the following information):

OS: Windows 10
Sportsreference Version 0.4.2

Hey @jpclarkin! Thanks for the issue! Looks like former Terrapin Bruno Fernando is the first ever NBA player from Angola (at least according to sports-reference.com) which is very exciting! As a side effect, sportsreference doesn't include support for Angola yet, hence the error for the unknown nation's code. Luckily, this is an easy enough fix and I will include it shortly to resolve this issue.
Thanks again!

Hey @roclark ! I think I found another one -- the 76ers roster breaks in a similar fashion. The error code this time is 'td', which is the country code for Chad and the homeland for Sixers rookie Christ Koumadje.

Hey @jpclarkin, apologies for the delay! You are absolutely right - I am testing an update now but also identified another issue with the Roster module which I need to resolve. I will get this taken care of though!


Event Log targets fails to load when program is published using Publish Single File

Hi! Thanks for reporting this bug!
Please keep / fill in the relevant info from this template so that we can help you as best as possible.
NLog version:  4.7.12
NLog.Extensions.Logging version: 1.7.4
NLog.WindowsEventLog version: 4.7.12
Platform: .NET 6.0
Current NLog config (xml or C#, if relevant)
{
  "NLog": {
    "autoReload": true,
    "internalLogFile": "c:\\temp\\nlog-internal.log",
    "throwExceptions": false,
    "internalLogLevel": "Debug",
    "targets": {
      "EventLog": {
        "type": "EventLog",
        "log": "My Product",
        "source": "My Product",
        "layout": "${level} | ${logger} | ${message}${onexception: | ${exception:format=tostring}}",
        "EntryType": {
          "type": "CompoundLayout",
          "layouts": [
            {
              "type": "SimpleLayout",
              "text": "${when:when=contains('${message}','SuccessAudit'):inner=SuccessAudit}"
            },
            {
              "type": "SimpleLayout",
              "text": "${when:when=contains('${message}','FailureAudit'):inner=FailureAudit}"
            },
            {
              "type": "SimpleLayout",
              "text": "${when:when=length('${event-properties:item=EntryType}')>0:inner=${event-properties:item=EntryType}",
              "IsFixedText": false
            }
          ]
        }
      },
      "ColoredConsole": {
        "type": "ColoredConsole"
      },
      "DebugConsole": {
        "type": "Debugger"
      }
    },
    "rules": [
      {
        "logger": "*",
        "minLevel": "Info",
        "writeTo": "EventLog"
      },
      {
        "logger": "*",
        "minLevel": "Info",
        "maxLevel": "Fatal",
        "writeTo": "ColoredConsole"
      },
      {
        "logger": "*",
        "minLevel": "Trace",
        "maxLevel": "Debug",
        "writeTo": "DebugConsole"
      }
    ]
  }
}

var config = new ConfigurationBuilder()
   .SetBasePath(Path.GetDirectoryName(Process.GetCurrentProcess().MainModule.FileName))
   .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
   .Build();

LogManager.Configuration = new NLogLoggingConfiguration(config.GetSection("NLog"));

What is the current result?
Event Log target throws an error "Failed to create Target Type"
What is the expected result?
Event log target to successfully load
Did you checked the Internal log?
Yes
Please post full exception details (message, stacktrace, inner exceptions)

2022-02-17 11:39:37.3468 Debug Skipping unknown 'NLog' property name=NLog
2022-02-17 11:39:37.3468 Debug ScanAssembly('NLog, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=5120e14c03d0593c')
2022-02-17 11:39:37.3747 Warn Ignoring assembly location because location is empty: NLog, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=5120e14c03d0593c
2022-02-17 11:39:37.3747 Debug Auto loading based on NLog-Assembly found location:
2022-02-17 11:39:37.3747 Warn Ignoring assembly location because location is empty: MyProduct, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null
2022-02-17 11:39:37.3761 Debug Auto loading based on GetEntryAssembly-Assembly found location:
2022-02-17 11:39:37.3761 Debug Auto loading based on AppDomain-BaseDirectory found location: {PublishedLocation}
2022-02-17 11:39:37.3761 Debug Search for auto loading files in location: {PublishedLocation}
2022-02-17 11:39:37.3761 Debug Start auto loading, location: {PublishedLocation}
2022-02-17 11:39:37.3761 Debug ScanAssembly('NLog.Extensions.Logging, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=5120e14c03d0593c')
2022-02-17 11:39:37.3761 Debug Auto loading done
2022-02-17 11:39:37.3761 Info Message Template Auto Format enabled
2022-02-17 11:39:37.3977 Debug Setting 'NLog.LayoutRenderers.Wrappers.UppercaseLayoutRendererWrapper.uppercase' to 'true'
2022-02-17 11:39:37.3977 Debug Setting 'NLog.Targets.ColoredConsoleTarget.name' to 'ColoredConsole'
2022-02-17 11:39:37.3977 Info Adding target ColoredConsoleTarget(Name=ColoredConsole)
2022-02-17 11:39:37.3977 Debug Registered target ColoredConsole: NLog.Targets.ColoredConsoleTarget
2022-02-17 11:39:37.3977 Debug Setting 'NLog.LayoutRenderers.Wrappers.UppercaseLayoutRendererWrapper.uppercase' to 'true'
2022-02-17 11:39:37.3977 Debug Setting 'NLog.Targets.DebuggerTarget.name' to 'DebugConsole'
2022-02-17 11:39:37.3977 Info Adding target DebuggerTarget(Name=DebugConsole)
2022-02-17 11:39:37.3977 Debug Registered target DebugConsole: NLog.Targets.DebuggerTarget
2022-02-17 11:39:37.4073 Warn Error has been raised. Exception: NLog.NLogConfigurationException: Failed to create target type: EventLog
---> System.ArgumentException: Target cannot be found: 'EventLog'
at NLog.Config.Factory`2.CreateInstance(String itemName)
at NLog.Config.LoggingConfigurationParser.CreateTargetType(String targetTypeName)
--- End of inner exception stack trace ---
2022-02-17 11:39:37.4073 Warn Error has been raised. Exception: NLog.NLogConfigurationException: Target 'EventLog' not found for logging rule: *.
2022-02-17 11:39:37.4073 Debug --- NLog configuration dump ---
2022-02-17 11:39:37.4073 Debug Targets:
2022-02-17 11:39:37.4073 Debug ColoredConsole Target[ColoredConsole]
2022-02-17 11:39:37.4231 Debug Debugger Target[DebugConsole]
2022-02-17 11:39:37.4231 Debug Rules:
2022-02-17 11:39:37.4231 Debug logNamePattern: (:All) levels: [ Info Warn Error Fatal ] appendTo: [ ]
2022-02-17 11:39:37.4231 Debug logNamePattern: (:All) levels: [ Info Warn Error Fatal ] appendTo: [ ColoredConsole ]
2022-02-17 11:39:37.4231 Debug logNamePattern: (:All) levels: [ Trace Debug ] appendTo: [ DebugConsole ]
2022-02-17 11:39:37.4231 Debug --- End of NLog configuration dump ---
2022-02-17 11:39:37.4231 Info Validating config: TargetNames=ColoredConsole, DebugConsole, ConfigItems=25
2022-02-17 11:39:37.4231 Debug Unused target checking is started... Rule Count: 3, Target Count: 2
2022-02-17 11:39:37.4231 Debug Unused target checking is completed. Total Rule Count: 3, Total Target Count: 2, Unused Target Count: 0
2022-02-17 11:39:37.4387 Debug Debugger(Name=DebugConsole): System.Diagnostics.Debugger.IsLogging()==false. Output has been disabled.
2022-02-17 11:39:37.9131 Debug Targets for MyNamespace by level:
2022-02-17 11:39:37.9131 Debug Trace => DebugConsole
2022-02-17 11:39:37.9131 Debug Debug => DebugConsole
2022-02-17 11:39:37.9131 Debug Info => ColoredConsole
2022-02-17 11:39:37.9131 Debug Warn => ColoredConsole
2022-02-17 11:39:37.9131 Debug Error => ColoredConsole
2022-02-17 11:39:37.9131 Debug Fatal => ColoredConsole
2022-02-17 11:39:37.9316 Info AppDomain Shutting down. Logger closing...
2022-02-17 11:39:37.9316 Debug Flush completed
2022-02-17 11:39:37.9316 Debug Targets not configured for logger: MyNamespace
2022-02-17 11:39:37.9386 Debug Closing logging configuration...
2022-02-17 11:39:37.9386 Debug Closing target 'ColoredConsole Target[ColoredConsole]'.
2022-02-17 11:39:37.9386 Debug Closed target 'ColoredConsole Target[ColoredConsole]'.
2022-02-17 11:39:37.9386 Debug Closing target 'Debugger Target[DebugConsole]'.
2022-02-17 11:39:37.9386 Debug Closed target 'Debugger Target[DebugConsole]'.
2022-02-17 11:39:37.9386 Debug Finished closing logging configuration.
2022-02-17 11:39:37.9386 Info Logger has been shut down.

Are there any workarounds? yes/no
Yes, disable single file publishing
Is there a version in which it did work?
Not sure
Can you help us by writing an unit test?
Maybe

The problem is this:
2022-02-17 11:39:37.4073 Warn Error has been raised. Exception: NLog.NLogConfigurationException: Failed to create target type: EventLog
---> System.ArgumentException: Target cannot be found: 'EventLog'
at NLog.Config.Factory`2.CreateInstance(String itemName)
at NLog.Config.LoggingConfigurationParser.CreateTargetType(String targetTypeName)
--- End of inner exception stack trace ---
2022-02-17 11:39:37.4073 Warn Error has been raised. Exception: NLog.NLogConfigurationException: Target 'EventLog' not found for logging rule: *.

The EventLog-target on NetCore-platform requires a nuget-package: https://www.nuget.org/packages/NLog.WindowsEventLog
And it has to be included in the NLog.config:
{
  "NLog": {
    "extensions": [
      { "assembly": "NLog.WindowsEventLog" }
    ]
  }
}

See also: https://github.com/NLog/NLog.Extensions.Logging/wiki/NLog-configuration-with-appsettings.json

NLog 5.0 includes #4602 that helps people that forgets to add extensions.
NLog 5.0 also allows you to declare extension-targets like this:
      "EventLog": {
        "type": "EventLog, NLog.WindowsEventLog"
      }

Thanks, it works.

Closing issue as resolved.


math.MaxInt64 overflows int on ARM

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
go version go1.9.2 linux/arm
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
GOARCH="arm"
GOBIN=""
GOEXE=""
GOHOSTARCH="arm"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/alarm/go"
GORACE=""
GOROOT="/usr/local/go1.9.2.linux-armv6l"
GOTOOLDIR="/usr/local/go1.9.2.linux-armv6l/pkg/tool/linux_arm"
GCCGO="gccgo"
GOARM="6"
CC="gcc"
GOGCCFLAGS="-fPIC -marm -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build788013818=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
What did you do?
Here is a minimum working example of the error.
package main

import (
	"math"
)

func main() {
	min := math.MaxInt64
}

What did you expect to see?
This program should generate no errors.  (actually it complains that min is declared and not used, but that's not important right now)
What did you see instead?
# command-line-arguments
./mwe.go:8:6: constant<PHONE_NUMBER>854775807 overflows int

I have just learned that armv7l is 32-bit so of course int would be 32 bits wide and overflowing is understandable.

After a discussion on the Slack community, I have learned that the current behavior is part of the specification.  Kale Blankenship helpfully cited the specific section:  https://golang.org/ref/spec#Constants.  This section says the default type for this constant would be int which is what's happening.  I'm going to close this issue as works-as-expected.

I was just typing up the same response and here is a duplicate issue that was closed https://github.com/golang/go/issues/19621. Thank you for the find and for closing the issue.

@ianlancetaylor @robpike do you think this qualifies as a candidate for a doc/faq entry? I ask because I've encountered this before and the internet helped me with the fix, someone else in #19621 did and so has @mathuin right now, who knows who else.

I'm not sure it quite qualifies as a FAQ.  It's more like a bug that we can't fix because of the Go 1 compatibility guarantee.  I think the best resolution would be to use explicit types in Go 2, though I don't see a proposal open for that.

Gotcha @ianlancetaylor, thank you for the advice. I'd open the bug/proposal but I wouldn't have as much substance or context as you would have, so feel free to go ahead and open one, if you please.  However, I'd be delighted to contribute to it i.e. working on a fix or helping out,

Filed #23087.

Awesome, thank you @ianlancetaylor!

I do understand that it is by the spec, but can you please provide an workaround snippet on how to overcome this?

min := int64(math.MaxInt64)

Thanks. Helped a lot.


Deactivation of urllib3 warnings does not work for requests 2.2.1 but it does for requests 2.9.1

How should we handle this versioning issue?   As a workaround in my local dev environment I set:
requests >= 2.9.1 in requirements.txt.

We've set the requirements to enforce a sufficiently recent requests.


Truthy/Falsey values don't do what you want them to

Carried over from: https://github.com/reactiveops/autohelm/issues/7

I have a suggested method of proceeding forward with a change. Perhaps instead of converting all the values in the course.yml into --set xyz params, we could run them as files (native yaml) to the helm command. This would avoid the fidelity loss of type when converting from yaml to --set

I think that may be a good way to go and would likely not require any change in the course.yml

@endzyme IMO,
Using values-file: for the new behavior is the simplest solution since it's a new key.
Then you can just move values: -> set: and alias values: with a deprecation notice.
A future version could move the alias target from set: to values-file:.
Detecting keys that would only work with --set would be useful.
Basically anything with an unescaped period will change in behavior.
There's probably a few ways you could create a smooth migration path.

@stealthybox - I've decided to bite the bullet and attempt to see what it might look like to warn people about the values: {} behavior change and support sets: {} functionality.
The current PR implementation merges the values: {} and sets: {} settings for a chart, and sets: {} take precedence (just like the CLI). If someone is using values: {} a deprecation notice is displayed as a warning. I also added some debug messages which note that any values will be actually used in --set for helm. I am still not sure on the timeframe to fully deprecate the behavior, but at least the fix for today is rename your values: {} key to sets: {} and you should chug happily along.

This may be a bit pedantic or nit-picky, but I think sets is a very confusing way to name that.  Maybe something like set-values that is slightly more descriptive to the user. I believe there are other concerns as well, but I will wait until I test out this PR.

Thanks @sudermanjr! I don't think that's a nit-pick at all. Valuable feedback and a worthwhile change.

I suggested set: (singular) simply after the flag name.
set-values:, set-flags:, set-kv:, set-map: all make sense to me.

Additional information here: All environment variable values do their interpolation by iterating a list of arguments on the command line and interpolating the values in each "arg" string. This will not work and will need a refactor when we support native support values.

I'm having a hard time parsing the meaning here:

All environment variable values do their interpolation by iterating a list of arguments on the command line and interpolating the values in each "arg" string. This will not work and will need a refactor when we support native support values.

I'm trying to call out that the way we do environment variable interpolation will be affected by the work that needs to happen with this issue. It is extra scope.

ah, I didn't know reckoner supported env var interpolation -- that's super useful.
We might just consider supporting that for the entire course.yaml.

#131 resolved this


Add Creative Commons (CC) licence for content

We need to add a CC license for our content—if anyone knows how to about doing this (some more information here), your help would be appreciated! ©

I think this issue is already solved, isn't it? @lasse-offerzen issue should be closed

Thanks @cmarchena !


Directly hitting Route leads to 404

If I hit Directly http://localhost:3000/features it leads to Not Found . Without this feature essentially it can't be used in consumer app.

That's a problem in your server setup most likely. The server needs to redirect every connection to /.

Nopes . There's no such route match in your express . So this should fail for everybody

app.get('*', /* … */);

https://github.com/mxstbr/react-boilerplate/blob/master/server/middlewares/frontendMiddleware.js#L25-L33
That matches every connection (no matter what URL) to the express server, and returns the correct file.
So, the issue must be that the connection never reaches the express server!

For what it's worth, I'm having the same issue.
I've pulled a fresh copy of this, and visiting the /features page through the button on the page works fine, but hitting the URL or refreshing once I'm on that page, it returns a 404.

Ahhh, I know why, that's my bad! Fixing asap.

Sorry, I didn't realize a recent change broke this, that was my bad! Thanks for notifying me, this is now fixed! :tada:


66 remove changelog workflow

Fix #67. Fix #68.
Removes the changelog workflow entirely.
Updates documentation for make_example_data.

@indialindsay Can you add a brief description and citation to the docstring for the rainfall data in L129 of make_example_data.py? Should be okay to add a similar citation to refs.bib as minku2010.


Sync subdirectories recursively excluding some

Hi Everyone... just tried RClone for the first time. Read for about 20 mins but could not get it straight.
I have a deep directory structure with 1000s of images inside 100s of subdirectories to be sync'd recursively to my Google Drive, but leaving out ./cache and ./thumbnails directory.
Issue #1 rClone does not copy subdirectories
rclone sync /synctest/images GDrive:/images
this ... only sync files in the dir specified
Why is it not sync'ng and creating the directory structure ? What arguments need to be passed to sync all subdir recursive?
issue #2 How to exclude certain directories
is this right ?
rclone sync /synctest/images GDrive:/images --exclude "/thumbnails/**"
I read you can make and load and exclusion file... but if I only need to exclude one or two directories, what will be the right parameter to pass ?
Thanks

Issue 1 rClone does not copy subdirectories

rclone does copy subdirectoreis by default.  However it doesn't copy empty directories - is that your problem?

issue 2 How to exclude certain directories

--exclude "/thumbnails/**" this will exclude thumbnails and everything in it.  If you have thumbnail directories not at the root, then you want --exclude "thumbnails/**".  However I suspect you are missing the leading ., so what you want is --exclude "/.thumbnails/**"


Does preprocess.sh have an effect if using datasets other than phoneix?

Does preprocess.sh have an effect if using datasets other than phoneix? I have successfully generated the *-goundtruth-[dev,test,train].stm file. Is that alone enough to train with another dataset?
I look forward to the response.
Thank You

Dear @yulrio,
As indicated in Line 14, the simplifications pertain specifically to the phoenix14/14T datasets. Please adjust the preprocessing step according to the evaluation requirements of any other dataset.
I hope this resolves your question.


FLAS-60: Integrate SimpleMDE Editor for Blog Post Creation and Update

Description of the Change
This pull request integrates the SimpleMDE editor into the blog post creation and update pages. The change enhances the user experience by providing a WYSIWYG markdown editor, allowing users to format their blog posts more easily.
How It Addresses the Issue
The issue titled "WYSIWYG" requires all work to be forked off and merged into the sept-16th-demo branch. This PR addresses the issue by incorporating the SimpleMDE editor, which provides a WYSIWYG interface for editing blog posts, aligning with the requirement for a more user-friendly editing experience.
Relevant Changes

Added SimpleMDE CSS and JS links to base.html to ensure the editor is available across the application.
Updated create.html and update.html to initialize SimpleMDE on the blog post body textarea, enhancing the text editing capabilities.

Checklist

[ ] Add tests to demonstrate the correct behavior of the change.
[ ] Update relevant documentation in the docs folder and in code.
[ ] Add an entry in CHANGES.rst summarizing the change and linking to the issue.
[ ] Add .. versionchanged:: entries in any relevant code docs.

Linked Issues
fixes #FLAS-60
This PR does not fully resolve the issue as it focuses on integrating the editor rather than addressing the complete scope of the issue description.

When the post is rendered, the WYSIWYG isn't rendering as html, it's rendered as plain text.
For example, the following in the WYSIWYG editor:
Hello!
* DkaJk
* * jklfsj

Shows up as preceded by asterisks, not as bullet points.


[Bug] _dispatchMessage has a multiple queue issue

Here is my case:
I have a Webview with text and some imgs,i used WebViewJavascriptBridge to get the img src(generally more than one pic) and downloaded in local use for-loop with SDWebImage succesfully.Like it:
for (NSUInteger i = 0; i < imageUrls.count; i++) {
      NSString *_url = imageUrls[i];

      [manager downloadImageWithURL:[NSURL URLWithString:_url] options:SDWebImageLowPriority progress:nil completed:^(UIImage *image, NSError *error, SDImageCacheType cacheType, BOOL finished, NSURL *imageURL) {
          if (image) {

              [_allImagesOfThisArticle replaceObjectAtIndex:i withObject:image];

              NSString *key = [manager cacheKeyForURL:imageURL];
              NSString *imageDiskPath = [manager.imageCache defaultCachePathForKey:key];
              NSLog(@"图片在硬盘中的地址:%@",imageDiskPath);

              [_bridge callHandler:@"imagesDownloadComplete" data:@[key,imageDiskPath]];

          }

      }];

  }

When I call   [_bridge callHandler:@"imagesDownloadComplete" data:@[key,imageDiskPath]]; , it invoked the function in JS named imagesDownloadComplete.Here is the JS code:
function imagesDownloadComplete(pOldUrl, pNewUrl) {
    var allImage = document.querySelectorAll("img");
    allImage = Array.prototype.slice.call(allImage, 0);
    allImage.forEach(function(image) {
        if (image.getAttribute("src") == pOldUrl || image.getAttribute("src") == decodeURIComponent(pOldUrl)) {
            image.src = pNewUrl;
        }
    });
}

But the fact is,the webpage cannot load all images successfully,there always has some pics didn't load.Like this:

pls.

Hi,
Without a repro I won't be able to help you :(
The good news is that we have automated tests now! :)
If you create a PR with a failing test then I will definitely give you a hand (see https://github.com/marcuswestin/WebViewJavascriptBridge/blob/master/Tests/WebViewJavascriptBridgeTests/BridgeTests.m and https://github.com/marcuswestin/WebViewJavascriptBridge/blob/master/Tests/WebViewJavascriptBridgeTests/echo.html).
You could also create a PR with an example in Example Apps with the problem you're seeing in - that would definitely help me help you.
Cheers!


Jenkins smoke tests should test terraform-based clusters

PXE boot additional clusters in parallelized Jenkins smoke tests to cover terraform-based example clusters.

simple-install (perhaps)
etcd3-install
bootkube-install

Note, this requires terraform be installed on the Fedora bare-metal executor nodes.

https://github.com/coreos/matchbox/pull/520
https://github.com/coreos/matchbox/pull/535

bootkube-install tests (#535) were temporarily disabled (#542). Re-opening to track adding this coverage back.


include devcontainer-collection.json

related to https://github.com/github/codespaces/issues/6384 and https://github.com/microsoft/dev-container-spec/issues/7

Keeping publish-definitions disabled in the action for now


*: add spec BackupServiceStatus

We should have separate backupapi/ and spec/ ’s BackupServiceStatus.

this should live inside backup pkg. the spec bit might need to be duplicated in spec though.

fixed

LGTM. Thanks!


When parsing DBC, use long names in SignalGroup.signal_names to match Signal.name

Currently, when loading from a DBC with a SystemSignalLongSymbol attribute defined for a given signal, the signal name used in the Signal object will be full "long name" of the signal. But when loading signal groups, the "short names" are used in the SignalGroup object. It is difficult for the calling code to reconcile this discrepancy because the "short name" is not maintained anywhere in the Signal object.
This PR makes the SignalGroup objects use the "long names" of signals when loading from DBC, copying the same logic used when loading Signal objects.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>9
Details

14 of 14 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.01%) to 93.584%

Totals

Change from base Build<PHONE_NUMBER>3:
0.01%

Covered Lines:
7278

Relevant Lines:
7777

💛  - Coveralls

Would it be possible to make a release to get this in the pypi package?

done


Stdlib compat

This is missing tests and documentation but I'd like to get some comments on it.

I’m closing this b/c it looks abandoned and the open PR with a green checkmarks annoys me. :)  Feel free to open an updated PR anytime.


how to install

i have no experience on rofi so how to install pls

What Linux distribution are you using?

arch linux hyprland WM

Excellent! I think you should read this article, which is very detailed about how to install and use rofi.

Ok, no need to really install anything: just clone the repo:
gh repo clone pdonadeo/rofi-web-search

and copy the script rofi-web-search.py in a directory in your $PATH.
Then configure Hyprland to launch the script. I'm not an Hyprland user so I cannot help in this.

thanks man for help


Wrong match when using s as a trigger key

Checklist

[X] Have you tried updating the plugin to the latest version?
[X] Have you checked the Breaking Changes issue?
[X] Have you read through :h nvim-surround to see if there might be any relevant information there?

Neovim Version
v0.8.3
Plugin Version
Tagged (Stable)
Minimal Configuration
vim.cmd([[let $PLUGIN_DIRECTORY = '~/.local/share/nvim/plugtest']])
vim.cmd([[packadd nvim-surround]])

require("nvim-surround").setup({
  surrounds = {
    -- strong emphasis for markdown
    ["s"] = {
      add = { "**", "**" },
      find = "%*%*.-%*%*",
      delete = "^(%*%*?)().-(%*%*?)()$",
      change = {
        target = "^(%*%*?)().-(%*%*?)()$",
      },
    },
  },
})

Sample Buffer
This is a test sentence with **emphasis**.

Another one (for good measure).

Keystroke Sequence
dss
Expected behavior
When the cursor is on **emphasis**, pressing the keystroke above deletes the surrounding asterisks, leaving emphasis. as a result.
Actual behavior
The parentheses in the second paragraph are deleted, leaving Another one for good measure. as a result.
Additional context
I adapted the example in this comment, where I changed the trigger key from * to s.
If I change it back to * or replace it with another letter such as e, it works without any issues.

Have you tried setting the alias s to false in the setup function? By default, s is an alias that matches "any built-in surround".

I've only tried that just now. It solves the problem. Thank you!


Main

This is a first example. The initial code is just kept as it was, but this will be modified later (!). Now, we just add some 20 lines to model this as a dictionary, with entries and senses.
The next step will be to

make our extraction of concepts ideally in a way to be applicable for other datasets, maybe using some small library we can later write
making a manual refinement of the wordlist

Note that we can add the phonetic transcription to the entries.csv later.

Great, the conversion script is working, so I've pushed the new version containing entrires.csv, and senses.csv to the repository and will now try to implement the extraction of concepts, I have some ideas how to do this


[Announce] Nacos Spring 1.0.0 released

Hello Community,
I am glad to announce that Nacos Spring 1.0.0 is released.
You can check detailed release notes here: https://github.com/nacos-group/nacos-spring-project/releases/tag/1.0.0
If you have any questions using this version, please report the issue on Github.
Best regards,
Alibaba Open Source Team

please upgrade nacos-client dependency to com.alibaba.nacos:nacos-client:1.4.0


Internet Explorer stinks-- I mean, "isn't working"

Imagine my surprise when I noticed that "everybody's favorite browser" is having troubles again... It seems to not be loading airports somehow, which leads to it failing to start the game. :angry:
It dies on ui.js:113 when it can't find prop.airport.airports, because prop does not have a property airport. Do the same thing in Chrome or Firefox, and all the airports are there.

It would be very useful to specify what version of IE this is affecting.  (I have limited access to IE and rarely touch it)

This error can't yet be reduced to a smaller use case. Can't it? Maybe adding conditional console.log (Error.stack) or debugger; statement can help?

This bug doesn't occur in edge.

In ie 11, this line of code is causing an error:
https://github.com/zlsa/atc/blob/e985d0bc8d2ca678366805cded9038b61c691430/assets/scripts/airport.js#L735
I'm not sure if that is causing a snowball effect or not (making the value undefined), and why ie 11 didn't stop executing.

This is also the case in Safari.


comments in modules

How can we sync the comments in modules to the name manager? After each save in modules, the comments in the name manager disappear

Are you using the format:
/**
 A comment
 */
myName = "has a comment";

Thanks for the correction. I used the wrong format


oauth: Notion connector dual-flow + front gating

Description

Implenents dual support for Nango and oauth for Notion
Change validation token code to call me vs search (faster?)
Gate notion for oauth setup in front.

Risk
Low
Full sync + update tested locally
Deploy Plan

deploy connectors
deploy front

PMRR


Parameterize tenantid when checking stream version

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: Jordan466 sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

@Jordan466 Thanks for taking the time to troubleshoot this and do the PR! Do see the comment though, you have to account for whether or not multi-tenancy is present for the event store.

Hey all, just to speed this up, I'm taking it in, then making the easy fix just to get 5.1 out. Thanks!


pull request for e2e test at 02/05/2017 07:26:06

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


retain original model serializer

The purpose of this PR is to modify the field_dict field on the VersionSerializer such that it uses the original detail serializer. in this way the request returns the version of the model in the same format as its current representation and not just a JSON blob of the model-record.
e.g. in the following example we looking at the version of a parent model that has and m2m relationship to a child model
{
  "id": 1,
  "revision": {
    "date_created": "2021-12-12T10:32:12",
    "user": 12,
    "comment": "added new child"
  },
  "field_dict": {
    "id": 11,
    "children": [
      {
        "id": 1,
        "name": "Bart"
      },
      {
        "id": 2,
        "name": "Lisa"
      },
      {
        "id": 3,
        "name": "Maggie"
      }
    ]
  }
}

rather than
{
  "id": 1,
  "revision": {
    "date_created": "2021-12-12T10:32:12",
    "user": 12,
    "comment": "added new child"
  },
  "field_dict": {
    "id": 11,
    "children": [
      1,
      2,
      3
    ]
  }
}

Thanks again @gecBurton 😄


Training Slows Down -- But Speeds up If Entire Net is Reloaded

NOTE: Only file GitHub issues for bugs and feature requests.  All other topics will be closed.
What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?
There are a few issues but none that simulate the same problem.
Environment info
Operating System: Ubuntu 14.04, tf 0.12
Installed version of CUDA and cuDNN: 8.0
(please attach the output of ls -l /path/to/cuda/lib/libcud*):
If installed from binary pip package, provide: tensorflow pip gpu 0.12

A link to the pip package you installed:
The output from python -c "import tensorflow; print(tensorflow.__version__)".

Problem
I am training a network, but after about 4000 backward passes, the training of the network starts to slow down. I have tried reloading the network after saving it. And again, 4000 steps later it starts to slow down. This means that the slow down is independent of what global step the training is on. The network size never changes, which is usually the problem with these slow downs. Average Step times are like this:
2.4, 2.4, 2.4, 4.6, 3.9, 12.2, 4.2, 2.4, 6.7
I am feeding the model with placeholders, so there is no FIFO Queue problems, and feeding it with text data so its pretty light. I have tested this on three separate machines and the same behavior is replicated throughout them. It is the actual tensorflow session.run command that slows it down tremendously.
When I look at the gpu usage, both gpu's are at 0 percent the entire time (even during the session.run command). They spike up only when the actual forward and backward passes occur but remain dormant the rest of the time. I'm using allocator type 2 in the tf.gradient operation. This perhaps is the most revealing part.
My htop indicates there's no swap memory problems at all.
From what I have concluded the problem must be with tensorflow. The fact that You can reload the model and its completely fine for a little bit is really weird. I've also found that running the following command does not help:
sync; echo 3 > /proc/sys/vm/drop_caches
INFO:tensorflow:global step 2001 learning rate 0.0003882 step-time 2.94 perplexity 28.97 loss 3.3662
INFO:tensorflow:get_batch_step_time 0.016 actual_step_time 2.929
INFO:tensorflow:16:01:09 01/10/17 EST
INFO:tensorflow:global step 2251 learning rate 0.0003867 step-time 2.94 perplexity 27.17 loss 3.3022
INFO:tensorflow:get_batch_step_time 0.016 actual_step_time 2.920
INFO:tensorflow:16:13:25 01/10/17 EST
INFO:tensorflow:global step 2501 learning rate 0.0003853 step-time 2.94 perplexity 26.03 loss 3.2594
INFO:tensorflow:get_batch_step_time 0.016 actual_step_time 2.929
INFO:tensorflow:16:25:48 01/10/17 EST
INFO:tensorflow:global step 2751 learning rate 0.0003838 step-time 2.97 perplexity 24.80 loss 3.2109
INFO:tensorflow:get_batch_step_time 0.016 actual_step_time 2.956
INFO:tensorflow:16:44:37 01/10/17 EST
INFO:tensorflow:global step 3001 learning rate 0.0003824 step-time 4.52 perplexity 23.77 loss 3.1686
INFO:tensorflow:get_batch_step_time 0.016 actual_step_time 4.500
INFO:tensorflow:16:58:05 01/10/17 EST
INFO:tensorflow:global step 3251 learning rate 0.0003809 step-time 3.23 perplexity 23.10 loss 3.1396
INFO:tensorflow:get_batch_step_time 0.016 actual_step_time 3.216
INFO:tensorflow:17:18:34 01/10/17 EST
INFO:tensorflow:global step 3501 learning rate 0.0003795 step-time 4.92 perplexity 21.82 loss 3.0828
INFO:tensorflow:get_batch_step_time 0.015 actual_step_time 4.902

Any help would be greatly appreciated!

What about using tcmalloc?

One way to narrow this down is to use google performance tools to take performance profile before and after slowdown, and see if there's some obvious difference in where the time is spent

Hey @yaroslavvb thanks for your response. I will use tcmalloc and try to diagnose what exactly is going on. Do you recommend just running it like this?
LD_PRELOAD=/usr/lib/libtcmalloc.so.4 HEAPPROFILE=/tmp/profile python timestep_network.py

Something like
export LD_PRELOAD="/usr/lib/libtcmalloc_and_profiler.so.4"
export CPUPROFILE="cpu-profile"
python lm_benchmark_grpc.py --num_gpus=1
export exe=which python
google-pprof --pdf $exe $CPUPROFILE > profiling_output.pdf
On Tue, Jan 10, 2017 at 6:59 PM, Nick Shah<EMAIL_ADDRESS>wrote:

Hey @yaroslavvb https://github.com/yaroslavvb thanks for your response.
I will use tcmalloc and try to diagnose what exactly is going on. Do you
recommend just running it like this?
LD_PRELOAD=/usr/lib/libtcmalloc.so.4 HEAPPROFILE=/tmp/profile python
timestep_network.py
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/6779#issuecomment-271764841,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABaHLawIUrOEAKit_v34cn1YJGtYILlks5rREWagaJpZM4LgF36
.

Hey @yaroslavvb I just wanted to update you and let you know for right now, tcmalloc seems to have resolved the issue. I haven't fully trained all the way but so far, it does seem to be pretty stable! Thanks! I'll close for right now.


Added AES256 keyfile decryption support

Description:
Added OpenSSL compatible AES256 decryption capabilities to the pre-commit script. If a .enc file is supplied for the key parameter, the script will prompt the user for the decryption password. If the password is invalid a ValueError is raised and the user is notified the password was incorrect.
If the user supplies a .json file it will work as it was originally built and skip the decryption process.
Example Usage:
Encrypt the file:
openssl aes-256-cbc -in your_key_file.json -out your_key_file.enc

Decrypt the file:
./pre-commit -k your_key_file.enc

OR
openssl enc -d -aes-256-cbc -k "Your_Password_Here" -in "your_key_file.enc" -out "your_key_file.json"

Testing Steps:

[ ] Change directories to king-phisher-plugins and run the script with ./pre-commit -k your_key_file.enc
[ ] Verify password decryption works
[ ] Verify ValueError exception is raised with an invalid password
[ ] Verify script works with a json key as it was originally built

I reworked this quite a bit and got it added into both King Phisher (commit securestate/king-phisher@2d1a105bac13a05151c8771a54695ccf0dc92481) and the pre-commit script (commit cbe8550ccb96f10e9d15c8d0aa328035b72e686f). This PR is no longer necessary so I'm going to close it out.