[SPARK-17620][SQL] Determine Serde by hive.default.fileformat when Creating Hive Serde Tables

What changes were proposed in this pull request?
Make sure the hive.default.fileformat is used to when creating the storage format metadata.
Output
scala> spark.sql("SET hive.default.fileformat=orc")
res1: org.apache.spark.sql.DataFrame = [key: string, value: string]

scala> spark.sql("CREATE TABLE tmp_default(id INT)")
res2: org.apache.spark.sql.DataFrame = []

Before
scala> spark.sql("DESC FORMATTED tmp_default").collect.foreach(println)
..
[# Storage Information,,]
[SerDe Library:,org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,]
[InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,]
[OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,]
[Compressed:,No,]
[Storage Desc Parameters:,,]
[  serialization.format,1,]

After
scala> spark.sql("DESC FORMATTED tmp_default").collect.foreach(println)
..
[# Storage Information,,]
[SerDe Library:,org.apache.hadoop.hive.ql.io.orc.OrcSerde,]
[InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,]
[OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,]
[Compressed:,No,]
[Storage Desc Parameters:,,]
[  serialization.format,1,]

How was this patch tested?
Added new tests to HiveDDLCommandSuite

Test build #65742 has started for PR 15190 at commit 042a94e.

Test build #65744 has started for PR 15190 at commit f60e760.

Test build #65742 has finished for PR 15190 at commit 042a94e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65742/
Test PASSed.

Test build #65752 has started for PR 15190 at commit f2b93de.

Please update the PR description. This is not for orc only

Test build #65744 has finished for PR 15190 at commit f60e760.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65744/
Test PASSed.

Test build #65752 has finished for PR 15190 at commit f2b93de.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65752/
Test PASSed.

PR title should be
Determine Serde by hive.default.fileformat when Creating Hive Serde Tables

As we set default serde/inputFormat/outFormat, I think we don't need to set default values for them in CreateHiveTableAsSelectCommand again. What do you think?

@viirya I think we can come here from multiple code paths like visitCreateTableUsing. I think we can come to DataSinks's CreateTable case without serde being set.

oh. right. Should we set default serde in visitCreateTableUsing?

Currently the default serde setting path looks scattered. I think it is better to avoid that.

@viirya In my understanding, thats the datasource table code path. I am not sure if we should look at hive.default.fileformat property to set the default storage for data source tables ? In my opinion, its probably better to leave it the way it is to make sure we always have a default setting in the common path so we don't miss it.

hmmm, as we prohibit CREATE TABLE USING" with hive serde table, can we reach DataSinks's CreateTable from the path if visitCreateTableUsing?

@dilipbiswal yeah. I see. I still think to set default serde at different places looks not good to me, especially in other places it is hard-coded.

@dilipbiswal nvm. It is not related to this change.

Can you try CREATE TABLE tmp_default(id INT) as select .... and see if the table will be converted to parquet format?

@yhuai Hi Yin,
create table ... as select ... would respect the setting of hive.default.fileformat.
scala> spark.sql("SET hive.default.fileformat=parquet")
res14: org.apache.spark.sql.DataFrame = [key: string, value: string]

scala> spark.sql("CREATE TABLE tmp_default4 select * from tmp_default")
res11: org.apache.spark.sql.DataFrame = []

scala> spark.sql("DESC FORMATTED tmp_default4").collect.foreach(println)
...
[# Storage Information,,]
[SerDe Library:,org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,]
[InputFormat:,org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat,]
[OutputFormat:,org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat,]
...

scala> spark.sql("SET hive.default.fileformat=orc")
res14: org.apache.spark.sql.DataFrame = [key: string, value: string]

scala> spark.sql("CREATE TABLE tmp_default5 select * from tmp_default")
res15: org.apache.spark.sql.DataFrame = []

scala> spark.sql("DESC FORMATTED tmp_default5").collect.foreach(println)
...
[# Storage Information,,]
[SerDe Library:,org.apache.hadoop.hive.ql.io.orc.OrcSerde,]
[InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,]
[OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,]
...

@dilipbiswal Are they data source tables?

@gatorsmile Thanks.. didn't realize we wanted to find out the CTAS behaviour. Here is the result..
When covertCTAS is set to true, we create a data source table with parquet format.
scala> spark.sql("SET spark.sql.hive.convertCTAS=true")
res23: org.apache.spark.sql.DataFrame = [key: string, value: string]

scala> spark.sql("CREATE TABLE tmp_default6 as select * from tmp_default")
res24: org.apache.spark.sql.DataFrame = []

scala> spark.sql("DESC extended tmp_default6").collect.foreach(println)
[# Detailed Table Information,CatalogTable(
    Provider: parquet
    Storage(Location: file:/home/mygit/apache/spark/bin/spark-warehouse/tmp_default6, InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat, OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat, Serde: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe, Properties: [path=file:/user/hive/warehouse/tmp_default6, serialization.format=1])),]

@dilipbiswal Based on your tests and the source codes, I think your fix does not break anything.

Looks good.

Thanks @dilipbiswal! This change works for my case (I filed the JIRA).

@dilipbiswal Could you please check whether your manual test case is included in the master branch? If not, can you add it? Thanks!

@gatorsmile We have test in SQLQuerySuite "CTAS without serde without location" where we check for default data source. I had actually added the following test before i saw the test in SQLQuerySuite. So i didn't push the added test.
val defaultDataSource = TestHive.sessionState.conf.defaultDataSourceName
    // Make sure only default datasource is used when converting CTAS to use data source.
    withSQLConf("hive.default.fileformat" -> "orc") {
      withSQLConf(SQLConf.CONVERT_CTAS.key -> "true") {
        val (desc, exists) =
          extractTableDesc("CREATE TABLE IF NOT EXISTS test select 1 from foo")
        assert(exists)
        assert(desc.provider == Some(defaultDataSource))
      }
    }

@dilipbiswal Great! Thank you!
LGTM

cc @yhuai @cloud-fan

retest this please

retest this please

Test build #66685 has started for PR 15190 at commit f2b93de.

Test build #66685 has finished for PR 15190 at commit f2b93de.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66685/
Test PASSed.

Merged build finished. Test PASSed.

cc @yhuai @cloud-fan Based on the above PR discussion, it sounds like this PR is ok to merge. What do you think? Thank you!

If we have spark.sql.hive.convertCTAS=true and hive.default.fileformat=orc, what format will we use when we create a table through a CTAS statement?

@yhuai We will use Parquet format in your example. We look at SQL spark.sql.sources.default  configuration to decide on the format to use ?
Here is the output for your perusal.
spark-sql> set spark.sql.hive.convertCTAS=true;
spark.sql.hive.convertCTAS	true
Time taken: 3.309 seconds, Fetched 1 row(s)
spark-sql> set hive.default.fileformat=orc;
hive.default.fileformat	orc
Time taken: 0.053 seconds, Fetched 1 row(s)
spark-sql> CREATE TABLE IF NOT EXISTS test select 1 from foo;
spark-sql> describe formatted test;
...
# Storage Information
SerDe Library:	org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
InputFormat:	org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
OutputFormat:	org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat

Now change spark.sql.sources.default=orc
spark-sql> set spark.sql.sources.default=orc;
spark.sql.sources.default	orc
spark-sql> CREATE TABLE IF NOT EXISTS test2 select 1 from foo;
Time taken: 0.451 seconds
spark-sql> describe formatted test2;
...
# Storage Information
SerDe Library:	org.apache.hadoop.hive.ql.io.orc.OrcSerde
InputFormat:	org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
OutputFormat:	org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat

Please let me know if you have any further questions.

Added them to your test cases. : )

Test build #66919 has started for PR 15190 at commit f32fe21.

Test build #66919 has finished for PR 15190 at commit f32fe21.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66919/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #66944 has started for PR 15190 at commit f32fe21.

Test build #66944 has finished for PR 15190 at commit f32fe21.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66944/
Test FAILed.

retest this please

Test build #66949 has started for PR 15190 at commit f32fe21.

Test build #66949 has finished for PR 15190 at commit f32fe21.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66949/
Test PASSed.

Merged build finished. Test PASSed.

@gatorsmile @yhuai I have added a new test.  Can we please take a look at this again ?

@dilipbiswal That makes sense. Thank you for testing that. I do not have any other question.

LGTM

Merging to master! Thanks!

Thank you @yhuai @gatorsmile @cloud-fan @viirya @dafrista

Seems this breaks the build. https://amplab.cs.berkeley.edu/jenkins/view/Spark QA Compile/job/spark-master-compile-sbt-scala-2.10/2843/console

I am reverting this patch. Sorry.

Reverted

@yhuai very sorry Yin.. Let me look what happened here.. Is there a way to open this pull request ? or i need to open a new one ?

It sounds like the other merged PRs made a change and impact your code. Please resolve it. Thanks!

Yea. Looks like so. No worries. Let's get it tested again.

@gatorsmile @yhuai Its due to difference between scala 2.10 and 2.11 compiler in the way they deal with named parameters. Looks like 2.10 is less forgiving :-) . I have opened https://github.com/apache/spark/pull/15495 with the fix.


Working  together with java agent?

Say I have project A writing in PHP, and project B writing in JAVA, A call B via curl, both A AND B use skywalking,  but in Topology they are not connected  and  in Traces they are not work together.

One possible issue, which version of header is uesed? Java agent has already moved to v2 header by default, you could find how to active v1 header.  @heyanlong which version of PHP SDK is using?

One possible issue, which version of header is uesed? Java agent has already moved to v2 header by default, you could find how to active v1 header. @heyanlong which version of PHP SDK is using?

See some sw3 in php agent source code, but I am not familiar with php extension, will try  v1 header. Thanks.

sw3 is v1 header.

One possible issue, which version of header is uesed? Java agent has already moved to v2 header by default, you could find how to active v1 header. @heyanlong which version of PHP SDK is using?

Using v1 header works. Thanks.


fastly: 10.8.5 -> 10.8.7

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/fastly/cli/releases.
meta.description for fastly is: Command line tool for interacting with the Fastly API
meta.homepage for fastly is: https://github.com/fastly/cli
meta.changelog for fastly is: https://github.com/fastly/cli/blob/v10.8.7/CHANGELOG.md
Updates performed

Golang update

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 10.8.7 with grep in /nix/store/80x85585815dbqzyhi5chw210bn5i3pk-fastly-10.8.7
found 10.8.7 in filename of file in /nix/store/80x85585815dbqzyhi5chw210bn5i3pk-fastly-10.8.7

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
fastly

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/80x85585815dbqzyhi5chw210bn5i3pk-fastly-10.8.7 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A fastly https://github.com/r-ryantm/nixpkgs/archive/5f9b1cadd617ee212109498ffcc9937f9c374a50.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/5f9b1cadd617ee212109498ffcc9937f9c374a50#fastly

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/80x85585815dbqzyhi5chw210bn5i3pk-fastly-10.8.7
ls -la /nix/store/80x85585815dbqzyhi5chw210bn5i3pk-fastly-10.8.7/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package built:

fastly

Maintainer pings
cc @ereslibre @shyim for testing.

10.8.8 was released


Add shapely.geometry.polygon.orient to all

Add orient to __all__ in polygon.py, supersedes #1805.
Fixes #722 where linting warnings like this appear:

'orient' is not declared in __all__

Thanks!
(Note that the errors are unrelated)


[Package Issue]: Microsoft.Office does not have previous versions available

Please confirm these before moving forward

[X] I have searched for my issue and not found a work-in-progress/duplicate/resolved issue.
[X] I have not been informed if the issue is resolved in a preview version of the winget client.

Category of the issue
Other
Brief description of your issue
Old versions of this package seem to be completely absent?

Now compare to another package:

Steps to reproduce
Run show command for Microsoft.Office
Actual behavior
The old versions are not available
Expected behavior
That other versions are available like other packages. This is essential behaviour when the recurring Office hash version mismatch occurs. We should be able to pick an older version if we cannot use the latest.
https://github.com/microsoft/winget-pkgs/issues?q=Microsoft.Office+label%3AError-Hash-Mismatch
Environment
Windows Package Manager v1.4.10173
Copyright (c) Microsoft Corporation. All rights reserved.

Windows: Windows.Desktop v10.0.22621.1413
System Architecture: X64
Package: Microsoft.DesktopAppInstaller v1.19.10173.0

Logs: %LOCALAPPDATA%\Packages\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe\LocalState\DiagOutputDir

User Settings: %LOCALAPPDATA%\Packages\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe\LocalState\settings.json

Screenshots and Logs
No response

Microsoft.Office uses a URL that always points to the latest version which is why there's no previous versions.
As a workaround for hash issues, you can run winget settings --enable InstallerHashOverride and pass --ignore-security-hash to the winget upgrade command.

Yes, it is true. Office Update is always point to catalog.update.microsoft.com but sometimes the link is directly passing to a newer version. If you want to know more about the office, Please see https://learn.microsoft.com/en-us/officeupdates/release-notes-microsoft365-apps


migrate log4jto slf4j

org.openrewrite.java.logging.slf4j.Log4jToSlf4j
when i'm using this recipie it says recipie not found other all recipies working fine in my gradle project.

Did you add the required dependency? Sounds like the recipe is not available to the plugin. The full config is here.
plugins {
    id("org.openrewrite.rewrite") version("6.8.2")
}

rewrite {
    activeRecipe("org.openrewrite.java.logging.slf4j.Log4jToSlf4j")
}

repositories {
    mavenCentral()
}

` {
    rewrite("org.openrewrite.recipe:rewrite-logging-frameworks:2.4.3")
}

As per: https://docs.openrewrite.org/recipes/java/logging/slf4j/log4jtoslf4j
It looks like you're missing that rewrite("org.openrewrite.recipe:rewrite-logging-frameworks:2.4.3") under dependencies { ... }. Please double check and let me know if that helps.
I'll monitor for replies here but will close the issue already, as this isn't as much an issue with the docs but a question about the use. You're also welcome to ask questions via our Slack.


Failing test case for Pair with Target Sum (easy)

https://dvpr.gitbook.io/coding-interview-patterns/2.-pattern-two-pointers/2.1-pair-with-target-sum-easy#:~:text=int[] copyArray,return result%3B
This code does not work for following examples

When there are duplicates in the array.
[2,5,5,11]
10

Actual Output: [2,2]
Expected output : [1,2]

@anchitbhuhan we have to make sure to iterate our index after comparing one variable. Also we can introduce a check i.e. arr[i] != arr[i+1].

@anchitbhuhan In our case, we are not considering duplicates in the input array.
You can incorporate @ngarg19's suggestion.


Secrets are not shown in GM screen

If a Journal entry contains a secret block, it should be shown to the GM in the GM screen, but it is currently hidden.
Reproduce

Create a journal entry with two paragraphs
Select second paragraph and apply format "Custom / Secret"
Save journal entry and add it to GM screen

Oh curious. I bet this is because I disable editing for all GM screen journal entries. Nice find and thanks for the report!


Grub: default is signed int. Fixes #42152

Motivation for this change
grub.default is type signed int but grub supports passing "saved" as the default option to boot the previous os.
Breaks compatibility with exsisting nixos configs but just requires changing the option from int to str.
Things done
set boot.loader.grub.default to type.str and set install-grub.pl to use str

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[X] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests) tested against boot.nix boot-stage1.nix
[X] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[N/A] Tested execution of all binary files (usually in ./result/bin/)
[X] Fits CONTRIBUTING.md.

@GrahamcOfBorg test boot.biosUsb

No attempt on aarch64-linux (full log)
The following builds were skipped because they don't evaluate on aarch64-linux: tests.boot.biosUsb
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Success on x86_64-linux (full log)
Attempted: tests.boot.biosUsb
Partial log (click to expand)
machine# [  575.313458] [2031]: Failed to unmount /nix/.ro-store: Device or resource busy
machine# [  575.314430] [2032]: Unmounting '/iso'.
machine# [  575.314793] [2032]: Failed to unmount /iso: Device or resource busy
machine# [  575.316568] ACPI: Preparing to enter system sleep state S5
machine# [  575.317072] reboot: Power down
collecting coverage data
syncing
test script finished in 605.13s
cleaning up
/nix/store/p7ljxs6yc50y2cn81m36h3z963g6hdd1-vm-test-run-boot-bios-usb


[GSB] Consistently use nested type name match constraints.

We generated a mix of "inferred" and "nested type name match"
constraints for the case where we had two nested types with the same
name and inferred that they are equal. Make them consistent by always
using nested type name match constraints. This fixes a bug where we
would get different canonical generic signatures in different source
files because we inferred the same-type constraint with different
requirement sources.
Fixes rdar://problem/48049725.

@swift-ci please smoke test

@swift-ci please test source compatibility


Is there a way to display an upload progress?

Hi
I didn't a way to display a progress bar when create model with large file for example.
Any hints?
Thanks

I'm not sure if there is a reliable cross-platform solution for this.
It depends on a lot of factors, including how the MultipartFile instance was created/loaded (eg. fromBytes, fromFile, etc.). If you are targeting only web, the browser's (dart:html) XHR provides a upload progress event that eventually could be used.
I'll consider experimenting and eventually adding a onSendProgress callback/stream, but this for now is a very low priority.
You can also check https://github.com/dart-lang/sdk/issues/35427.

Thanks for the quick answer!
Let's wait for dart-lang to implement this. Like using fetch to enable stream on web
You can close this issue if you want to

This is a pretty important missing feature at the moment.
Working with files using pocketbase/s3 is a very pleasent experiance, but at the moment, not being able to recieve upload progress in either the admin panel or through the sdk makes using this quite tricky to provide feedback to users where uploads might take 30-60 seconds.


Update cats-effect to 3.3.3 in series/0.3

Updates org.typelevel:cats-effect from 2.5.4 to 3.3.3.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Applied Scalafix Migrations

org.typelevel:{cats-effect,cats-effect-laws}:3.0.0 (created no change)

github:typelevel/cats-effect/v3_0_0?sha=v3.0.0

org.typelevel:{cats-effect,cats-effect-laws}:3.3.0 (created no change)

github:typelevel/cats-effect/v3_3_0?sha=series/3.x

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.typelevel", artifactId = "cats-effect" } ]

labels: library-update, early-semver-major, semver-spec-major, scalafix-migrations, commit-count:1

This looks bizarre because we have Steward config with CE2 pin.

This looks bizarre because we have Steward config with CE2 pin.

Might that trailing comma have caused the file to fail to parse?

Might that trailing comma have caused the file to fail to parse?

Yep, confirmed with a test in scala-steward.  Our config is invalid HOCON (at least as Circe parses it).

Yep, confirmed with a test in scala-steward.  Our config is invalid HOCON (at least as Circe parses it).


Added a new --throttle option

hello,
Would it be possible to add a new --throttle option to easily define the fraction of time that threads should be idle?

what would the usecase be?

The options --threads and --throttle would allow us to finely configure the computing power and thus the system load. Especially since the official client already allows it:
.setThrottle(throttle)
Set the fraction of time that threads should be idle.

I support this idea
On Wed, Dec 13, 2017 at 11:04 AM, gnomelibre<EMAIL_ADDRESS>wrote:

The options --threads and --throttle would allow us to finely configure
the computing power and thus the system load. Especially since the official
client already allows it:
.setThrottle(throttle)
Set the fraction of time that threads should be idle.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/cazala/coin-hive/issues/72#issuecomment-351313670,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKqeD7zCLrTic0pyHcH0naXH4fXpdBpkks5s_4UBgaJpZM4Q-T3S
.

--
Much Regards
Branson G Kuria
Systems Developer
Sabek Systems
P.O Box 78024
Nairobi.
www.sabek.co.ke
+254 711 657108

sorry i misunderstood when I read this yesterday :P i thought you were asking for an option to specify the idle time (like in milliseconds) instead of a percentage.
Yes sure, i'll add the option

this is available on version v1.10.0


Httpbackend not working with Protractor v5.0+

Node Version: 6.9.0
Protractor Version: 5.1.1
Browser(s): Chrome
Breaking Change

I have my protractor tests set up to use http-backend-proxy which uses httpbackend, and everything works fine with protractor version 4.0.14, but in version 5.0+ they fail with an InvalidElementState error.

InvalidElementStateError: invalid element state: Failed to execute 'querySelector' on 'Document': The provided selector is empty.
(Session info: chrome=57.0.2987.133)
(Driver info: chromedriver=2.29.461585 (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X 10.11.4 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 13 milliseconds
Build info: version: '3.4.0', revision: 'unknown', time: 'unknown'
System info: host: '...'..., ip: '......', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.11.4', java.version: '1.8.0_73'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities [{applicationCacheEnabled=false, rotatable=false, mobileEmulationEnabled=false, networkConnectionEnabled=false, chrome={chromedriverVersion=2.29.461585 (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b), userDataDir=/var/folders/sz/n9py2wkd6zdg7nl5_phn7r497yn6vr/T/.org.chromium.Chromium.Wk2KMZ}, takesHeapSnapshot=true, pageLoadStrategy=normal, databaseEnabled=false, handlesAlerts=true, hasTouchScreen=false, version=57.0.2987.133, platform=MAC, browserConnectionEnabled=false, nativeEvents=true, acceptSslCerts=true, locationContextEnabled=true, webStorageEnabled=true, browserName=chrome, takesScreenshot=true, javascriptEnabled=true, cssSelectorsEnabled=true, unexpectedAlertBehaviour=}]
Session ID: 0a8f87dee00b4e60542806e0aa4e417e

Code that now fails:
httpBackend = new HttpBackend( browser );

httpBackend
.when('GET', this.URL + 'user/self')
.respond( function() {
       return [200, $httpBackend.context.userSelf];
});

I tried updating it to the following, just to see if that would help at all, but it still fails with the same error:
httpBackend
                .whenGET(this.URL + 'user/self')
                .respond( function() {
                    return [200, {}];
                });

Any advice would be greatly appreciated!! Thanks!

I'm not super familiar with HTTPBackend but it isn't something we officially support. I'd suggest bringing this up on the HTTPBackend repo or on StackOverflow.
Good luck!

hm, ok thanks!
FYI: https://docs.angularjs.org/api/ngMock/service/$httpBackend

My apologies I mean to say http-backend-proxy, not HTTPBackend :)

This error: InvalidElementStateError: invalid element state: Failed to execute 'querySelector' on 'Document': The provided selector is empty is what you get if you call document.querySelector() with an empty string.
There is a bug where appRoot can be the empty string in some cases, but after looking through clientsidescripts.js, I don't see a case where that would make it to a document.querySelector() call.
Could you provide an example repro case? Also, a few questions:

What version of Angular is your app?
Are you using by.cssContainingText?
Do you set rootElement in your config? If you set it to 'body', does the problem go away?

Apologizes for the delay.  I didn't see a notification that you responded.  Thank you for looking into this.

Our Application is Angular version 1.5.3.
I do not currently use .cssContainingText in any of my tests.
I do not currently set rootElement in my config file.

I will send you more example information and try setting the body of root element today and get back to you!

Adding rootElement: 'body' to my config file does fix the issue!
To try to simplify the code to show you for repo, I put everything in the onPrepare here:

Do you have a simple angular site that I could point to?  I can't provide my app code for repo, but if you have an angular site that I could point to and set up to reproduce, I will do that.
Thanks again!!

I like to just use angularjs.org. That's what I did for the BlockingProxy demo.
Also, protractor has a test app in the repo. If you clone the repo and run npm start it'll bring up an app you can use for testing.

Hi!,
I have a working repo to demonstrate the issue for you: https://github.com/rfodge/protractor4249


v0.5.1

@ramppdev can you please make a new minor release with your opensmile fix? Anything else we need to pack in it?
UPDATE: I just realized this would also include the changes to the dataset format, which is technically a feature...So I guess we need to wait until #70
Turning this into a broader question then. Is it time to begin using dev branches for the new releases and keep the main free for bugfixes?

Hm yeah i think this is a very good idea, otherwise we have to postpone bug fixes in between features.


Close Keep-Alive connections following a parser error

Description

Resolves a bug in Keep-Alive handling, which results in an infinite loop (with epoll) or a crash (with GCD) when a malformed request is sent on a Keep-Alive connection, after a previous message has been successfully processed.
Motivation and Context
May address #228
Recreate is as simple as:
curl -X GET -H 'Content-Length: 0' -H 'Connection: Keep-Alive' -d"abcd" http://localhost:8080/

This problem does not occur if Connection: Close is sent. In this situation, after handling the initial request, whether successful or not, we close the connection.
The problem also does not occur if the first message from the client is invalid (regardless of the Connection header). In this case, we immediately respond with 400 Bad Request and close the connection.  This only happens due to ordering of when the clientRequestedKeepAlive flag is updated: we only set this flag to true once a request has been successfully parsed.
Symptom
In the case of epoll, we repeatedly attempt to parse the remaining buffer as the epoll_wait times out every 50ms, and we have a handler in the list of handlers that require deferred processing.
In the case of GCD, we attempt to invoke self.handleBufferedReadDataHelper() as part of a direct call to end() when a parser error has occurred.  In this situation, we attempt to synchronously enqueue a block to the queue we are already executing on, which results in a crash.
Solution
The solution presented here is to introduce a new piece of state: parserErrored, which is checked when evaluating isKeepAlive, in addition to clientRequestedKeepAlive.
How Has This Been Tested?

I ran the Kitura-net tests locally on Linux and Mac.  I tested with the curl example above to verify that the problem is fixed.
Checklist:

[ ] I have submitted a CLA form
[ ] If applicable, I have updated the documentation accordingly.
[ ] If applicable, I have added tests to cover my changes.

It's fantastic that you're fixing this stuff.  Can we add a test too?

Codecov Report

Merging #232 into master will decrease coverage by 0.04%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #232      +/-   ##
==========================================
- Coverage   74.84%   74.79%   -0.05%
==========================================
  Files          31       31
  Lines        4174     4178       +4
==========================================
+ Hits         3124     3125       +1
- Misses       1050     1053       +3

Flag
Coverage Δ

#CHTTPParser
57.52% <ø> (+0.33%)
:arrow_up:

#KituraNet
74.79% <100%> (-0.05%)
:arrow_down:

Impacted Files
Coverage Δ

Sources/KituraNet/IncomingSocketHandler.swift
78.03% <100%> (-3.66%)
:arrow_down:

...s/KituraNet/HTTP/IncomingHTTPSocketProcessor.swift
83.7% <100%> (+0.37%)
:arrow_up:

Sources/CHTTPParser/http_parser.c
57.34% <0%> (+0.34%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update db19e63...862d5c2. Read the comment docs.


webrtc.js:76 WebSocket connection to 'wss://<IP_ADDRESS>/' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED

I tun it on win
npm start
Then I go to :
https://<IP_ADDRESS>:8443/
https://prnt.sc/tykj1b
Can see my cam and got that error
webrtc.js:76 WebSocket connection to 'wss://<IP_ADDRESS>/' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED
Any idea how to fix that ?

Hey yarekc ....... Well comment line number 76 and uncomment line 75 ....and replace my ip in line 75 with <IP_ADDRESS>:8443 ... and i guess it will solve the issue . Right now it is unable to form connection with server. You can reach out again if the issue persists.

thanks. Now... I really don't unerstand how it works (or maybe it does not work)
user1 connects : he sees himself
user2 connects : he sees himself
in 90% of cases, NOTHING else happens. In 10% of cases, user1 sees user2 and user2 sees user1
It is random and mostly of time nothing happens. Sometimes, conference starts.
I have no JS errors, no server errors.
When user3 joins:

user3 can see user3 and user2
user2 can see user2 and user3
user1 sees user1

It is however very random and I never success in having correct result.
The raise hand has no effect at all, except printing "undefined" in the node console.
My question is: Is that working ?
If yes, I  suggest you publish a demo online

Do you have latency issues ?  Because if page takes more than 2 seconds to load then it breaks and can be modified in set-timeout function at line 81. Also i see you are getting random results. It occurs if you are a low bandwidth client. Are you getting an alert message of You are running on low bandwidth ??
Well algorithm should work like this - >
user1 sees user1(user 1 is the master)
user2 can see user2(own stream) and user1
user3 can see user3 and user1
user4 can see user4 and user2
user5 can see user5 and user3
user6 can see user6 and user4
user7 can see user7 and user5
basically a branched chain if formed.
See my working demo
https://detectronlive123.000webhostapp.com/1.mp4
https://detectronlive123.000webhostapp.com/2.mp4
The algorithm is as follows -
https://detectronlive123.000webhostapp.com/algo.png

I have a high speed fiber connection.
Infact I think don't understand the purpose of this project. What is the purpose user7 sees user 7 and user 5 ? (And not other users ?) In real life?
I think it can be useful if any user can see any other user he decides.
Regards

This was my research intern project at IIT Kanpur. You dont have to see any other user in the broadcast. What exactly happening here is that master's feed should be visible to all (since master is the one broadcasting his feed). Now master is sending its stream to user 2. user4 is getting the feed of master from user 2 (relaying of streams). User 6 is getting master's stream from user 4 and so on. This way there is no load on master and p2p streaming is taking place. You should read about the algorithm here - https://docs.google.com/presentation/d/1YLxouDWRVHrKxgG6sKfOq8Ge4pQXBH7tD2rU1WpY1Yo/edit?usp=sharing
This is quite scalable. We made this so that teacher can teach a class of 100 students. Students need not be able to see themselves but everyone will see teacher's stream.
If you want that everyone should see everyone then follow - https://github.com/akshat7101999/WebRTC-mesh-group-video-call
It is in no way scalable and with 5 6 users in video conferencing your bandwidth will be exhausted and so will be you cpu

I understand the concept of mesh, peer2peer and scalable broadcast.
Here is what you write:
user1 sees user1(user 1 is the master)
user2 can see user2(own stream) and user1
user3 can see user3 and user1
user4 can see user4 and user2
user5 can see user5 and user3
user6 can see user6 and user4
user7 can see user7 and user5
Question is: WHY all these users do not see the user1 ?
They should ALL be able to use user1 otherwise what is the purpose of having a mesh like
user19 is only able to see user598595 and user5455 only to see user511255 : for me that has no sense.
I guess the Goal of the project is that ALL users are able to see the SAME user (which would not be possible to obtain with a simple mesh, at least not when you have >20 users)

Question is: WHY all these users do not see the user1 (the master user) ?
They are all seeing master only ...... user4 can see user4 and user2--- By this i dont mean that user4 is getting stream of user2. By this i mean that user 2 is getting stream from master and then user2 is forwarding the received stream to user4. Basically user2 is relaying the master stream to user4. So user 4 instead of connecting directly to master is getting master's stream only but from user 2. I should have cleared it in the beginning.

ok, but still not unclear.

So the goal of this project is to have 1ToN broadcast.
Let say user1 is braodcatser, so user2, user3, user4, userN can watch him (N can be >20)
Is that correct ?

user2, user3, user4, userN  cannot watch user1 until... they stream themselves and thus they allow access to microphone and webcam. (if user2 does not accept webcam access, he has BLACK SCREEN)

Imagine that if you want to watch a TV stream on your PC, you have to enable access to ... your cam and microphone !
Would you accept that ?

Ya i agree but this project is just a good starting point i would say. It is far from perfect. My internship is over and the rest are building on top of it. One potential solution to the issue you just raised is to remove your stream once ICE candidates are exchanged which can be incorporated in webrtc.js file so that there is only one way streaming.
The point is that it is just a demonstration and there is quite a room for improvement.

ok
I think you really should work on that project : it can be a real success ... if it works in real life conditions.

Thanks i will try my best :)


VS Code Extension - Unsaved document issues

I'm having some issues when trying to format a document that has not been written to a file (an unsaved document tab in VS Code). When triggering the Format Document (Shift + Alt + F) command, VS Code indicates that the formatting is in progress. However, the formatting never finishes and VS Code keeps showing the busy indication indefinitely or until I switch to another document.
After starting the formatting, I can observe:

The document does not get formatted
VS Code Memory usage increases by about 10 MB per second (no end in sight)
VS Code uses about 12% CPU
When restarting VS Code and switching to the unsaved document I tried to format, the Memory leak and CPU usage shows the same pattern again (without triggering another format, see [1] for a possible cause)
Closing the document does not stop the Memory leak or the CPU usage
Disabling the CSharpier extension (and performing the reload that VS Code suggests) does not stop the Memory leak or the CPU usage
When the CSharpier extension is not installed or disabled when starting VS Code, non of the above behavior occurs (or in other words: everything works as expected)

[1]: After some further experimentation I noticed that as soon as I switch to an unsaved document in language mode C# (can be an empty document), the memory leak and CPU usage occurs. So this issue is not only triggered by formatting, but also by switching to an unsaved document in language mode C#. However, opening a new document (no language specified) and pasting C# code or manually changing the language mode to C# does not trigger this behavior (but selecting any other document and switching back to this new one does).

CSharpier Info:
Extension Version: v1.2.1
CSharpier is globally installed with version: 0.16.0
VS Code Info:
Version: 1.66.2 (user setup)
Commit: dfd34e8260c270da74b5c2d86d61aee4b6d56977
Date: 2022-04-11T07:46:01.075Z
Electron: 17.2.0
Chromium: 98.0.4758.109
Node.js: 16.13.0
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.19044

Thanks for all of the details. I was able to recreate this using the basic steps you provided so I think I have all the info I need to get a fix out. I am fairly certain I tested formatting unsaved documents with the initial version of the vscode extension, but I don't have it in my test cases so must have broken it at some point.
I should have a fix for this out this weekend, possibly even tonight if I can squeeze it in.

I have a quick fix for the problem already, it was a bad assumption inside of a loop that caused it to loop forever.
Version 1.2.2 should be available shortly and fixes the issue. I'll clean up the fix this weekend when I have more time.
Sorry for any pain and suffering this caused you!

Thanks for your quick reply and fix!
While the memory and CPU issue is fixed, the document still does not get formatted.
Output from a successful format of a saved document:
["INFO" - 07:44:36] Formatting started for <path-to-file/>.
["DEBUG" - 07:44:36] Got chunk of size 160
["DEBUG" - 07:44:36] Got last chunk with ETX at 159
["INFO" - 07:44:36] Formatted in 1.7976000308990479ms
["DEBUG" - 07:44:36] result is empty

Output when attempting to format an unsaved document:
["INFO" - 07:44:24] Formatting started for Untitled-1.
["INFO" - 07:44:24] Formatted in 0.30149996280670166ms
["DEBUG" - 07:44:24] result is empty

Ah whoops! I was doing my quick test with just an empty document so didn't notice that. I will have a proper fix out for it this weekend, and I'll make sure to test properly then.

Version 1.2.3 should be available shortly and actually formats unsaved documents now.
It will treat the unsaved document as if it were in the root of your workspace, using a local version of csharpier if there is one and the global version if there is not. In the case of a multi-root workspaces it will use the first workspace. I imagine c# projects don't often use that feature and wasn't sure how else to handle the situation.

Thanks, the issue seems to be resolved.


Disallow timeseries queries with ETERNITY interval and non-ALL granularity

A user might construct a timeseries query on inline data by mistake which contains no filters on time intervals (hence the query is on ETERNITY interval) and also specifies a non-ALL granularity. Currently, those queries will start generating a large number of time grains while generating the results which can lead to a very high chance for heap OOM or a very very long query runtime.
This change disallows such queries from execution to avoid the problems mentioned above. Semantically as well, there is less point in running a time granularity query over the ETERNITY interval.
An augmentation to the solution could be to find the time-interval of the inline data by traversing it and then using that interval in the RowBasedStorageAdapater. But doing that change today will either require recompuation of inline data or eager materialization of it in the adapater, both of which approaches have their own short-comings. So, for now this change is refrained - it can be considered in future if needed.
This PR has:

[x] been self-reviewed.

[ ] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)

[ ] added documentation for new or modified features or behaviors.
[ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[ ] added or updated version, license, or notice information in licenses.yaml
[x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[ ] added integration tests.
[x] been tested in a test Druid cluster.

Could we make an issue for this to track it? Reason being this seems sort of sad, but agree its better than running out of memory or whatever

I think the current check would be needed even if we evaluate the exact time interval for data in RowBaseStorageAdapter since there can be data a custom time column, empty data or a user degenrate case which can lead to an large time interval. Thinking about this again, do you think it would be better if we cap the number of time-grains possible to create? the interval and granularity could be used to find the estimated number of intervals.
Regardless, I will also open an issue for tracking exact time interval for inline data since that'll be generally helpful.

One more idea around this is to give a lazy support for descending time-grain iterators. The OOM I have seen are because the reversing of iterators materializes them which explodes the heap memory usage. With that change, the queries on large intervals with time-granularity will run slowly but will have much lesser chances of OOMing.

@rohangarg - shall we merge this PR for now?


[Feature Request] a CLI that output unused field/argument in the schema

Hello there 🙂
I would love that ferry, thanks to generated AST files, outputs unused part of my schema.
I can make a contribution if I have some directions.

Right now, I am blocked, I don't know how to import in a single file all generated AST files... except with a builder.
Then, how to compare definitions and usage ? By hand ? Is there a package I don't know that can help ?

Cheers !

Sounds interesting, but I can't give any meaningful input on that at the moment.
I do think that working on the graphql files directly would be easier though, like https://www.npmjs.com/package/@ovrsea/graphql-detect-unused-operations does for example


Create Local server

Operating system is not important but it have to support SQL Server ^2016 and web deployment in IIS Settings because the build proccess is using web publish profiles to upload and configure the app
https://ci.appveyor.com/project/Magik3a/patientmanagement-admin
Later we will discus CI/CD on meeting

Suggestion for cluster management
https://docs.linbit.com/

Suggestion for monitoring system
https://www.nagios.com/solutions/web-application-monitoring/

Suggestion for monitoring (free)
https://www.zabbix.com/

verry good article about the issue
https://www.digitalocean.com/community/tutorials/5-common-server-setups-for-your-web-application

floating IP
https://www.digitalocean.com/community/tutorials/how-to-use-floating-ips-on-digitalocean#how-to-implement-an-ha-setup

Monitoring Free
https://prometheus.io/

Load balancer
http://www.haproxy.org/


Add Motif.h to podspec

Currently #import <Motif/Motif.h> doesn't work when using Motif via CocoaPods

Fixed in https://github.com/erichoracek/Motif/commit/ceb4dfd103f25bc5819f443aa0333f172919ce6a

Ha! Thanks!
I'm integrating this into a project now, so I'll be documenting issues as I can as I go. If I see how to handle them myself, I'll make PRs in the evening/weekend. I was fully expecting to handle this one tonight :)

Sure thing! Glad to have you testing things out.


Compiler supports converting md files in workspace to js

What

The workspace command will now convert md files to js
Added file snapshots for JS/Swift/Documentation generated from tokens
Removed the "// Generated by ..." banner on snapshot files for less noisy diffs (in the future)

Note: The generated JS probably isn't exactly what we want, but it should at least be valid JS
@mathieudutour

One thing to note is that we should either generate .mjs files or compile the js files. You can't put export let x ='' in a node package.

Ah, good point. Hmm... I guess I'll use .mjs for now, and if it becomes an issue for people (or us) at some point, we can do something else

👍 We will still need compile jsx but we can do that "manually", no need for babel.


Consistent handling of sparse data in fit and transform

As per #81 there are some issues with how transform manages to deal with sparse/dense data. This should be made consistent so that you can (ideally) use any mix of sparse and dense as required cleanly with both fit and transform.

Hi,
Still showing error for umap-learn==0.5.1
should we use dense for  .transform ?
.fit works well with sparse


Fix #1359 through better table resizing

This PR fixes our VaporTable resizing by doing 3 things:

The VaporTable would previously insert a QWidget into each table cell.  For cells that only display text, we now use a QTableWidgetItem, which plays nicer with resizing headers.

This PR removes the "Type" column from the table displaying our renderers.  This can be re-added, but I believe that the Name and Type columns are redundant.

Removes unused logic for the VaporTable that allows us to select columns in the same way that we select rows, when we select our renderers.  I don't foresee us needing this logic for columns.

Old:

New:

Is there an issue that should be linked from this PR?

@clyne I guess adding the issue in the title does not create a link.  This is for #1359.

@clyne I guess adding the issue in the title does not create a link. This is for #1359.

The PR's and issues won't get cross-linked unless you add the issue number to the PR comment with syntax GitHub understands (e.g. Fixes #1359)

If you have a file with a long name (e.g. the Leigh Orf's CM1 files: 24Maycontrol.04000.000000.nc) the text wraps around and the cell expands. Is that what we want or should we just truncate the text (hovering with the mouse displays the full text)? Just wondering.

@clyne That's disappointing to see.  A test case I missed, and I don't know why this is happening.  My first guess is because of multiple decimal points in the file string.  When I load long WRF files, they get truncated with ellipses.  Screenshot below.

The changes in this PR require a fair amount of testing.  With next week's tutorial, I don't think I'll be able to submit a fix for this with confidence before Tuesday.  I do think the current PR is an improvement, but not maybe perfection.

@StasJ

You said you removed column selection because we wouldn’t use it for renderer selection but I think VaporTable is used for other things too.

Yes, we do use the VaporTable in other places but we don't use column selection anywhere, so I removed it instead of duplicating the logic being used for the rows.

@clyne Stas has approved, so I'm going to merge so I can kick off the installers


Error message when I try to goto the terminal.

Hi, Everything was working until I exited the terminal and tried GotoBuffer to that terminal. When doing so, I got:

I already resolved the issue and made it vim compatible.
Should I make a PR.

115% you should make a PR!
I would love to have this be vim compat as well.

Btw, I think I am going to rewrite this and make it better shortly!  So if you don't do anything, i'll be doing something shortly.

I have changed all the things so that this should not be an issue.
My guess is that this is old and is no longer an issue, if you wish to give it another try, cheers.
also, I have integrated a marking mechanism.  Its purdy fancy.  But there may be an issue on empty projects i'll try to take care of shortly.
For now I am going to close this.


How can I retrieve matrix from command

When I tried to operate matrix, it seems the library doesn't return it.
rio.$e({
  command: "a = matrix(1:6, nrow=2, ncol=3, byrow=T)"
}).then(data=>{
  console.log(data);
})

nothing printed. Is there a way to get the matrix value to a 2-d array so that my js code can use it to do something else?

Thanks for the feedback.
Firstly a solution.
rio.$e({
    command: "require(RJSONIO); a=matrix(1:6, nrow=2, ncol=3, byrow=T); toJSON(a)"
}).then(data => {
    console.log(JSON.parse(data));
}).catch(function(err) {
    console.log(err);
})

And the output
> rio.$e({
...     command: "require(RJSONIO); a=matrix(1:6, nrow=2, ncol=3, byrow=T); toJSON(a)"
... }).then(data => {
...     console.log(JSON.parse(data));
... }).catch(err => {
...     console.log(err);
... })
Promise { <pending> }
> [ [ 1, 2, 3 ], [ 4, 5, 6 ] ]

Coming back to the initial snippet, it is correct there is not anything displayed.
Of course, it doesn't mean rio works ok.
Indeed, adding the catch part an error is displayed.
Then, you can enable the debug flag: rio.enableDebug(true);.
> rio.$e({
...     command: "matrix(1:6, nrow=2, ncol=3, byrow=T)"
... }).then(data => {
...     console.log(data);
... }).catch(err => {
...     console.log(err);
... })
Promise { <pending> }
> Connected to Rserve
Supported capabilities --------------

Sending command to Rserve
00000000: 0300 0000 3400 0000 0000 0000 0000 0000  ....4...........
00000010: 0430 0000 6d61 7472 6978 2831 3a36 2c20  .0..matrix(1:6,.
00000020: 6e72 6f77 3d32 2c20 6e63 6f6c 3d33 2c20  nrow=2,.ncol=3,.
00000030: 6279 726f 773d 5429 0001 0101 0101 0101  byrow=T)........
00000040: 0101 0101                                ....

Data Header
00000000: 0100 0100 3800 0000 0000 0000 0000 0000  ....8...........

Data packet
00000000: a030 0000 1514 0000 2008 0000 0200 0000  .0..............
00000010: 0300 0000 1304 0000 6469 6d00 0100 0000  ........dim.....
00000020: 0400 0000 0200 0000 0500 0000 0300 0000  ................
00000030: 0600 0000                                ....

Type SEXP 160
Type 160 is currently not implemented
true
Disconnected from Rserve
Closed from Rserve

The relevant part is Type 160 is currently not implemented.
I am afraid I didn't implement that structure.
Contribution is welcome.

Great solution, very helpful. Thanks.

I investigated a bit.
160 === 0xA0 is the first byte of the data packet.
If the type is greater than 128, it means there are attributes, in terms of SEXP parsing.
Indeed  160 & ~128 is equal to 32, the type for integer array, exactly as the numbers contained in the matrix.
Really attributes implementation is quite cumbersome.
They are a sort of metadata and they make sense in R context.
In JS, when the structure is very complex, the JSON serialization is the way.
(Added in todo list; don't hold the breath) :)


Receive money via messaging

[ ] New attachment type: payment

We wont add a new type for this, we are going to use embedded apps instead.
see https://github.com/rogerthat-platform/rogerthat-backend/issues/448


Improve from

Description:
I tried to keep the changes down to a minimum here, if we feel like they need to split out into several PR's I'm okay with that.  This may add pain for consumers that don't use the es6.lib.d.ts declarations.  There is work on the TypeScript to help with this pain, see https://github.com/Microsoft/TypeScript/pull/6990 and https://github.com/Microsoft/TypeScript/issues/6974.  Today we could point them at spec/es5.d.ts as a small shim they can use.

Replaces Iterator<T> with Iterable<T>.
Introduces Observablesque<T> which uses the symbol.

NOTE: The reason these are grouped is both 1 and 2, are changes that break compilation.  We have to shim in SymbolConstructor and Iterable<T>.
Details:
Basically what's going on...
Right now we use Iterator<T> as part of ObservableInput<T> but really the interface we're looking for (in code that is) is Iterable<T>.  So this change fixes that.
interface Iterable<T> {
    [Symbol.iterator](): Iterator<T>;
}

Also we declare Symbol.observable but this isn't declared anywhere in the interfaces, so there isn't an easy way to add that to ObservableInput<T> without an interface.

Rebased.

LGTM

The last thing I thought about, from a release perspective.  Should we copy that es5.d.ts file into the dist outputs?  That way if a user comes along with a project that breaks because they don't have Symbol, they can add /// <reference path="./node_modules/rxjs/es5.d.ts" /> to easily fix the problem.
I think at the time being this mainly applies to commonjs builds only I think.

@david-driscoll what about publish it into typings and let it be installed as same as other type shims?

I like the idea.  Proposed name es6-rxjs-shim as it's specific to what rxjs needs.
I'll also add some notes to https://github.com/ReactiveX/rxjs/blob/master/doc/installation.md to direct users to install the typing if they are compiling their TypeScript project with es5 instead of es6.

I'll go through changes once again and check this in later today to tomorrow. type definition shim can be dealt with separate PR, I assume.

Sure thing, as long as the name works I'll start the work getting it into the typings registry in parallel.

Can we test this against core-js and the like? Anything that introduces types like Symbol to the global namespace is almost guaranteed to cause a huge amount of pain downstream...

@david-driscoll , let me hold off check in bit - would you able to clarify @robwormald 's request?

@robwormald does core-js have a like typescript typings?  This change doesn't actually shim Symbol anywhere, instead it just allows us to use Symbol in es5 like code, so that our interfaces can accept a spec based observable.
The following just won't work, unless the TypeScript code has Symbol.
export interface Observablesque<T> {
  [Symbol.observable](): Subscribable<T>;
}

The problem is lib.d.ts does not have Symbol defined and lib.es6.d.ts does.  Anyone target es6 from TypeScript will be able to consume this interface.  The es5.d.ts (name can be changed) is so that someone targeting es5 from TypeScript will be able to still use the library.  It is intentionally very slim and only introduces the types that rxjs needs to be able to consume Iterable<T> and Observablesque<T>.
interface Symbol { }
interface SymbolConstructor {
  iterator: symbol;
}
declare var Symbol: SymbolConstructor;
interface Iterable<T> {
    [Symbol.iterator](): Iterator<T>;
}
interface Iterator<T> {
    next(value?: any): IteratorResult<T>;
    return?(value?: any): IteratorResult<T>;
    throw?(e?: any): IteratorResult<T>;
}

Can we test this against core-js and the like?

Give me all of your PRs, @robwormald

@david-driscoll @kwonoj I labelled this "LGTM" a while ago... is there still work being done here?

Change itself is ready, wanted to ensure and clarify @robwormald 's question. I'm planning to check this in around tomorrow.

Along the lines of the discussion with @kwonoj and @robwormald I've created a typings repo, for es5.d.ts. I've added some content to the readme to explain the motivation behind it.
https://github.com/david-driscoll/rxjs-symbol-typings
If someone wants to move this to ReactiveX org, I'm happy to facilitate the move.  Or just make a new repo, clone and push this content up, that's cool too.
Once this gets merged into the typings registry, I can create a follow up PR that removes es5.d.ts and uses the new typings.  In addition I'll add a few notes to the getting started docs so that users targeting es5 will know that they need to have this additional typing installed to compile correctly.

rebased

@david-driscoll looks like there's still a build failure. Might want to dig into that.

Actually it looks like it has something to do with the merged mocha changes. cc/ @kwonoj

@blesh  Nah, it was my bad, I missed something on rebase... third time is the charm I think

Merged with c6ef2e8, thanks @david-driscoll


Can't get it to work

I read the instructions but I still can't get it to work. What am I doing wrong?

Try reading harder.


Using Hierarchical Partition Keys, and SLA

This feature is marked as "preview". Is this a public preview? What do developers need to do in order to make use of this feature? And is there a SLA for this preview feature?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d9e09a48-2012-f952-ddb5-cf15a6c8409b
Version Independent ID: 2c58e064-9a51-a518-5ca7-9c8589a55e5a
Content: Frequently asked questions on hierarchical partition keys in Azure Cosmos DB (preview)
Content Source: articles/cosmos-db/hierarchical-partition-keys-faq.yml
Service: cosmos-db
GitHub Login: @seesharprun
Microsoft Alias: sidandrews

@robtextrequest Thank you for the feedback. We are actively investigating and will get back to you soon.

#reassign:seesharprun
This is a public preview. The Getting Started documentation will guide you on how to try this feature out.
#please-close


Seeking someone to take on the project?

I would love some help with collecting and organizing the data? Seeking a new long term owner.

Hi Joe- I'd love to talk to you more about the data/project you're working on.

We can discuss here in open if you would like or email me. I'm looking for someone to give this repo some love and attention. It is rather stale right now.

Are you still looking for someone? @jokecamp

Yes

I am very interested. I am currently an undergrad CS student at the University of Michigan, Ann Arbor. My main area of expertise is C++ App design, Genetic Algo and such but I am trying to branch out into Python, Web dev etc. I would be able to devote ~50-60 hours to this over the next 10 days or so because of a break. And after that, it'd probably be close ~1-2 hours a day. Let me know if you have any questions.

@jokecamp

This is just a volunteer job. Looking for someone to add and organize football data. There are no qualifications needed. If you want please start making pull requests and we can get you involved to a point for taking over the repo. Does that make sense?

Yup! Makes sense. Thank you!

Hey @jokecamp, I know I'm quite late to this but keen to get involved in this project if you're still looking to keep it up to date and add to it.

Hey @jokecamp, I know I'm quite late to this but keen to get involved in this project if you're still looking to keep it up to date and add to it.

Please feel free to start with Pull Requests and eventually you can help own the project. Pretty much all PRs get approved.

Please feel free to start with Pull Requests and eventually you can help own the project. Pretty much all PRs get approved.


Bazel: fix proto_library cc rule [phab:D302]

Created by schroederc at 2015-06-02 17:19:25:

shahms accepted this revision at 2015-06-02 17:21:13

schroederc closed via commit 502ce4e9e56b633c2e1d76bbc6cab6e55374c0de at 2015-06-02 17:22:29


1.4.1 Beta - Crash on Community Patch Options menu

Instantly crashes the game when applying no changes in Community Patch Options menu

Fixed with #340 .


[SPARK-14111][SQL] Correct output nullability with constraints for logical plans

What changes were proposed in this pull request?
JIRA: https://issues.apache.org/jira/browse/SPARK-14111
We use output of logical plan as schema. Output nullablity is important and we should keep its correctness.
With constraints and optimization, in fact we will change output nullability of logical plans. But we don't reflect such changes in the output attributes. So the output nullablity is not correct now.
How was this patch tested?
Modified InferFiltersFromConstraintsSuite and AnalysisSuite. Existing tests.

Test build #53997 has started for PR 11926 at commit 5912fd3.

Test build #53997 has finished for PR 11926 at commit 5912fd3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

retest this please.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53997/
Test FAILed.

Test build #54002 has started for PR 11926 at commit 5912fd3.

Test build #54002 has finished for PR 11926 at commit 5912fd3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54002/
Test FAILed.

retest this please.

Test build #54015 has started for PR 11926 at commit 5912fd3.

Test build #54015 has finished for PR 11926 at commit 5912fd3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54015/
Test FAILed.

retest this please.

Test build #54024 has started for PR 11926 at commit 5912fd3.

Test build #54024 has finished for PR 11926 at commit 5912fd3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54024/
Test FAILed.

retest this please.

Test build #54045 has started for PR 11926 at commit 5912fd3.

Test build #54045 has finished for PR 11926 at commit 5912fd3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54045/
Test FAILed.

Test build #54164 has started for PR 11926 at commit 13ad172.

Test build #54164 has finished for PR 11926 at commit 13ad172.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54164/
Test FAILed.

Test build #54283 has started for PR 11926 at commit c6dabcd.

Test build #54283 has finished for PR 11926 at commit c6dabcd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54283/
Test FAILed.

Merged build finished. Test FAILed.

The failed tests are due to another bug in Expand operator's constraints. Submitted #11995 to fix it.

Close this and think better solution.


LiveEdit Research

We should make a research whether it is possible to perform JavaScript live sync without restarting the application.

Communication
The easiest way to communicate a change in a JavaScript file to the runtime is through the debugging channel. We already have an established connection that uses the debugging protocol so we may reuse it.
When we execute live edit command from Terminal, CLI should open a connection to 18181 port. When a JavaScript file is modified, CLI should send a Debugger.setScriptSource json message that would contain the modified content. Debugger.setScriptSource requires an integer that represents the identifier of the changed script. All script identifiers are received through Debugger.scriptParsed so when a connection is opened CLI should start listenning for those messages and record a map of identifier/file path.
Runtime Implementation

[Could be done later] Currently the debugger communication goes through a single hardcoded port. This means that it would be really hard (or impossible) to have an attached debugger and a live edit session at the same time. https://github.com/NativeScript/ios-runtime/issues/511
Extend Debugger.json with a Debugger.setScriptSource message.
Change the diff algorithm that is used in v8 so that it could fit into our needs. The diff algorithm would then tell us how lines/columns has changes so that we could update only the function exectables that are changed
Add a custom source provider that would derive from JSC::SourceProvider but would have the ability to modify its source.  When a Debugger.setScriptSource message is received we could find the exact module with the specific source id and change its source content through our custom source provider to the new source content (e.g. https://github.com/NativeScript/ios-runtime/commit/8b5973b04f0ad4c8b9f11260f9c5eeb8d3f2c0c1#diff-24027070357654805883c7fbd723a740R128)
When a Debugger.setScriptSource message is received we should iterate every cell and delete the code blocks of those JSFunction cells that are part of the changed file and are within the reported by the diff algorithm lines/columns. This would guarantee that the bytecode would be regenerated with the new content on next run. Those operations should be executed when the vm is idle, so this one should be fixed first https://github.com/NativeScript/ios-runtime/issues/521 (e.g. https://github.com/NativeScript/ios-runtime/commit/8b5973b04f0ad4c8b9f11260f9c5eeb8d3f2c0c1#diff-24027070357654805883c7fbd723a740R68)
We should also update every JSFunction cell SourceCode start line, start columnt, length etc with the values that are reported by the diff algorithm. The parser use this values so that it could determine where the function starts and ends. Currently those properties of SourceCode are read only, so we have to create setters.

The Proof Of Concept code could be found here

@NathanaelA you've also made a research on LiveEdit functionality for iOS (correct me if I am wrong). So it would be great if you share your opinion on the matter.

@KristinaKoeva you are way past my research.  All I've done is looking into the JSC code base to figure things out a simple method to do this -- yours is way cooler and better.  :grinning:
On number 3; your using JSC on iOS not v8.   I think you had a typo...
One part of the implementation you might want to change/add though is:

Ask JSC if file is loaded; if not loaded transfer then just new copy up and skip the rest.

Can the diff routine be in the debugger code on the client device; I know it wouldn't be as optimized; but the client actually has the "original" file -- at this moment unless you are transferring the original back from the device; the computer doesn't.
So if your basic steps are:

Upload the new file
Debugger takes new file and does checks and diff
does rest of steps you outlined
Then REPLACES old file with new file (because you know if it crashes for unrelated reasons and I restart the app on the device; I expect to be exactly where I was code wise on the device.

I love where you are going with this.

Take a look for the initial version here https://github.com/NativeScript/ios-runtime/pull/614
For more information follow this issue https://github.com/NativeScript/ios-runtime/issues/649


is there a way to disable the default cursor?

i want to repalce the cross cursor to a normal mouse cursor, is there a way to achieve that?

the default cursor is set by
this - https://github.com/rrag/react-stockcharts/blob/master/src/lib/ChartCanvas.js#L1058
function,
you could override those same styles with an !important or pass <ChartCanvas useCrossHairStyleCursor={false} ...


maximum element in array added

Have you read the Contributing Guidelines on Pull Requests?
(Yes)
Checklist

[x] I've read the contribution guidelines.
[x] I've checked the issue list before deciding what to submit.
[x] I've edited the README.md and link to my code.

Related Issues or Pull Requests
Fixes #357

please review @atarax665 @iamrajiv @siddharth25pandey sir

@shivammahajancse This PR is a Redundant PR, Work on Different Algorithm.


Some confusions on sdshdr5 & sdshdr8 on key/value actual memory usage and MEMORY USAGE command

I did some investigation on Redis source code 4.0 while I was doing my job. Something about sdshdr5 and sdshdr8 storage had raised my curiosity and something really confused me popped up. Here are the steps to reproduce the scenario:

Open redis cli
Type SET key value
So here from my perspective and observation, THE value has been stored with sdshdr8 and THE key has been stored with sdshdr5(through dbAdd->sdsdup)
I was guessing with the help of MEMORY USAGE, THE key should be analysed as sdshdr5, while THE key was passed via c->argv[2]->ptr, it will be first encapsulated with method createdEmbeddeStringObject which adopt sdshdr8 as the struct type. So it will be analysed as sdshdr8 rather than sdshdr5.
So, could anyone explain what is the actual reason for this and will this affect the accuracy of MEMORY USAGE command? Appreciated

Also, there is another question which might sound silly. Why we use 1<<5 (32) as one is the threahold for struct usage variation?


TGOS provider is invalid.

TGOS provider return API key error.
This provider works with stateful connection.

@virus-warnning Looks like they added an API key
TGOS HTTP Request Example
If you can provide me a URL that still works without an API key I can do it, but right now it's blocking the requests.

Re-open the issue when you find that URL

Thanks, I'll fork again.


Why am I getting overflows?

Hi! I don't know why I'm getting overflows training a NeuralNet. Actually I don't know if the problem is related to this project or to autograd. I think @mattjj could help, too.
With this sample data I'm getting overflows in exp and power, getting NaN as predictions. I can't understand where the problem is.
With Python3 you can execute this:
from io import StringIO

import pandas as pd

from mla.neuralnet import NeuralNet
from mla.neuralnet.layers import Activation, Dense
from mla.neuralnet.optimizers import Adam
from mla.neuralnet.parameters import Parameters

X_train_string = '-0.10410  6.106   6.232   97.50 -0.17490  0.16960 -0.4665  0.4242  289.4  723.8  2.6  2.6\n0.51340  5.517  12.160   90.83 -0.07771  0.07383 -0.2153  0.2133  361.6  706.9  2.6  2.6\n0.40280  5.359   9.403   79.45 -0.05685  0.06042 -0.2095  0.1873  331.9  676.0  2.6  2.6\n-0.02972  5.821   8.682   70.49 -0.05592  0.04690 -0.2113  0.2078  323.2  649.6  2.6  2.6\n-1.06600  6.893   3.798  132.00 -0.11060  0.10940 -0.3516  0.3494  245.4  800.6  2.6  2.6\n-0.66730  5.795   4.824   63.08 -0.04957  0.05160 -0.1446  0.1654  265.7  626.0  2.6  2.6\n0.11930  8.804  18.630  253.90 -0.17100  0.17270 -0.3759  0.3657  416.9  995.8  2.6  2.6'

y_train_string = '164.0\n156.5\n134.5\n142.5\n195.6\n134.2\n272.0'

X_test_string = '-0.1041  6.106   6.232  97.50 -0.17490  0.16960 -0.4665  0.4242  289.4  723.8  2.6  2.6\n0.5134  5.517  12.160  90.83 -0.07771  0.07383 -0.2153  0.2133  361.6  706.9  2.6  2.6\n0.4028  5.359   9.403  79.45 -0.05685  0.06042 -0.2095  0.1873  331.9  676.0  2.6  2.6'

y_test_string = '164.0\n156.5\n134.5'

X_train = pd.read_csv(StringIO(X_train_string), header=None, delim_whitespace=True)
y_train = pd.read_csv(StringIO(y_train_string), header=None, delim_whitespace=True)
X_test = pd.read_csv(StringIO(X_test_string), header=None, delim_whitespace=True)
y_test = pd.read_csv(StringIO(y_test_string), header=None, delim_whitespace=True)

net = NeuralNet(layers=[                    Dense(20, Parameters(init='normal')),
                                            Activation('sigmoid'),
                                            Dense(1),
                                        ],
                                        loss='mse',
                                        optimizer=Adam(),
                                        metric='mse',
                                        batch_size=256,
                                        max_epochs=200)

net.fit(X_train,y_train)

predictions = net.predict(X_test)

I'm getting this
/home/antonio/.conda/envs/intervalML/lib/python3.6/site-packages/autograd/core.py:81: RuntimeWarning: overflow encountered in exp
  result_value = self.fun(*argvals, **kwargs)
/home/antonio/.conda/envs/intervalML/lib/python3.6/site-packages/autograd/core.py:81: RuntimeWarning: overflow encountered in power
  result_value = self.fun(*argvals, **kwargs)
/home/antonio/.conda/envs/intervalML/lib/python3.6/site-packages/autograd/core.py:81: RuntimeWarning: invalid value encountered in multiply
  result_value = self.fun(*argvals, **kwargs)

So predictions is full NaN...
In [2]: predictions
Out[2]:
array([[ nan],
       [ nan],
       [ nan]])

Where's the problem?
Thank you! Your projects are really useful :)

Hi,  your weights are exploding.  Thus gradient descent algorithm moves towards very big values.
My recommendation:

Normalize your data

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

Change Activation('sigmoid') to Activation('linear')


feat: asset selection JIT market-data dispatch

Description
Follows-up on https://github.com/shapeshift/web/pull/7443 and actually implement swapper market-data fetching "JIT".
TODO:

[x] Find a way to be more lazy/debounce things here, we probably do not want to run this on every item visible in the list for users who scroll fast. Maybe RTK prefetch is the guy? https://redux-toolkit.js.org/rtk-query/usage/prefetching

Pull Request Type

[ ] :bug: Bug fix (Non-breaking Change: Fixes an issue)
[ ] :hammer_and_wrench: Chore (Non-breaking Change: Doc updates, pkg upgrades, typos, etc..)
[x] :nail_care: New Feature (Breaking/Non-breaking Change)

Issue (if applicable)

closes https://github.com/shapeshift/web/issues/7305

Risk

High Risk PRs Require 2 approvals

What protocols, transaction types or contract interactions might be affected by this PR?

Testing

Engineering

Operations

Screenshots (if applicable)

Closing as already handled gracefully by swapper:
https://github.com/shapeshift/web/blob/eaed57daeac52fdbcab5da460ba0a25408876a02/src/state/apis/swapper/swapperApi.ts#L63-L66


Added .npmignore

Ah, actually, will leave review-requirements for this one PR, then I can directly merge and point the releases to master. And afterwards we can then switch over and slowly turn this on.


Predict future satellite images using machine learning (replace optical flow)

See Jack's (pencil-and-paper) notebook for modelling ideas

#17

Researching the prediction of future satellite images won't be done in this repo.  Instead, researching prediction of future satellite images will be done in https://github.com/openclimatefix/satflow


scrape manager buffer reload channel

This prevents the scrape reloads from being skipped by the default
case in the select statement, and only takes that branch in the event
a reload is already pending in channel.

            select {
            case m.triggerReload <- struct{}{}:
            default:
            }

Is where the race condition happens, if the reloader() routine isn't ready to wait on triggerReload then the default branch is taken and a reload never happens.
This manifested itself in conprof coming up and never scraping from the static targets because the initial reload was ignored.

Oh wow, great find! lgtm :+1:


need ability to ban bot use based on identifier

Several users have requested the ability to ban members from the bot based on their identifier. This will probably require sending some part of the name seed to the client which can be used for a lookup. We will also want to ensure this prevents DMing.

Done with https://github.com/nminchow/Voltaire/commit/2c0d1fb70f007f959dadfda7b99dafa97a0807d0


Document data.json format for interface

Document the structure of the data.json or add a "How to add a new set of data" section to the README to make it easy to contribute new aligned audio + text data.

Currently data.json is of the form,

{
    'data': [
        {
            'key': 'text-used-to-refer-to-the-file',
            'name': 'name displayed in the corpus list',
            'audio_url': 'url that will be used as is (can still be relative',
            'word_alignment': 'path to the file containing word alignment',
            'sentence_alignment': 'path to the file containing sentence alignment, unused'
        }, ...

    ]
}

There's scope for adding corpus details such as name, description etc in the top-level, (currently the name is "deduced" in the flask file as the parent directory name of the data.json (which is a bit clumsy).
Adding new data is basically equivalent to adding a new directory at the top level (besides others such as ramayana, meghaduta, and a data.json file inside them.
I'll put this tentative information in a README later. It feels like the process can be definitely made smoother.

PR #4 handles this for the most part

Closed via PR #4


Can't bind to 'clearUnselected' since it isn't a known property of 'ng2-completer'.

Simple example as described in README works fine:
<ng2-completer [(ngModel)]="customerName" [datasource]="completerData" [minSearchLength]="0" [autoMatch]="true" (selected)="onSelected($event)"></ng2-completer>
But when I add 'clearUnselected' attribute, I'm getting template parse errors: Can't bind to 'clearUnselected' since it isn't a known property of 'ng2-completer'.
<ng2-completer [(ngModel)]="customerName" [datasource]="completerData" [minSearchLength]="0" [clearUnselected]="true" [autoMatch]="true" (selected)="onSelected($event)"></ng2-completer>
Similar Error when I add 'openOnClick' attribute.

I got the same issue
<ng2-completer [(ngModel)]="selectedCustomer" [datasource]="customerCompleter" (selected)="saveCustomer($event)" [minSearchLength]="0"
[textNoResults]="'No patient found'" [textSearching]="'Wait...'" [openOnFocus]="true"></ng2-completer>

With clearSelected instead of openOnFocus, it works well though, and like @hadarweiss it doesn't work with openOnClick either. Some of the attributes works and others won't.
I followed the installation except for the System.js configuration (I currently use webpack, maybe I'm missing something when binding ng2-completer ?)

I'm also seeing this error: Can't bind to 'openOnClick' since it isn't a known property of 'ng2-completer'. Did this work for you?

uninstall and reinstall ng2-completer

@dieyne had the same issue that disappeared by reinstalling.


Merge. They added Stable UnCLIP and updates to LoRA training that I needed

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


No Translations for Global Settings?

I've just tested this... No matter what language you've picked, descriptions for options in Global Settings would be always English. I checked the source code and...no one was ever working on this...

Yes actually its not there @LBYPatrick . But it can be added, but a lot of work. Will you be interested to write a translation for that? If you are interested then I can add the support infrastructure to help out :)

True It would be a lot of work...setting up a support infrastructure is beneficial for the future. I might be able to help out with Simplified Chinese after May 27th (My semester will end on that day).

@ziahamza I am ready for working on the translation work. Please start working on the framework.


vehicleWheel.get_translation() return wrong position

Godot version
3.6 custom build.
System information
Windows 11 on OpenGL ES 3
Issue description
I'm try to implement force feedback in Godot, the single difference between master branch and my FFB branch is input_default.h input_default.cpp, input.h, input.cpp, joypad_windows.h and joypad_windows.cpp to send data to joystick.
For track wheel direction collision I need to get wheel position, but all wheel return same position, the Rear Right Wheel's position.
You could try this downloading from my github.

Steps to reproduce
Build car, add 4 wheel in different positions, add this car in main scene.
Try to printt() position of the wheels.
Minimal reproduction project
If previous steps work property, try to download this project and click pause button.
Force Feedback Test Minimal.zip

Thanks for the report! Consolidating in https://github.com/godotengine/godot/issues/63657.

I'm try to implement force feedback in Godot, the single difference between master branch and my FFB branch is input_default.h input_default.cpp, input.h, input.cpp, joypad_windows.h and joypad_windows.cpp to send data to joystick.

If you are looking to upstream this work, consider joining the Godot Contributors Chat's #input channel :slightly_smiling_face:

Thanks for the report! Consolidating in #63657.

I'm try to implement force feedback in Godot, the single difference between master branch and my FFB branch is input_default.h input_default.cpp, input.h, input.cpp, joypad_windows.h and joypad_windows.cpp to send data to joystick.

If you are looking to upstream this work, consider joining the Godot Contributors Chat's #input channel 🙂

I need to solve too much things before Pull Request.


Control the expansion of individual nodes

Hello. I found this library for json data display, but i do not see a way how to calculate the initial expansion and collapse state of each individual node. Ideally, there could be a function that can be implemented and passed to the JsonView component called:
      shouldExpandNodeInitially={(
        keyPath: (string | number)[],      // denotes the keypath of the current node in the json structure
      ): boolean => {}

if the result is true - node should be expanded
if the result is false - node should be collapsed
This way the initial expansion could be calculated for each node separately and it would be possible to decide for each node if it should be opened or not. The current attribute 'collapsed' lets me control the collapsion state of the whole tree, which does not seem to allow to configure each node separately and limits my needs.
Any idea on how this issue could be solved?

To help understand the problem, here is the current code i am trying to implement:
`        <JsonView
id={rootNode}
key={treeKey}
value={tranformedTree}
style={customTheme}
keyName={rootNode}
// collapsed={
//   searchQuery || selectedDatapoints.length > 0 ? false : true
// }
collapsed={true}
enableClipboard={false}
displayObjectSize={false}
displayDataTypes={false}
onExpand={({ expand, value, keyid, keyName }) => {
console.log('expand', expand, value, keyid, keyName);
}}
>
<JsonView.Colon render={() => <></>} />
<JsonView.Quote render={() => <></>} />
<JsonView.BraceLeft render={() => <></>} />
<JsonView.BraceRight render={() => <></>} />
<JsonView.Ellipsis render={() => <></>} />
<JsonView.KeyName
render={(
{ children, ...props },
{ keyName, value, keys, parentValue },
) => {
// Define whether the current node is a leaf node
const isLeafNode = isValueALeafNode(value);
          // if (!keys) {
          //   return children;
          // }
          // const reversedKeys = [...keys].reverse();

          // console.log('reversedKeys', reversedKeys);
          // console.log('precomputedPaths', precomputedPaths);

          // const shouldBeExpanded = precomputedPaths.some(
          //   (precomputedPath) =>
          //     reversedKeys.every(
          //       (key, index) =>
          //         precomputedPath[
          //           precomputedPath.length - reversedKeys.length + index
          //         ] === key,
          //     ),
          // );

          // console.log(
          //   `keyName: ${keyName}, keys: ${JSON.stringify(
          //     keys,
          //   )}, isLeafNode: ${isLeafNode}, value: ${JSON.stringify(
          //     value,
          //   )}, expanded: ${shouldBeExpanded}`,
          // );
          // props['data-expanded'] = shouldBeExpanded;

          // Console log every incoming parameter
          // console.log('children', children);
          // console.log('props', props);
          // console.log('keyName', keyName);
          // console.log('keys', keys);
          // console.log('value', value);
          // console.log('parentValue', parentValue);
          // console.log('');

          if (!isLeafNode || !keyName || !keys) {
            return children;
          }

          const selection = keypathToMPIDDPID(
            [...(keys || [])].reverse(),
            tranformedTree,
          );

          if (!selection) {
            return children;
          }

          const assets = findAssetMemberAndMeasuredValue(
            monitoringSiteDetails,
            selection,
          );
          const isSelected = isDatapointSelected(selection);
          const isAllowedToSelectForThisDatapoint = isAllowedToSelect(
            selection,
            selectedDatapoints,
          );

          const displayValue = `${keyName as string} (${
            assets.measuredValue?.uomCode || '1'
          })`;

          return (
            <span
              onClick={() =>
                isAllowedToSelectForThisDatapoint &&
                handleValueClick(selection.dpid, selection.mpid)
              }
              style={{
                cursor: isAllowedToSelectForThisDatapoint
                  ? 'pointer'
                  : 'not-allowed',
                color: isSelected
                  ? 'var(--sdx-color-int-blue)'
                  : isAllowedToSelectForThisDatapoint
                    ? 'var(--sdx-color-int-blue)'
                    : 'gray',
                fontWeight: isSelected ? 'bold' : 'normal',
              }}
            >
              {highlightMatch(displayValue, searchQuery || '')}
            </span>
          );
        }}
      />
      <JsonView.Arrow
        // eslint-disable-next-line @typescript-eslint/no-explicit-any
        render={(props: any) => {
          const isExpanded = props['data-expanded'];
          return (
            <img
              style={{
                cursor: 'pointer',
                width: '0.7rem',
                marginRight: 10,
                rotate: isExpanded ? '0deg' : '270deg',
              }}
              src="/icons/forms_selector.svg"
            />
          );
        }}
      />
      <JsonView.ValueQuote render={() => <></>} />
    </JsonView>`

@TomsBicans  My understanding is that you might need an onExpand event?
https://github.com/uiwjs/react-json-view/blob/5a3a4497543fc8b6572f6b459dbe6babd4dfa251/core/src/index.tsx#L65

Hello. It is not exactly what i mean. The onExpand event is triggered through user interaction, which is not exactly what i am talking about. I can try to describe the problem i am facing.
I have a problem, that i need to open up the json tree in a specific way for the initial state - some nodes of the json tree should be opened, some nodes should be closed. Currently i see no way on how to control this mechanism without user interaction.
Another library i found has this kind of function i am talking about, but it lacks in other areas unrelated to the node expansion - it lets the developer implement the ShouldExpandNodeInitially function and the function is called automatically on the initial render for each node, so it is kind of easy to control the initial open/closed state for each individual node of the json tree. Have a look here: https://github.com/reduxjs/redux-devtools/blob/3c39eb49f2e5cb5ee18dcf8bc0cb9d66ec75573b/packages/react-json-tree/src/types.ts#L29
Maybe something like this can be implemented also in this project.
Thank you for your time.

@TomsBicans upgrade v2.0.0-alpha.29
https://github.com/uiwjs/react-json-view/blob/044ff958ed04e50fddf9424fa0dddf3150091ef9/core/README.md?plain=1#L734-L742


Clean-up and wrap-up

Last PR before I leave for the summer. It's huge because I'm trying to leave things in as good a state as possible. The main work done here is to put in place all the proof structure for mm_map_level and mm_map_root, with just a few admits in those proofs. In mm_map_root, I left detailed comments explaining how to solve each admit. I've also fixed a bunch of TODOs by moving things around and deleting unused lemmas.

LGTM!


Cut individual mask and save separately

I followed the steps in the tutorial to get the masks of an image:
https://blog.roboflow.com/how-to-use-segment-anything-model-sam/
Now that I have generated the masks, how can I save the individual masks as separate images?

I think you can try the SuperVision .fromSam() function. Also you can refer to the Roboflow notebook on SAM.


Skinning

I'd like some basic skinning feature, basically change colors of buttons, steps, etc..

Hi @Trapulo,
The stylesheet for Hopscotch is compiled using LESS. For styling tweaks, I'd recommend pulling this repository, making style changes locally to fit your website, then using Grunt to rebuild Hopscotch. Compiled files should output to the /tmp directory... from there, you can copy the final CSS file to another directory.
I hope this helps!


Ignore/support _comments and _ignore

According to https://github.com/Kong/kong/blob/master/kong/templates/kong_yml.lua _ignore and _comment fields are allowed .
However when validating, deck does not allow this:
$ cat kong.yaml
_format_version: "1.1"
services:
- name: name
  _comment: something
  host: localhost
  protocol: https
$ deck validate -s kong.yaml
Error: reading file: validating file content: 1 errors occurred:
        services.0: Additional property _comment is not allowed

the declarative format for DBless and the deck format are similar, but not the same.
Here's what I've jotted down a while ago, though new changes have been added in the mean time.


[otp_ctrl] Further code and Hjson cleanup

Please only consider the last commit in this PR - all others will be rebased away once #7765 has been merged.
It cleans up a few things in RTL (corrected IRQ register labels, removed redundant OR operations), and it adds reset values to registers that did not specify any reset value before.
@cindychip I reran all OTP tests to make sure this does not brake anything.

This commit has been merged as part of #7911 - closing down.


Know any algorithm implementation in any language?

Hi, everyone!

Do you know any algorithm implementation in any language, that you wanna share with us?

Open a pull request in any of the programming languages available, or recommend another language you want to implement algorithms!

I know bubble sort in java, would you like me to implement that somewhere?

Ah, my apologies, the link to the file didn't work so I assumed it wasn't done.
How about binary search?

It doesn't look like it's implemented yet, I'll do it in java.

I have added one coin change problem solution that uses recursive pattern.
https://github.com/AllAlgorithms/algorithms/pull/9

Hi - I want to write for Magic Sqaure and Pascal's triangle in Python. Pl let me know

@bhuvanakundumani that would be great, check out https://github.com/allalgorithms/python

PR done. Sorry . Am a newbie and hence the trouble.
On Wed, Oct 24, 2018 at 2:56 PM Bhuvana Kundumani <
<EMAIL_ADDRESS>wrote:

Hi ,
I am unable to PR . have sent the link of the github repo
https://github.com/bhuvanakundumani/python-1.git
let me know the next steps.
Thanks
On Tue, Oct 23, 2018 at 9:56 PM Abraham Hernandez <
<EMAIL_ADDRESS>wrote:

@bhuvanakundumani https://github.com/bhuvanakundumani that would be
great, check out https://github.com/allalgorithms/python
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/AllAlgorithms/algorithms/issues/4#issuecomment-432317256,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADi8Q1A0JlU_fa3YNErL15pQOpe_1osjks5un0NEgaJpZM4XDDUd
.

--
Thank You,
Bhuvana K A

--
Thank You,
Bhuvana K A