aws-s3-eventbridge

Similar to receiving S3 notifications on SQS, SNS, or Step functions, S3 event notifications can also be sent to Amazon Event Bridge. Event Bridge provides some advantages over SQS and SNS as detailed in this blog post https://aws.amazon.com/blogs/compute/using-dynamic-amazon-s3-event-handling-with-amazon-eventbridge/
Use Case

I am plan to use this construct in my solution to build S3 PutObject event notification.
Proposed Solution

The solution will be very similar to what is proposed in the blog post https://aws.amazon.com/blogs/compute/using-dynamic-amazon-s3-event-handling-with-amazon-eventbridge/ using AWS CloudTrail
Other

[x] :wave: I may be able to implement this feature request
[ ] :warning: This feature might incur a breaking change

This is a :rocket: Feature Request

We like this idea - we actually already have most of it in place. aws-s3-stepfunctions goes thru EventBridge, so we can create this just by extracting the s3 to eventbridge portion of that construct. We've added that task to our roadmap (no date promised :-)

After looking deeper into this, an aws-s3-eventbridge construct targeting the default event bus wouldn't do anything, since those events are already sent to the default bus. If we allow a custom event bus, then it adds value - but the value is all within EventBridge. It seems to us a better construct to write would be aws-eventbridge-eventbridge, that allows customers to move event from the default bus to a custom bus. This would enable sending S3 events to a custom bus, but also provide the same functionality for all the AWS services that send events to the default bus. The alternative would be createing aws-sqs-eventbridge, aws-ec2-eventbridge, etc.
What we lose in doing it this way rather than with aws-s3-eventbridge is the option to automatically create a CloudTrail. But setting up CloudTrail in every account is a significant best practice - hopefully every account has set this up. And if an account does not have it set up, perhaps not including it in this construct will encourage the account owner to do so in a way less transient than as part of a dynamically created and destroyed construct.

This discussion is impacted by this announcement. CloudTrail is no longer needed for S3 events. An aws-s3-eventbridge construct at this point would only allow clients to send S3 events to a custom event bus.

What this construct wants to do is being done by enabling EventBridge for the S3 bucket, so as described this functionality is now available without this implementation. This opens up a discussion of how to link new services to be triggered by EventBridge events from S3 using constructs, and also how to get S3 notification events to an custom Event Bus.
Leaving this open for discussion and further thought.

Enabling eventbridge is a property within the S3 bucket. Its only the rules and filtering that needs to be defined. Note that, this only allows sending events to the default bus. (Any custom bus targets would still require additional rules configuration and defining targets).

We're thinking about an aws-eventbridge-eventbridge construct to take and event source feeding the default bus and forward them to a custom bus. It would take an event source, such as an S3 bucket, as a prop. But event sources are tricky, lambda has it's own CDK library for responding to various events.
We've also opened an issue on the CDK repo to ad enabling eventbridge to the L2 S3 object (our recent release does it by directing setting the CloudFormation on the underlying L1 construct)


feat(df-repr/bridge): upgrade datafusion to 43.0.0

Despite the upgrade,

New create_df_context to be used across all crates to create a datafusion context with optd. We had too much duplicate code before to set up the context.
The main refactor is about the aggregation expressions. Datafusion has a new way of doing that.
Datafusion removed cross join. We didn't. We can eventually remove it but now it's blocked on two-stage cascades: if we simply treat cross join the same as inner join, we would time out.
Several other refactors to adapt to datafusion (i.e., limit node now takes i64, empty relation / placeholder row executor)
Keep as much as the original datafusion cli crate as possible. We now only patch main.rs and exec.rs.
There's one more breaking change that we might encounter later when doing sort physical properties. Now datafusion logical plan will remove duplicate sorts if there are no limits present. I feel this is a bad move b/c it's not a direct mapping from the original SQL statement...

Waiting to review until after sqllogictest failures are fixed


Client/server code for MinIO testing

[x] add a function for minio testing (server)
[x] add a Go client for invoking that function
[x] add scripts that test the corresponding workflow using Docker containers and in Knative
[x] include the workflow into the CRI tests

The function is a gRPC server that saves a payload to the MinIO server that is deployed in vHive, then immediately loads it from MinIO.
Signed-off-by: Dmitrii Ustiugov<EMAIL_ADDRESS>

@amariucaitheodor this test works but I'd like to add this test to the CI before merging

@shyamjesal please don't forget to delete the branches that are merged


set excludedocs to zypper configuration

so that the image does not get the docs when running zypper update.

That looks good, however I just realized we ship tumbleweed's root FS uncompressed. That does not work on OBS. Can you create a root.tar.bz2 also for tumbleweed?

sure

should I update the version number in the kiwi file?

Yes please, can you also take care of propagating the changes to the Docker
hub?
On 16 Jul 2015 15:58, "Jordi Massaguer Pla"<EMAIL_ADDRESS>wrote:

should I update the version number in the kiwi file?
—
Reply to this email directly or view it on GitHub
https://github.com/openSUSE/docker-containers/pull/10#issuecomment-121964724
.


Transaction pool wastes CPU

I've noticed that when doing full sync, all transaction from all imported blocks are re-validated by the transaction pool after the block has been imported. This happens here:
https://github.com/paritytech/substrate/blob/9772209ee962fe36c3502afb638e1bd835c23d9e/client/transaction-pool/src/graph/pool.rs#L277
This wastes a lot of resources and hurts sync performance. Transaction pool should only re-validate transactions that are actually in the pool, shouldn't it?. Or at least skip the whole thing when the pool is empty.

Transaction pool should not be called while doing a full/major sync as we don't send block import notifications?

It looks like finality events are still being sent when GRANDPA imports a justification during major sync. Transaction pool then computes a tree route to the block from the previous notification and processes all blocks.

Ahh! Ty for the report.
@michalkucharczyk could you please fix this? :D

@arkpar looking at your https://github.com/paritytech/substrate/pull/12973#issuecomment-1357859847, does the above proposal (https://github.com/paritytech/substrate/issues/12903#issuecomment-1357594101) make sense to you?

Without fixing the tree route issue, it will only make things worse. Now, at least, you get at most few hundred blocks between notifications because of how grandpa sends them when finalizing. If you stop processing them at all when syncing, as I've said in the mentioned comment, you can potentially receive notification for block that's a million blocks later than the the previous notification.

The first notification after major_syncing completion would need to be force_update which sets the finalized block w/o computing tree_route:
https://github.com/paritytech/substrate/blob/master/client/transaction-pool/src/enactment_state.rs#L140

The first notification after major_syncing completion would need to be force_update which sets the finalized block w/o computing tree_route:

This could work.
@bkchr what do you think?

If you stop processing them at all when syncing, as I've said in the mentioned comment, you can potentially receive notification for block that's a million blocks later than the the previous notification.

I think it would be much better for now to not build a tree route or completely ignore the update when the difference between the last updated block and the current event block is greater than some constant. I think we could pick something like 20 for this constant.
I would really not like to introduce some new kind of event. I'm looking forward to the future where the tx pool is just not running while we are not in "keep up sync mode".

Why did you suggest 20? Is it based on some stats?

"Educated guess" :laughing:
Currently we are switching to major sync mode when we are 5 blocks behind the tip of the chain. So, 20 sounds reasonable. We could maybe lower it.


Why does Hourglass require Internet Explorer to be installed?

I read through the details on the Hourglass page and am wondering why Hourglass requires Internet Explorer to be installed.
Also, is there a possibility to make the program truly portable by removing the requirement of Microsoft .NET Framework 4.0 Client Profile?
Lastly, what are the advantages of installing Hourglass over using the portable version? All I know is that installed programs typically create registries to store settings while portable versions tend to save all their settings in a text file--not sure if this is the only difference between the two versions and if registries are somehow preferable to text files.
Otherwise, great job on the application.

Hourglass does not have any system requirements above and beyond what is required for Microsoft .NET Framework 4.0 Client Profile.

Internet Explorer is listed as a system requirement for Hourglass only because Microsoft lists it as a system requirement for Microsoft .NET Framework 4.0 Client Profile.

The Hourglass user interface uses Windows Presentation Foundation, and all of the code is written in C#. Removing the Microsoft .NET Framework 4.0 Client Profile would require rewriting the application in native code. I don't currently have plans to do that -- it would be quite a bit of work.

Installing Hourglass places a shortcut in your Start Menu, puts Hourglass.exe somewhere sensible (typically "C:\Program Files (x86)"), and stores your settings under in your user profile (typically "C:\Users<user>\AppData\Local").

If you use the portable version, you have to put Hourglass.exe somewhere sensible yourself, and launch the app by double-clicking the Hourglass.exe or manually create a shortcut somewhere. With the portable version, settings are stored in a file that lives next to the .exe file.


If link or service enablement did not change, should not result be "No Changes" for an operation rather than "Success?"

Describe the bug
Maybe this the chroot connection, not sure. But facts do work...
Anyways, If I use openrc to enable a service in a chroot install, it works (Success).
However, if I rerun I don't get "No Changes" I get "Success".
So the same it enables or does not already have to enable due already been enabled.
I tried a symlink instead of openrc service enable, as i the end the result is supposed to be the same.
Same behavior, upon making the symlink, "Success". But when rerunning, "Success" (not "No Changes").
I'm use to ansible for these two things. (service enabling and symlinks).
I can collect facts and check them, and only make symlink if it is needed. However, if I have to do this, how is it any better than just running low level "automation" scripts.
This is on an initial run (no symlink exists)
--> Loading config...
--> Loading inventory...
    The @chroot connector is in beta!

--> Connecting to hosts...
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Connected

--> Preparing operations...
    Loading: ./initial_prep.py
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Ready: ./initial_prep.py

--> Proposed changes:
    Groups: @chroot
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs]   Operations: 22   Commands: 26

--> Beginning operation run...
--> Starting operation: Apk/Update
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Apk/Upgrade (available=True)
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Default packages
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add devfs to runlevel sysinit
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add dmesg to runlevel sysinit
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add hwdrivers to runlevel sysinit
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add mdev to runlevel sysinit
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add bootmisc to runlevel boot
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add hostname to runlevel boot
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add hwclock to runlevel boot
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add modules to runlevel boot
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add networking to runlevel boot
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add swap to runlevel boot
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add sysctl to runlevel boot
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add syslog to runlevel boot
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add urandom to runlevel boot
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add acpid to runlevel default
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add crond to runlevel default
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add dropbear to runlevel default
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add killprocs to runlevel shutdown
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add mount-ro to runlevel shutdown
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Add savecache to runlevel shutdown
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Results:
    Groups: @chroot
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs]   Successful: 22   Errors: 0   Commands: 26/26

If I run it again with no check, then each symlink operation gets still gets "Success" (and I don't know if it had to create the symlink, or if it already existed)
This is on a rerun (symlink already exists) if I check before making the symlink:
--> Loading config...
--> Loading inventory...
    The @chroot connector is in beta!

--> Connecting to hosts...
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Connected

--> Preparing operations...
    Loading: ./initial_prep.py
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Ready: ./initial_prep.py

--> Proposed changes:
    Groups: @chroot
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs]   Operations: 3   Commands: 2

--> Beginning operation run...
--> Starting operation: Apk/Update
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Apk/Upgrade (available=True)
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] Success

--> Starting operation: Default packages
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs] No changes

--> Results:
    Groups: @chroot
    [@chroot<redacted>alpine/alpine-miniroot-extend/rootfs]   Successful: 3   Errors: 0   Commands: 2/2

@chroot<redacted>alpine/alpine-miniroot-extend/rootfs
	service devfs already in runlevel sysinit
	service dmesg already in runlevel sysinit
	service hwdrivers already in runlevel sysinit
	service mdev already in runlevel sysinit
	service bootmisc already in runlevel boot
	service hostname already in runlevel boot
	service hwclock already in runlevel boot
	service modules already in runlevel boot
	service networking already in runlevel boot
	service swap already in runlevel boot
	service sysctl already in runlevel boot
	service syslog already in runlevel boot
	service urandom already in runlevel boot
	service acpid already in runlevel default
	service crond already in runlevel default
	service dropbear already in runlevel default
	service killprocs already in runlevel shutdown
	service mount-ro already in runlevel shutdown
	service savecache already in runlevel shutdown

To Reproduce
Here is the python deployment script with the fact checking logic in it.
# vim: set expandtab tabstop=4 shiftwidth=4 filetype=python3 :

import click
from pyinfra import host
from pyinfra.operations import apk, files
from pyinfra.facts.files import FindLinks, Link
from pyinfra.api import BaseStateCallback

BASE_PACKAGES="mlocate mkinitfs openrc busybox-initscripts rsync-openrc dropbear rsync micro htop"

SERVICES = {
    "sysinit": "devfs dmesg hwdrivers mdev",
    "boot": "bootmisc hostname hwclock modules networking swap sysctl syslog urandom",
    "default": "acpid crond dropbear",
    "shutdown": "killprocs mount-ro savecache",
}

class Session(BaseStateCallback):
    def __init__(self):
        self.have = [host.name]
    def host_disconnect(self, state, host):
        if len(self.have) > 1:
            click.echo(click.style(f'\n{self.have[0]}\n', fg='white') + click.style('\n'.join(self.have[1:]), fg='blue'))
    def append(self, message):
        self.have.append(message)

def main():
    session = Session()
    host.state.add_callback_handler(session)

    apk.update()
    apk.upgrade(available=True)

    apk.packages(name='Default packages', packages=BASE_PACKAGES.split())

    for runlevel in SERVICES.keys():
        links = [x.strip() for x in host.get_fact(FindLinks, f'/etc/runlevels/{runlevel}', quote_path=False)]
        for service in SERVICES[runlevel].split():
            path, target = f'/etc/runlevels/{runlevel}/{service}', f'/etc/init.d/{service}'
            if (path in links) and (host.get_fact(Link, path)["link_target"].strip().removesuffix("'") == target):
                session.append(f'\tservice {service} already in runlevel {runlevel}')
            else:
                files.link(name = f'Add {service} to runlevel {runlevel}', path = path, target = target, force=True)

main()

Expected behavior
I should not have to put in a check for an operation.
If I enable a service and rerun, it should be "no changes" if no steps are needed to get result.
If I create a symlink and rerun, it should be "no changes" if no steps are needed to get result.
Is this valid expected behavior? Am I doing something wrong so I'm not getting "no changes" where applicable?
Meta

Include output of pyinfra --support.
--> Support information:
If you are having issues with pyinfra or wish to make feature requests, please
check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues .
When adding an issue, be sure to include the following:
System: Linux
Platform: Linux-5.15.11-arch2-1-x86_64-with-glibc2.33
Release: 5.15.11-arch2-1
Machine: x86_64
pyinfra: v1.5
Executable: ./pyvenv/bin/pyinfra
Python: 3.10.1 (CPython, GCC 11.1.0)

How was pyinfra installed (source/pip)?
Pip in a virtual env

Comments
I like what I see so far of pyinfra, but... It needs a lot of work to replace my use of ansible with it.
I like it no python is needed on the target hosts.
I like it that the "playbooks" are python source rather than a DSL in yaml..
However, unless I'm doing something wrong in regard to "no changes" vs "success" it is going to be a lot of work to bring pyinfra up to par. (correcting behavior for openrc service enablement and file symlink creation?)

Here is a more exhaustive/comprehensive repro steps.
Needs to be run on a systemd Linux distro with host python/pip/venv installed.
User needs sudo.
Put these 4 files in your directory and type make basic-prep after renaming 3 of the files.
requirements.txt
Makefile.txt
initial_prep.py.txt
chroot.py.txt
Rename 3 of the files
mv Makefile.txt Makefile
mv chroot.py.txt chroot.py
mv initial_prep.py.txt initial_prep.py
make basic-prep

Here is a more exhaustive/comprehensive repro steps.
Needs to be run on a systemd Linux distro with host python/pip/venv installed.
User needs sudo.
Put these 4 files in your directory and type make basic-prep after renaming 3 of the files.
requirements.txt
Makefile.txt
initial_prep.py.txt
chroot.py.txt
Rename 3 of the files
mv Makefile.txt Makefile
mv chroot.py.txt chroot.py
mv initial_prep.py.txt initial_prep.py
make basic-prep

From initial_prep.py above:
    for runlevel in SERVICES.keys():
        links = [x.strip() for x in host.get_fact(FindLinks, f'/etc/runlevels/{runlevel}', quote_path=False)]
        for service in SERVICES[runlevel].split():
            path, target = f'/etc/runlevels/{runlevel}/{service}', f'/etc/init.d/{service}'
            if (path in links) and (host.get_fact(Link, path)["link_target"].strip().removesuffix("'") == target):
                session.append(f'\tservice {service} already in runlevel {runlevel}')
            else:
                files.link(name = f'Add {service} to runlevel {runlevel}', path = path, target = target, force=True)

Notice that I had to clean up the link_target fact for the Link path (there was and extraneous trailing single quote.)
I've not dug into the files.link code, it is possible if it is trying to check, that the check is failing. (As my initial check did as well, until I looked at what was being returned and cleaned it up).

From initial_prep.py above:
    for runlevel in SERVICES.keys():
        links = [x.strip() for x in host.get_fact(FindLinks, f'/etc/runlevels/{runlevel}', quote_path=False)]
        for service in SERVICES[runlevel].split():
            path, target = f'/etc/runlevels/{runlevel}/{service}', f'/etc/init.d/{service}'
            if (path in links) and (host.get_fact(Link, path)["link_target"].strip().removesuffix("'") == target):
                session.append(f'\tservice {service} already in runlevel {runlevel}')
            else:
                files.link(name = f'Add {service} to runlevel {runlevel}', path = path, target = target, force=True)

Notice that I had to clean up the link_target fact for the Link path (there was and extraneous trailing single quote.)
I've not dug into the files.link code, it is possible if it is trying to check, that the check is failing. (As my initial check did as well, until I looked at what was being returned and cleaned it up).

The equivalent in Ansible, while the output is cleaner, required me to removed the soft link of sh to busybox and make it a hard link.
Also, I had to run ansible via sudo to get to work with the chroot.
With pyinfra because I was able to use a chroot wrapper, so I did not need to run pyinfra via sudo.
Both Ansible and Pyinfra took about the same time. As I initially stated though, Ansible was more "correct" in determining what needed to be done.
That being said, Pyinfra shows a lot of promise and is by far more flexible.

The equivalent in Ansible, while the output is cleaner, required me to removed the soft link of sh to busybox and make it a hard link.
Also, I had to run ansible via sudo to get to work with the chroot.
With pyinfra because I was able to use a chroot wrapper, so I did not need to run pyinfra via sudo.
Both Ansible and Pyinfra took about the same time. As I initially stated though, Ansible was more "correct" in determining what needed to be done.
That being said, Pyinfra shows a lot of promise and is by far more flexible.

See #732 for discussion of the quoting issue that led the path link fact not being correct.

For #729 @Fizzadar wrote:

@bogen85 Re: systemd-nspawn - are you using the @chroot connector at the moment? There's two approaches here:
* A new connector specific for `systemd-nspawn`

* Add a configuration field `host.data.X` to change the behaviour of the `@chroot` connector (this is probably nicer?)

Regarding systemd-nspawn, yes, I'm using the @chroot connector along with a chroot wrapper script in my path. (Details already discussed in this issue). I'm open to either (new connector or new configuration field) as long as I have the some functionality I'm getting from the wrapper (systemd-nspawn run with sudo, not all of pyinfra) (with ansible and chroot, all of ansible has to be run as root).

I think this is fixed? At least the init.d operation. Please re-open if not!


[SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots

What changes were proposed in this pull request?
Improved warning message in Barrier Execution Mode when required slots > maximum slots.
The new message contains information about required slots, maximum slots and how many times retry failed.
Why are the changes needed?
Providing to users with the number of required slots, maximum slots and how many times retry failed might help users to decide what they should do.
For example, continuing to wait for retry succeeded or killing jobs.
Does this PR introduce any user-facing change?
Yes.
If spark.scheduler.barrier.maxConcurrentTaskCheck.maxFailures=3, we get following warning message.
Before applying this change:
19/08/18 15:18:09 WARN DAGScheduler: The job 2 requires to run a barrier stage that requires more slots than the total number of slots in the cluster currently.
19/08/18 15:18:24 WARN DAGScheduler: The job 2 requires to run a barrier stage that requires more slots than the total number of slots in the cluster currently.
19/08/18 15:18:39 WARN DAGScheduler: The job 2 requires to run a barrier stage that requires more slots than the total number of slots in the cluster currently.
19/08/18 15:18:54 WARN DAGScheduler: The job 2 requires to run a barrier stage that requires more slots than the total number of slots in the cluster currently.
org.apache.spark.scheduler.BarrierJobSlotsNumberCheckFailed: [SPARK-24819]: Barrier execution mode does not allow run a barrier stage that requires more slots than the total number of slots in the cluster currently. Please init a new cluster with more CPU cores or repartition the input RDD(s) to reduce the number of slots required to run this barrier stage.
  at org.apache.spark.scheduler.DAGScheduler.checkBarrierStageWithNumSlots(DAGScheduler.scala:439)
  at org.apache.spark.scheduler.DAGScheduler.createResultStage(DAGScheduler.scala:453)
  at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:983)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2140)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2132)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2121)
  at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
  at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:749)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2080)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2120)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2145)
  at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:961)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  at org.apache.spark.rdd.RDD.withScope(RDD.scala:366)
  at org.apache.spark.rdd.RDD.collect(RDD.scala:960)
  ... 47 elided

After applying this change:
19/08/18 16:52:23 WARN DAGScheduler: The job 0 requires to run a barrier stage that requires 3 slots than the total number of slots(2) in the cluster currently.
19/08/18 16:52:38 WARN DAGScheduler: The job 0 requires to run a barrier stage that requires 3 slots than the total number of slots(2) in the cluster currently (Retry 1/3 failed).
19/08/18 16:52:53 WARN DAGScheduler: The job 0 requires to run a barrier stage that requires 3 slots than the total number of slots(2) in the cluster currently (Retry 2/3 failed).
19/08/18 16:53:08 WARN DAGScheduler: The job 0 requires to run a barrier stage that requires 3 slots than the total number of slots(2) in the cluster currently (Retry 3/3 failed).
org.apache.spark.scheduler.BarrierJobSlotsNumberCheckFailed: [SPARK-24819]: Barrier execution mode does not allow run a barrier stage that requires more slots than the total number of slots in the cluster currently. Please init a new cluster with more CPU cores or repartition the input RDD(s) to reduce the number of slots required to run this barrier stage.
  at org.apache.spark.scheduler.DAGScheduler.checkBarrierStageWithNumSlots(DAGScheduler.scala:439)
  at org.apache.spark.scheduler.DAGScheduler.createResultStage(DAGScheduler.scala:453)
  at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:983)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2140)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2132)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2121)
  at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
  at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:749)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2080)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2120)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2145)
  at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:961)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  at org.apache.spark.rdd.RDD.withScope(RDD.scala:366)
  at org.apache.spark.rdd.RDD.collect(RDD.scala:960)
  ... 47 elided

How was this patch tested?
I tested manually using Spark Shell with following configuration and script. And then, checked log message.
$ bin/spark-shell --master local[2] --conf spark.scheduler.barrier.maxConcurrentTasksCheck.maxFailures=3
scala> sc.parallelize(1 to 100, sc.defaultParallelism+1).barrier.mapPartitions(identity(_)).collect

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14364/
Test PASSed.

Test build #109298 has started for PR 25487 at commit bc5042b.

Test build #109298 has finished for PR 25487 at commit bc5042b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109298/
Test PASSed.

cc: @jiangxb1987

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14563/
Test PASSed.

Test build #109507 has started for PR 25487 at commit 1207e00.

Test build #109507 has finished for PR 25487 at commit 1207e00.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109507/
Test PASSed.

Test build #109586 has started for PR 25487 at commit 1967e9c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14641/
Test PASSed.

Test build #109586 has finished for PR 25487 at commit 1967e9c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109586/
Test PASSed.

Merged to master


pkgs: added ArangoDB 2.5.2

Is it possible to share the v8 with the one built in nixpkgs? This would be yet another bundled v8 otherwise.

Is it possible to share the v8 with the one built in nixpkgs? This would be yet another bundled v8 otherwise.

that's what I tried to do: https://github.com/arangodb/arangodb/issues/1316

Thanks for your effort to convince the upstream. I see it's rather hardcoded in the configure... so let's add this for now though I'm unhappy yet another time. Built locally, looks fine thanks.

@flosse any plan to add a package for ArangoDB 3.x?


How to restore dynamically added queues

How to restore queues with data from redis after the application restart, which have been added dynamically Verk.add_queue(:new, 10)
At the initial startup, the supervisor only reads the data from the config
https://github.com/edgurgel/verk/blob/062811eeb706133bd5d55e85cd36c017a7ee2da7/lib/verk/supervisor.ex#L19

Verk does not saves the queues that were started dynamically. The initial configuration should have them or when Verk starts, add the queues back again dynamically.
One common use case is to start a supervision tree with the Verk.Supervisor then a process that adds the dynamic queues when it starts. So if the supervision tree restarts it will guarantee that the queues were added. I can add more information and examples around this if needed.

thanks I got it)

You could for example have a supervisor that has the following supervision tree:
tree = [supervisor(Verk.Supervisor, []), worker(QueuesLoader, [])]
opts = [name: MySupervisor, strategy: :rest_for_one]
Supervisor.start_link(tree, opts)

The QueuesLoader can be a GenServer that looks like this:
defmodule QueuesLoader do
  use GenServer
  require Logger

  @doc false
  def start_link do
    GenServer.start_link(__MODULE__, [], name: __MODULE__)
  end

  @doc false
  def init(_args) do
    for queues <- MyQueues.all do
      Verk.add_queue(...)
    end
    { :ok, nil }
  end
end

Notice that this supervisor is a rest_for_one so that if Verk.Supervisor crashes, it will restart the QueuesLoader worker as it's defined after the Verk.Supervisor.


Sync eng/common directory with azure-sdk-tools for PR 2380

Sync eng/common directory with azure-sdk-tools for PR https://github.com/Azure/azure-sdk-tools/pull/2380  See eng/common workflow

/check-enforcer override


Tidy up 'tap to replay'

Looks unpolished. @andraskangyal any suggestions?

"Replay" is enough I think.

cancelled


fix grammar and section numbering in Terms of Service

Fixed section numbering from "26" to "27" for "Dispute Resolution..." section
There was a duplicate section number 26
This ensures correct sequential numbering of sections

Updated grammar in indemnification clause from "wishes" to "will wish"
Improved future tense consistency in legal text
Better aligns with the conditional nature of the statement

Fixed typo from "verifable" to "verifiable"
Corrected spelling error
Ensures technical accuracy

Standardized terminology from "datafeeds" to "data feeds"
Maintains consistent terminology throughout documentation
Matches common industry usage

🟡 Heimdall Review Status

Requirement
Status
More Info

Reviews

🟡
0/1

Denominator calculation

Show calculation

1 if user is bot
0

1 if user is external
0

From .codeflow.yml

1

Additional review requirements

Show calculation

Max
0

0

From CODEOWNERS

0

Global minimum
0

Max

1

1

1 if commit is unverified
0

Sum
1


[BUG] Fixed issue with index out-of-range on empty search

Issue
When the search is supplied an empty string, the table still allows the user to try and access the manga page. This leads to an index out-of-range issue as mangaList.Results is empty.
Fix
Add a simple check before displaying the manga page to ensure mangaList.Results is not empty.

ill review this in time


Add Missing Channel Types

[x] I have checked for similar issues.
[x] I have updated to the [latest JDA version][download].
[x] I have checked the branches or the maintainers' PRs for upcoming features.

Feature Request
JDA is missing the news, and thread channel types.
Discord Documentation
ChannelType.java

[ ] Add GUILD_NEWS channel type
[ ] Add GUILD_NEWS_THREAD channel type
[ ] Add GUILD_PUBLIC_THREAD channel type
[ ] Add GUILD_PRIVATE_THREAD channel type

those channel types will be added with jda5 which includes thread support

Also, News channels do exist in JDA, they're just a variant of a TextChannel at the moment

Oh right. Im used to GitLab where the "cards" in projects are actual issues, so I didn't see it was getting worked on.

@MinnDevelopment I suggest the feature request template has a checkbox for checking the Projects to see if the feature is already planned.


Installation Wizard - Add Nodes

As a node operator, I want to be able to add nodes that I want to monitor.
Requirements

A "home page" for Nodes setup must be created and be displayed at /installation-wizard/nodes.
The page must have two central elements:

A button with the label "Add Node", allowing the node operator to click and open the form. The form will be shown in a modal (no redirection to a different page).
A svc-data-table showing the nodes that have been added so far or a message telling the node operator "Hey, you need to add your first node..." if no nodes have been added yet. Once the data table has been populated, a special column will be rendered, allowing the node operator to perform edit/delete actions over the nodes.

When the node operator clicks in the "edit" button, a modal is shown with a form populated with the repository data.
When the node operator clicks in the "delete" button, a dialog is shown asking the node operator to confirm the action.

The page must have other 3 secondary elements:

A "go back" button in the header, placed at the far left. It leads the node operator to the previous step (Add Sub-Chain).
A "help" button so the node operator can click and get to know more information about what he's configuring at the moment. This in-depth info must be displayed in a modal.
A Sub-Chain filter, so the node operator can filter the data table. The implementation looks something like this: an svc-select wrapped by an svc-filter to fire events with the selected sub-chain and then update the data table as necessary.

Notice: in the prototypes some icons are missing in the footer, stick with what we have currently for PANIC UI.

When the user clicks in the "Add Node" button.

When the user clicks in the "Add Node" button. [2]

With data.

The scope of this ticket is limited to CL and Cosmos only. Substrate will be tackled individually before the release of the P1 metrics

Post review @zimaah


Bug: A helper named 'carousel-container' could not be found

If you put anything other than an  into the {{carousel-item}} block, you get the A helper named 'carousel-container' could not be found error.
{{#carousel-item}}

{{/carousel-item}}

More complicated example:
{{#if data.length}}
  {{#carousel-container transition-interval=400}}
    {{#carousel-body}}
      {{#each data as |datum|}}
        {{#carousel-item}}
          {{#each datum as |item idx|}}
            {{#with (get-field group.fields idx) as |field|}}
              {{#if (eq field.displayType 'image')}}

              {{else}}
                {{#if (eq field.displayType 'url')}}
                  Link
                {{else}}
                  {{item}}
                {{/if}}
              {{/if}}
            {{/with}}
          {{/each}}
        {{/carousel-item}}
      {{/each}}
    {{/carousel-body}}

        {{#carousel-arrow direction="left" tagName="button" class="btn btn-link btn-block"}}
          Previous
        {{/carousel-arrow}}

        {{#carousel-arrow direction="right" tagName="button" class="btn btn-link btn-block"}}
          Next
        {{/carousel-arrow}}

  {{/carousel-container}}
{{/if}}

In this example, the  throws the error.

After trying a few scenarios, it seems like the issue happens if the element is in the {{carousel-body}} tag, and not necessarily the {{carousel-item}}.

So I tried to recreate it, but wasn't able to. I deleted my tmp/ and dist/ folders and today it's working fine. You said it @runspired :smile:


[5.5] customizable logout redirection

add flexibility to an application

Thanks, but I don't think this is really needed, and I think has been proposed in the past? Just override that method in your own implementation? Is probably sufficient?

Overwrite method, call parent, redirect to where you want.


Update dependencies to latest version 🚀

Update dev dependencies to latest version.

closes #118
closes #119
closes #120


fix: Set boder-bottom-width  to 0 to prevent from double border of tab.

General

✏ Mark the necessary items without changing the structure of the PR template.

[x] Pull request template structure not broken

Type

ℹ What types of changes does your code introduce?

👉 Put an x in the boxes that apply

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Checklist

ℹ Check all checkboxes - this will indicate that you have done everything in accordance with the rules in CONTRIBUTING.

👉 Put an x in the boxes that apply.

[x] My code follows the style guidelines of this project
[x] Is the code format correct
[x] Is the git submission information standard?
[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules

There are double bottom border of tab. This PR remove the border from ant-design-vue tab component while the thicker one is set explicitly in src/layouts/default/tabs/index.less#L24.


refactor K8sManifest

differentiate between K8sObjectManifest and K8sCredentialManifest but
have both of them inherit from a higher level K8sManifest
for each resolver, use its local call to retrieve K8s resources. this
allows custom emission of events where needed, particularly as deployment
of resources progresses. see the example case for when a K8s resource is
deployed, which cause of an oversight was not working as expected
add tests to ensure we emit deploying, and deployed events for k8s resources

great idea with adding our own annotations to resources the plugin creates. added #42 to track it.


Support 128bit integers

serde 1.0.62 1.0.60 was recently released with optional support for 128bit numbers. It would be good if we could wire them up to bincode. There's a convenient serde_if_integer128! macro that serde exposes we could use to automatically detect newer compilers, but that might not line up so well with byteorder's explicit i128 feature. Maybe we could do something like this:
In Cargo.toml:
[features]
i128 = ["byteorder/i128"]

In ser.rs or de.rs:
impl Serializer {
    serde_if_integer128! {
        #[cfg(feature = "i128")]
        fn serialize_i128(self, v: i128) -> Result<Self::Ok, Self::Error> {
            // use `byteorder` to serialize our value
        }

        #[cfg(not(feature = "i128"))]
        fn serialize_i128(self, v: i128) -> Result<Self::Ok, Self::Error> {
            let _ = v;
            Err(Error::custom("i128 is not supported. Enable the `i128` feature of `bincode`"))
        }

        ...
    }
}

so that vanilla bincode keeps working on older compilers, but has a more helpful error message if you accidentally use 128bit numbers without enabling the feature?
Or maybe we could work around it using some build.rs wizardry?

Hmm actually, if we wanted to do something with an explicit i128 feature on bincode maybe it would be better if it looked like:
impl Serializer {
    #[cfg(feature = "i128")]
    fn serialize_i128(self, v: i128) -> Result<Self::Ok, Self::Error> {
        // use `byteorder` to serialize our value
    }

    serde_if_integer128! {
        #[cfg(not(feature = "i128"))]
        fn serialize_i128(self, v: i128) -> Result<Self::Ok, Self::Error> {
            let _ = v;
            Err(Error::custom("i128 is not supported. Enable the `i128` feature of `bincode`"))
        }

        ...
    }
}

So that enabling it always tried to implement the method, but if it's available and not implemented then we suggest the feature.

This looks like a good plan!
Would you feel comfortable implementing it?  If not, I can get to it later this week.

@TyOverby Sure! I'm happy to submit a PR for this.

Hi @TyOverby! Are you happy to push out a 1.0.1 patch with the 128bit number support?

Published!


Add support for primary_label in specifying line/col information

As discussed in #91 and #208, it is sometimes desirable for the line/column numbers in a diagnostic to reference a label as opposed to the start of the context. However, there has to be a way of identifying which label to reference.
This PR implements one possible mechanism: a primary_label attribute which acts just like label, except it marks a single label as being "primary". For example, this program:
use miette::{Diagnostic, SourceSpan};
use thiserror::Error;

#[derive(Error, Debug, Diagnostic)]
#[error("Primary span error")]
struct PrimarySpanError {
    #[source_code]
    src: String,
    #[primary_label] // note the new attribute
    span: SourceSpan,
}

fn main() -> miette::Result<()> {
    Err(PrimarySpanError {
        src: String::from("Hello\nworld"),
        span: (7..10).into(),
    }
    .into())
}

Produces this output:
Error:   × Primary span error
   ╭─[2:2]
 1 │ Hello
 2 │ world
   ·  ───
   ╰────

Note that the line/column info is 2:2 rather than 1:1 which you would get with a normal #[label].
The core technical changes are:

Adding a primary: bool field to LabeledSpan.
Adding a method LabeledSpan::new_primary_with_span that constructs a LabeledSpan with primary: true.
Extending the derive macro to support primary_label.
Modifying the GraphicalHandler to look for a primary label and use it if possible.
Added a test to graphical.rs to check for this behavior.

These changes were specifically made to be fully backwards compatible.
This is my first time in the miette codebase, so I may have missed a code path. Let me know if I should change anything.

I think this kind of change is a long time coming and I really appreciate you taking the time to do this!
As far as specifics go: instead of having to add a new attribute, how would you feel about an API where, by default, the first attribute tagged with #[label] is treated as the primary one, and if you want a subsequent attribute to be the primary instead, you do #[label(primary)]? I think 99.9999% of the time, people don't really want the current "viewport" to be what shows up in that [line:col] display, and it's been an ongoing issue. I think a lot of people would be very happy if, without having to make any changes, that display just magically pointed to one of their spans, which will for the most part be the only one to begin with.

I would be happy with that design as well. Two questions:

When you say "an API", do you mean a change to the default API everyone uses, or a distinct new part of the miette API surface?
If a user wants a format string in a primary label, should the expected syntax be #[label(primary, "{}", arg)]?

Change #[label] itself so it starts marking things as "primary"
Yes. Basically in https://github.com/zkat/miette/pull/291/files#diff-a775e9b52316667cfcabc8d12a083cba8d18759534d48b7a5da6cd647bc6fe6dR394, to .or_else(|| labels.first()) (or whatever)

Ok I just updated the code and the tests. How does it look?


Optimize storage of dependency keys

Bug: https://bugs.swift.org/browse/SR-1140
@ddunbar This is working except that no entry from keys table is deleted right now.
Can you check if this is right approach.

Yes, this seems like the right approach to me... added a bunch of nit-level comments.
The major high level comment is needing to understand the performance impact. I would like to understand two things:

Performance impact on a full build (when the database is empty, and we are creating a bunch of data).

My guess is that we need some kind of cache for the keys, but I'm unclear of how we should go about structuring that... I don't think I want us to cache all of them, that seems wasteful and also means we are just duplicating work from SQLite.
This is one area where I am a little worried that we may need to change the interface to the BuildDB in order to be able to get the space win and the performance we want. If we changed the way dependencies were stored at the BuildDB interface level, then the rest of the engine might only need the ID into the Rule, which would then mean we wouldn't have to do a ton of lookups just to resolve it. If that makes sense...
I need to mull on this some more, and would be interested if you have other thoughts about what we could do here...

maybe we could try with a small LRU cache for keys and test the performance but I guess that won't be enough for a large build and managing cache could be more difficult than we want?
I like the idea of Rule having keyID instead of key that should definitely fix the performance issues but idk what the overall impact on the code would be, though definitely worth exploring.
Another thing I was thinking was to try with nested queries which maybe faster than two individual queries.

@ddunbar I am done with nits, will look into performances

I got these results so far :
Master: average: 0.332
Current patch: average: 0.360
If I use this query the average is actually increasing:
*findRuleDependenciesStmtSQL =
  "SELECT key FROM key_names WHERE id IN (SELECT key_id FROM rule_dependencies WHERE rule_id == ?);";

Query: average: 0.461

My guess is you want a JOIN, not a nested query.
FWIW, I often prototype this by just using sqlite3 on the command line to find the best query, then implement it in code.

On Apr 13, 2016, at 2:18 AM, Ankit Agarwal<EMAIL_ADDRESS>wrote:
I got these results so far :
Master: average: 0.332
Current patch: average: 0.360
If I use this query the average is actually increasing:
*findRuleDependenciesStmtSQL =
"SELECT key FROM key_names WHERE id IN (SELECT key_id FROM rule_dependencies WHERE rule_id == ?);";
Query: average: 0.461
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub https://github.com/apple/swift-llbuild/pull/15#issuecomment-209326997

@swift-ci please test

Updated to join, now:
patch -> average: 0.285
master -> average: 0.335
:D

Nice! Is it worth optimizing the initial rule_result query to also use a join so only one query is needed?
Also, does this pass all tests?

current patch is average: 0.243
with
static constexpr const char *findRuleResultStmtSQL =
    "SELECT rule_results.id, value, built_at, computed_at FROM rule_results "
    "INNER JOIN key_names ON key_names.id = rule_results.key_id WHERE key == ?;";

it becomes average: 0.268
So I guess not worth it, however I removed the unused keyForKeyID method
there are 3 lit tests failing, looking into that.

I wonder why it got slower, seems like it should be strictly less work...

@swift-ci please test

@ddunbar fixed the tests

@swift-ci please test

@swift-ci please test

LGTM!

Looks like something is wrong here, my performance monitoring bots noticed a major regression in the initial build time (null build time did decrease, as expected, although unfortunately not by the same percentage).
This regression is pretty huge, unfortunately:

@aciidb0mb3r can you take a look and see if it is something simple you think can be resolved in short order? If not we should probably revert this while the investigation takes place.

Oh, also the static analyzer flagged this:
Assertion: Value stored to 'result' is never read
File: llbuild/lib/Core/SQLiteBuildDB.cpp:446

Introduced by Ankit Agarwal (b3b71cd)


Create modal search for Client Information

Task:
A Modal Client Search function needs to be developed to allow users to look up client numbers.

Acceptance Criteria:
[ ] Paginated search output
[ ] Users can look up by name, acronym, client number

Subtasks related to the issue:
#497
#498
#499


videos for nov/dec must be assigned to FS and BB

each person does ca half a day
fabians posts his list here, BB takes then exactly the rest

See #337


support metrics removal from default allowlist

if a metric name ending in -,  we will delete this metric from the default allowlist.
https://github.com/open-cluster-management/backlog/issues/12106
Signed-off-by: Song Song Li<EMAIL_ADDRESS>

/lgtm

@songleo
for "match" section in allowlist, we also need to support removal


Query: SqlFunctionExpression instance should be SqlExpression rather than Expression

Small typo/mistake but its a public surface so we can't change it without breaking.

also, rename parameter name of SqlExpressionVisitor.VisitSqlUnary from sqlCastExpression to sqlUnaryExpression

... also SqlExpressionVisitor missing overload for SqlConstantExpression


Changetracker is in CBLite: Waiting: true state and not fetching docs.

Hi Folks,

Changetracker gets deleted docs from syncgateway and validator rejects them.

@[10/09 13:10:23:429] [622] ChangeTracker<EMAIL_ADDRESS>OneShot, @224820bc}: Making request to POST [https://---:---@syncqc7.company.com/data/_changes?feed=normal&heartbeat=90000&style=all_docs&since=9382042%3A8493520&filter=sync_gateway%2Fbychannel]

Changetracker makes request.

CBLite: setPaused: true

[10/09 13:10:24:937] [590] Sync: com.couchbase.lite.replicator.PullerInternal$9@e0903f2 insertDownloads() updating completedChangesCount from 2710 -> 2711
[10/09 13:10:24:937] [590] Sync<EMAIL_ADDRESS>pull, 88db8}: Incrementing completedChangesCount count from 2710 by adding 1 -> 2711
[10/09 13:10:24:937] [590] CBLite: setPaused: true

https://github.com/couchbase/couchbase-lite-java-core/blob/release/1.4.0/src/main/java/com/couchbase/lite/replicator/PullerInternal.java#L1085

CBLite: Waiting: true and never recovers.

Thanks
Nithin

cbl_logs.txt
Detail Logs attached..

A wait state is caused by the sum of items in batcher and pendingSequences are more than MAX_PENDING_DOCS which is 200. It indicates that tasks that pull the actual revision from the SG are slower than retrieving next changes information by ChangeTracker. It could frequently occur if many changes exist on the SG. If pull replication is never finished, pulling the actual revision or inserting it into the database encounters the problem. But from the log file, I could not find the indication.
    protected void pauseOrResume() {
        int pending = batcher.count() + pendingSequences.count();
        changeTracker.setPaused(pending >= MAX_PENDING_DOCS);
    }

In the log file, I could observe many 404 errors.
[10/09 13:09:05:630] [614] Sync<EMAIL_ADDRESS>pull, 88db8}: giving up on {s919999910122_ptx1506373952461_meta.doc #3-a1f3d7939d53685c1de36d4acd1c4919 @2027}: com.couchbase.lite.CouchbaseLiteException, Status: 404 (HTTP 404 not_found)

But, in case of facing 404, pull replicator remove items from pendingSequence container.
NOTE: 404 is not transientError
    private void revisionFailed(RevisionInternal rev, Throwable throwable) {
        if (!Utils.isTransientError(throwable)) {
            Log.v(TAG, "%s: giving up on %s: %s", this, rev, throwable);
            pendingSequences.removeSequence(rev.getSequence());
            pauseOrResume();
        }
        completedChangesCount.getAndIncrement();
    }

@hideki
1)When CBL(user A) is busy fetching old seq numbered docs.
2)Will the latest pushed docs(It will processed by Syncgateway and belongs to user A) will be part of the changes immediately?
Please share your thoughts..

Hi @NitzDKoder,
I recommend asking Sync Gateway question at Couchbase Forum or file the ticket on sync_gateway repository issue if you found the Sync Gateway problem.
Thanks

@hideki One more instance..

Logs attached
cbl_logs.txt
cbl_logs_old.txt
..

Hi @NitzDKoder
Do you have additional information that you can share with this regarding this issue?

Hi @NitzDKoder,
I just read your other ticket #1674. From that ticket, I assume this ticket #1670 could be caused by a deadlock. Can you also capture the thread-dump when you encounter this problem again?
Thanks!

@hideki we are trying to get threaddump..

@hideki
Use case:
1)2 continuous pull replication started.
2)Server has many docs(~14577) for the Client.
Client stuck in CBLite: setPaused: true and not pulling any docs.
CBL and thread dump losg added.
issue seen when client has more docs.
cbl_logs.txt
threads_report.txt
threads_report1.txt
threads_report2.txt
threads_report4.txt
threads_report5.txt
threads_report6.txt
threads_report8.txt
threads_report9.txt
threads_report10.txt

@hideki Syncgateway logs added..
Please check and raise if any syncgateway issue..
sync_gateway_access.log
sync_gateway_access-2017-11-14T17-01-16.584.log
sync_gateway_access-2017-11-14T17-04-41.797.log
sync_gateway_access-2017-11-14T17-05-51.885.log
sync_gateway_access-2017-11-14T17-06-21.440.log
sync_gateway_access-2017-11-14T17-07-10.957.log
sync_gateway_access-2017-11-14T17-08-10.008.log
sync_gateway_access-2017-11-14T17-08-43.940.log
sync_gateway_access-2017-11-14T17-09-18.273.log
sync_gateway_access-2017-11-14T17-09-52.473.log
sync_gateway_access-2017-11-14T17-10-15.993.log
sync_gateway_access-2017-11-14T17-11-02.612.log
sync_gateway_access-2017-11-14T17-11-25.492.log
sync_gateway_access-2017-11-14T17-11-51.614.log
sync_gateway_access-2017-11-14T17-12-26.975.log
sync_gateway_access-2017-11-14T17-12-49.960.log
sync_gateway_access-2017-11-14T17-13-26.685.log
sync_gateway_access-2017-11-14T17-13-47.647.log

Note to Self:

From thread-dump, none of the threads are blocked by other threads. So no dead-locking is observed. All replicator related threads are a waiting state.
log message: CBLite: Waiting: true indicates too many tasks in a queue, so ChangeTracker will not take new tasks (changes). https://github.com/couchbase/couchbase-lite-java-core/blob/3ed5056ee4edfe27e67f816f08d6e342fd7f8b73/src/main/java/com/couchbase/lite/replicator/ChangeTracker.java#L650-L661
log message: CBLite: setPaused: true indicates too many tasks in the queue, so tell ChangeTracker to pause to retrieve changes. https://github.com/couchbase/couchbase-lite-java-core/blob/3ed5056ee4edfe27e67f816f08d6e342fd7f8b73/src/main/java/com/couchbase/lite/replicator/ChangeTracker.java#L640-L648 https://github.com/couchbase/couchbase-lite-java-core/blob/4f52cd55b50d32d7ef2cf79a3c0eb4bf38cde770/src/main/java/com/couchbase/lite/replicator/PullerInternal.java#L1090-L1093
log message: ReplicationInternal$2@25cd0502: BEGIN processInbox (1 sequences) indicates only one item is in the batcher queue. And, pendingSequences.count() returns 200 (MAX_PENDING_DOCS) or more.

From above indications, items in pendingSequences is not correctly removed by removeSequence(long) method. It could be expected sequence number by CBL client and sequence number with revision from SG does not match?

@NitzDKoder,
Question: When you encounter this issue, Does Sync Gateway and Couchbase Server cluster switch or something happen on the server side?

Hi @NitzDKoder,
First I'd like to confirm that Waiting: true log message is not the error message and it shows the state of ChangeTracker module.  In general,  retrieving changes from Sync Gateway is much lighter weight task than fetching the document/revision content from Sync Gateway or inserting the content into the database. If ChangeTracker keeps retrieving changes from the SG and creating subsequent tasks (fetching doc/rev from SG), it causes Out of Memory because these tasks (consuming memory) are queued as subsequent tasks. To prevent Out of Memory, pull replicator makes ChangeTracker to waiting state till subsequent tasks size becomes smaller. Many Waiting: true is expected log messages.
Insight from cbl_logs.txt that was recently attached: If you observed that pull replication did not replicate all expected documents/revisions from Sync Gateway,  I believe it was caused by 503 Service Unavailable HTTP error that was occurred on the server side. I could not find Service Unavailable string in sync_gateway repo. So, this error could be returned by Proxy or Load Balancer.

[11/14 17:16:37:917] [2109] Sync: retryReplicationIfError() state=IDLE, error=com.couchbase.lite.replicator.RemoteRequestResponseException: {code=503, message=Service Unavailable}, isContinuous()=true, isTransientError()=true

Insight from thread-dump: Following is  ChangeTracker's thread's stack trace. The thread waited to read data from SSL socket. The thread did not wait at waitIfPaused() method. If this thread-dumps were captured at the right timing, it proves Waiting: true, pausing logic did not cause the problem that you observed. The root cause was something else. I guess 503 error from the server.
"ChangeTracker-https://---:---@locqc3.domain.com/ptxdata@24035" prio=10 runnable
  java.lang.Thread.State: RUNNABLE
	  at com.android.org.conscrypt.NativeCrypto.SSL_read(NativeCrypto.java:-1)
	  at com.android.org.conscrypt.OpenSSLSocketImpl$SSLInputStream.read(OpenSSLSocketImpl.java:728)
	  - locked <0x5ea7> (a java.lang.Object)
	  at okio.Okio$2.read(Okio.java:140)
	  at okio.AsyncTimeout$2.read(AsyncTimeout.java:238)
	  at okio.RealBufferedSource.request(RealBufferedSource.java:66)
	  at okio.RealBufferedSource.require(RealBufferedSource.java:59)

Thread should wait at LINE 656 if pausing logic causes problem.
https://github.com/couchbase/couchbase-lite-java-core/blob/3ed5056ee4edfe27e67f816f08d6e342fd7f8b73/src/main/java/com/couchbase/lite/replicator/ChangeTracker.java#L650-L661
Please let us know if this makes sense?

@hideki So you mean after fetching docs from Server the changetracker will move out of the wait state and starts pulling?
We see this issue when user has lot of data.
SO we need to wait for longer time such that CBL will pull all data?
Thanks
Nithin

@"retryReplicationIfError() state=IDLE, error=com.couchbase.lite.replicator.RemoteRequestResponseException: {code=503, message=Service Unavailable}, isContinuous()=true, isTransientError()=true"
Sever was not down at that point of time. Will check the use case again and see if i can get different use case.
Will share logs.

@NitzDKoder

So you mean after fetching docs from Server the changetracker will move out of the wait state and starts pulling?

Yes

We see this issue when user has lot of data.

I tested pull replicating 400,000 documents today. With my test environment, I could not observe the problem.
Waiting: true log message itself is not the problem. If ChangeTracker can not move out from the state, it is a problem. However, I can not observe the indication from the latest log file and thread-dump.

SO we need to wait for longer time such that CBL will pull all data?

It depends on the environment such as a number of docs, size of docs, network condition, or device.

@hideki other thing to notice is that client pulled lot of deleted docs(1000s) from server.
https://github.com/couchbase/couchbase-lite-java-core/issues/1264#issuecomment-323161266
We have validator to reject those.
"Database: reject PTX server TTL doc delete or channel remove"

@hideki
1)When CBL(user A) is busy fetching old seq numbered docs.
2)Will the latest pushed docs(It will processed by Syncgateway and belongs to user A) will be part of the changes immediately?
Line 25410: [11/14 17:10:44:653] [1980] CBLite: ---> Added: {s919888880032_ptx1510659644491_meta.doc #1-bda4d3a410a0f2934e78397acb1f28d5 @30} as seq 30
Please share your thoughts..

@NitzDKoder
Can you ask the last question through the customer support, Couchbase Forum or  Sync Gateway?

@hideki
@[10/09 13:10:24:935] [590] Sync: com.couchbase.lite.replicator.PullerInternal$9@e0903f2: Remote rev failed validation: {s919999910122_ptx1506127718008_meta.doc #3-fc81f6330ef76ade75e4f76ec8d36adb @2712 DEL}
[10/09 13:10:24:936] [590] Sync: com.couchbase.lite.replicator.PullerInternal$9@e0903f2: finished inserting 1 revisions
[10/09 13:10:24:936] [590] Sync: com.couchbase.lite.replicator.PullerInternal$9@e0903f2: inserted 1 revs in 11 milliseconds
[10/09 13:10:24:937] [590] Sync: com.couchbase.lite.replicator.PullerInternal$9@e0903f2 insertDownloads() updating completedChangesCount from 2710 -> 2711
Going back to initial logs which was shared.. CBLite: setPaused: true
[10/09 13:10:24:937] [590] Sync<EMAIL_ADDRESS>pull, 88db8}: Incrementing completedChangesCount count from 2710 by adding 1 -> 2711
Change tracker thread [622] CBLite: Waiting: true for 3 mins..when would it recover and its not pulling any doc from sever either..
cbl_logs (1).txt

@NitzDKoder

other thing to notice is that client pulled lot of deleted docs(1000s) from server.

I tested 1000 deleted pull replication and also 400000 regular docs and 1000 deleted document pull replication. Both scenarios work fine with my env.

@NitzDKoder
From cbl_logs (1).txt, I could observe following messages between Waiting: true. setPaused(false); makes ChangeTracker out of paused state.  So ChangeTracker frequently entered pause state and out of pause state. So this is expected behavior to prevent Out of Memory.
[10/09 13:12:44:450] [599] Sync: setPaused: false

@NitzDKoder,
I have a suggestion. I believe you could compile CBL from source code. So you could add logging at LINE660 to make sure if ChangeTracker exists from pause state?
https://github.com/couchbase/couchbase-lite-java-core/blob/3ed5056ee4edfe27e67f816f08d6e342fd7f8b73/src/main/java/com/couchbase/lite/replicator/ChangeTracker.java#L660

@hideki
@	Line 14: [10/09 13:12:49:129] [622] CBLite: Waiting: true
Line 92: [10/09 13:12:54:129] [622] CBLite: Waiting: true
Line 93: [10/09 13:12:59:130] [622] CBLite: Waiting: true
Line 94: [10/09 13:13:04:131] [622] CBLite: Waiting: true
Line 125: [10/09 13:13:09:132] [622] CBLite: Waiting: true
Line 156: [10/09 13:13:14:133] [622] CBLite: Waiting: true
Line 172: [10/09 13:13:19:133] [622] CBLite: Waiting: true
Line 173: [10/09 13:13:24:134] [622] CBLite: Waiting: true
Line 174: [10/09 13:13:29:135] [622] CBLite: Waiting: true
Line 223: [10/09 13:13:34:136] [622] CBLite: Waiting: true
Line 224: [10/09 13:13:39:137] [622] CBLite: Waiting: true
Line 255: [10/09 13:13:44:137] [622] CBLite: Waiting: true
Line 266: [10/09 13:13:49:138] [622] CBLite: Waiting: true
Line 297: [10/09 13:13:54:139] [622] CBLite: Waiting: true
[622] is still in waiting state, can you clarify on that.
Will add logs to ChangeTracker exists from pause state.

@NitzDKoder As I wrote before, ChangeTracker enters the waiting (pausing) state and out of state repeatedly if there are many changes. CBL logs only the waiting state, not the non-waiting state. So, many Watiting: true logged strings cannot prove if ChangeTracker is stuck in waitIfPaused() method.

@hideki will expirement on with logs added..
Thanks!

@hideki
Have added more logs..
  protected void waitIfPaused() {
       Log.v(Log.TAG, "waitIfPaused Start: " + paused  +" running: "+running);
       synchronized (pausedObj) {
           while (paused && running) {
               Log.v(Log.TAG, "Waiting: " + paused);
               try {
                   // every 5 sec, wake by myself to check if still needs to pause
                   pausedObj.wait(TIMEOUT_FOR_PAUSE);
               } catch (InterruptedException e) {
               }
           }
           Log.v(Log.TAG, "waitIfPaused Exit: " + paused  +" running: "+running);
       }
   }

@	Line 102613: [11/23 11:43:35:864] [531] CBLite: waitIfPaused Start: true running: true
Goes to wait state
@[11/23 14:05:09:453] [531] CBLite: Waiting: true
Still in waiting.
[11/23 14:05:09:453] [531] CBLite: Waiting: true
Please check , logs added.
3.txt
4.txt
5.txt
6.txt
one.txt
two.txt
Thanks
Nithin

From the log files, waitIfPaused Exit: message is not observed.  It indicates paused variable was not changed to false. I assume other threads can not process succeeding tasks. Then pending >= MAX_PENDING_DOCS was always true.
protected void pauseOrResume() {
     int pending = batcher.count() + pendingSequences.count();
     changeTracker.setPaused(pending >= MAX_PENDING_DOCS);
 }

@hideki
[11/23 11:43:36:724] [520] Sync<EMAIL_ADDRESS>pull, 6afd0}: Progress: set error = com.couchbase.lite.replicator.RemoteRequestResponseException: {code=503, message=Service Unavailable}
[11/23 11:43:36:725] [520] Sync: com.couchbase.lite.replicator.RemoteBulkDownloaderRequest[/ptxdata/_bulk_get]: RemoteRequest execute() finished, url: https://919888881462:3617cc40910fb5470b71f0647d78fb53@syncqc3.kodiakhandset.com/ptxdata/_bulk_get?revs=true&attachments=true

Puller is getting "code=503, message=Service Unavailable" is this issue here..Puller should retry and come out of wait state isnt..???
Thanks
Nihtin

503 is a tentative error. So the replicator should retry.

Same as https://github.com/couchbase/couchbase-lite-java-core/issues/1670


For Android, anyway to expand the notification to see all the text if my message is too long?

Title.
I plan to have a longer alert message and they cannot fully be shown

You can send a larger message using the big_text parameter, and in the alert send it in a summarized way.

For testing this is my payload :
             "alert": "alert",
             "title": "this is title",
             "icon" : "http://i.imgur.com/IGxUq3Z.jpg",
             "vibrate": true,
             "sound": true,
             "big_text" : "this is big text"
             "force_show_in_foreground" : true,
             "id": 2,

However I cant see the message "this is big text" from my notification. I am only seeing "alert" and "this is title", and the notification is not expandable

I tested it here and it worked normally, it follows the information from my test, I hope something helps you figure out what might be happening.
Payload
{
    "data": {
        "id": "2",
        "vibrate": true,
        "title": "Netto Ramalho",
        "icon": "https://s-media-cache-ak0.pinimg.com/avatars/neto_ramalho_1_1403441759_140.jpg",
        "alert": "3 new messages",
        "big_text": "- Message 1\n- Message 2\n- Message 3"
    }
}

Prints
https://drive.google.com/open?id=0BylxnWR5r-V3RjFKdzFnSEd2R0E
Environment
sdk-version: 5.5.1.GA
module-version: 1.6.2
android-version: 6.0

hmm. I am using exactly the same environment as yours, only that I am using a rooted device, not sure if that is the problem.
Ill try to figure it out. thanks for the help

OK. I copied and tested your payload. It is working.
Ill make some changes accordingly
Thanks again

Hi I have another question
In your screenshot, there is a small icon at the bottom right of your notification icon, what is that? I am having a similar thing, but an empty circle for me. Any way to remove it?

See section Set the icon for the tray in the readme file, explain how to change this icon, I do not know how to remove, but anyway I think it is not a good idea, this icon is displayed in other places where it is important the user to identify the symbol of its application

You can't remove that icon.
If you don't set the icon, no notification has been displayed; we put an empty circle as a default for this reason.


Support table cell entry @rotate attribute

Based on this DITA Users List feedback:

I have tried text rotation in cells using  No rotation occurred. Does anyone have any suggestions as to why this might not be working?

As a possible solution in the XSLT stylesheet:
      DITA-OT\plugins\org.dita.pdf2\xsl\fo\tables.xsl

the template with mode "processTableEntry" could be modified as such:
            <xsl:template match="*" mode="processTableEntry">
                <xsl:call-template name="commonattributes"/>
                <xsl:call-template name="applySpansAttrs"/>
                <xsl:call-template name="applyAlignAttrs"/>
                <xsl:call-template name="generateTableEntryBorder"/>
                <fo:block-container reference-orientation="{if (@rotate eq '1') then 90 else 0}">
                    <fo:block xsl:use-attribute-sets="tbody.row.entry__content">
                        <xsl:apply-templates select="." mode="ancestor-start-flag"/>
                        <xsl:call-template name="processEntryContent"/>
                        <xsl:apply-templates select="." mode="ancestor-end-flag"/>
                    </fo:block>
                </fo:block-container>
            </xsl:template>

Only Antenna House seems to have decent support for this from what I tested. Both XEP and Apache FOP do not properly render the table.

We should not generate fo:block-container elements unless they are absolutely needed. They will mess up @start-indent and other other similar properties. Thus, if something like this is added, we need to only generate the container if it's rotated.

👍

They will mess up @start-indent and other other similar properties.

What the meaning of the phase "mess up"?
Even if fo:block-container is located inside the fo:table-cell, the @start-indent is inherited from fo:table-caption to inside the fo:block-container. This inheritance should be the feature of XSL-FO specification.
So there should be no influence to generate  redundant fo:block-container inside fo:table-cell.

How is this issue related to #1778?

@kirkilj this appears to be a duplicate of 1778 - I had not noticed the extra copy, or I would have closed one of them a while back. Thanks for helping to clean up the list of issues.
#3161 is about ready to go, and will handle the @rotate attribute in the upcoming 3.3 release for this issue / for 1778.


FR: BR_SetItemEdges -1 value for not moving edge

Hi !
It would be nice if we could pass "-1" (or any negative number) as parameter value into BR_SetItemEdges, to ignore one edge 😄
Thx !

@nofishonfriday what is this tag ? "good first issue" ? ^^

@X-Raym
https://help.github.com/en/articles/helping-new-contributors-find-your-project-with-labels
I put it there, hoping to make it easier for potential new SWS contributors to get started (seeing which issues are relatively 'straightforward' to tackle.). :)

The relevant bit of code which would have to be modified is here btw. :)
https://github.com/reaper-oss/sws/blob/next/Breeder/BR_Util.cpp#L1067

@nofishonfriday I think there might be some optimizaiton possible there,
for eg, if you have an item at position > 1 and you say
reaper.BR_SetItemEdges(-1, 1)
We might expect that the function doesn't do anything as it is invalid,
but in fact it will move item left edge to -1 and keep right edge in place.
What do you think ?

This is not new: that function has always swapped the supplied left/right positions if the former is larger than the later.

@cfillion ok I didnt know about this auto swapping, The API doc doesn't mention that case. Thx !


[VL] Use collect_list aggregate function in velox

What changes were proposed in this pull request?
https://github.com/facebookincubator/velox/pull/9231 implemented a new collect_list aggregate function based on the semantics of Spark. Gluten can switch to use this new aggregate function without the need to rewrite the collect_list function in Spark.
How was this patch tested?
Exists CI tests.

cc @ulysses-you.

Hi @liujiayi771, I note @ulysses-you once proposed a PR to make null-handling aligned with Spark: https://github.com/apache/incubator-gluten/pull/4805. Do we still need it for collect_list?

Hi @liujiayi771, I note @ulysses-you once proposed a PR to make null-handling aligned with Spark: #4805. Do we still need it for collect_list?

Just note collect_list is excluded for that logic. Then, the PR looks good!


Automatic screen name includes object address and makes it unusable

Expected Behavior

Current Behavior

Currently screen name is captured in a way that includes object address, which is different each time object instantiated. This way I can't filter by actual Screen Name because there are plenty of records for single screen name.

Possible Solution

There are plenty of possible solutions, but the easiest one would be to just print Class of an object, and not include brackets, object address etc. So that it would be:
FriendProfile instead of <Evenly.FriendProfile: 0x10501e800>
Steps to Reproduce

Just enable defaultTracking so that screenViews are tracked, so defaultTracking: .init(screenViews: true) or defaultTracking: .ALL and see what's received by Amplitude

Environment

SDK Version:  Amplitude-Swift 1.1.0
OS Info:  iOS 17.0.1, iOS Simulator iPhone 15 Pro Max

Hi @nezhyborets - this should be fixed in the latest release.


UI Dropdown - Placeholder

Description
Dashboard 1.0 offers the configuration of "placeholder" for a ui-dropdown
Have you provided an initial effort estimate for this issue?
I have provided an initial effort estimate

We already have "label", so I do feel like this is a low priority, event hough it's available in D1.0.

Closing as "label" in the new design covers this


Can't update payment method, save button disabled

When trying to update a payment method in the subscriptions screen, it can be that the save button stay disabled, preventing the user to perform the intended action.
https://opencollective.freshdesk.com/a/tickets/1483

Hello @znarf

I request to be assigned this issue
Could you please provide a method through which I can replicate this issue?

Thanks

I'm not sure there's an issue with this or maybe a more context can help

As I see from the above gif, the save button is disabled when the value (payment method) is the same as the last and is enabled when there is a change.
This is the same behaviour we follow when updating descriptions, (disabled when the same i.e. no changes are done)

@flickz There is definitely an issue with this. We just have not figured the exact steps. I think by looking carefully at the code, someone can find the reason. I can't share more than.

To reproduce:

Make sure you have a first Credit Card Payment Method in your account
Make a contribution with a new Credit Card Payment Method
In the database, delete the created Credit Card Payment Method (in the case of this user, we can assume it expired by itself)
Edit the subscription, the first Credit Card Payment Method is selected, but save is disabled.


Fix formatting of \0 in tests

Tests were broken.
This follows this PR in rustc:
https://github.com/rust-lang/rust/pull/95345
Which breakage was discussed here:
https://github.com/rust-lang/rust/issues/95732
Because this landed in 1.61.0, the MSRV has to be updated accordingly.

Interesting. Sadly your solution is not a fix, since we do support Rust 1.31+
I would suggest using \u{1} instead of \0 in tests. This would not require MSRV bump.

Thanks for the quick review. I've changed the diff accordingly. :)

Thanks!


[PE-D][Tester A] Error Starting autom8 jar file on windows

Images tester sent:

Need to update docs to match our jar file name in releases*
Need devs who use windows to debug this @junlee1991 @9fc70c892

If possible can debug this asap, bcs we gg if some ppl with windows cant start the JAR file at all


Russian spell checker not work

My Setup

Operating System: Windows 10
App Version: 2.17.7
Installation type: exe

Description
I'm copiyng file *.aff and dic to c:\Program Files\Rocket.Chat\resources\dictionaries
If i'm try to add dictionary from programm i get error:
Error ccpiyng dictionary file: Error invoking remote method  "spell-checking/install-dictionaries": [object Object]
Current Behavior
Input incorrect word not have red lines
Expected Behavior
Input incorrect word must be with red lines
I`m use this dictionary
dictionaries.zip

I`m Creating folder in c:\Users\wowansm\AppData\Roaming\Rocket.Chat\ and copying dic and aff filles to there.
Nothing change, spell checker for russian langusge doesnt work.
No red lines under incorrect word

Related #1565

Fixed in 3.0

Please test if the latest release did solve this. If not please reopen the issue or create a new one and tell us. I will be closing this one for now.
Thank you!


Vscode terminate automatically without any error message after few minute turning it on

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: 2022
OS Version: Windows 11 Home

Steps to Reproduce:
Turn on the vscode it will automatically switch off not matter what operation you done.
I tried to disable all the extension, using the vscode insider, uninstall the vscode then install it and it still reproduce the same problem.
these are what the log produce in last few lines but seems normal
[15872:0926/183740.567:INFO:CONSOLE(630)] "%cTRACE color: #888 [text file model] resolve() - enter file:///c%3A/Users/wilso/Downloads/ass1.sql", source: vscode-file://vscode-app/c:/Users/wilso/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/workbench/workbench.desktop.main.js (630)
[15872:0926/183740.567:INFO:CONSOLE(630)] "%cTRACE color: #888 [text file model] resolveFromFile() file:///c%3A/Users/wilso/Downloads/ass1.sql", source: vscode-file://vscode-app/c:/Users/wilso/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/workbench/workbench.desktop.main.js (630)
[main 2023-09-26T08:37:40.733Z] resolveShellEnv(): skipped (Windows)
[15872:0926/183740.962:INFO:CONSOLE(630)] "%cTRACE color: #888 CommandService#executeCommand _setContext", source: vscode-file://vscode-app/c:/Users/wilso/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/workbench/workbench.desktop.main.js (630)
this is the crash report
67b213ff-c68f-499b-b29d-02cf22dd48bd.zip

I tried to read the dmp online and the last line is
"This program cannot be run in DOS mode"
can this be the problem.
This problem appered since I install MSYS2, ubuntu and a virtual machine player.

Can you attach the entire log output from code --verbose

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


[5.2] Reset global scopes when booted models get cleared

This replaces #11341 (was on branch 5.1)


Legacy branding (CDS) should be changed to Dataverse

I would suggest, at a minimum, adding a note to the README  mentioning that this SDK targets Dataverse, and that CDS has been rebranded as such. There is currently no mention of Dataverse in the repo, aside from a discussion topic.

There is a larger rework in progress currently.
We will be renamespacing the client in the next drop


[WIP]CI Files for opensds

What this PR does / why we need it:
The PR includes CI Tasks like Static Checks (GoMetaLint) , Unit Testing, Integration Test scripts that will be invoked on PR or on specific label
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
scripts for Static Check with GoMetalint, UnitTest, Integration Test, e2e Test flows
Special notes for your reviewer:
please review the separation of jobs
Release note:

@satya-gorli Could you design a shell file to download those binary files automatically? I think it's not allowed to embed them directly into the project, because it's too large for users to git clone.


[WIP] Docker build grid-gateway, grid-node

Description
Automated docker build for pysyft-grid-gateway, pysyft-grid-node to docker hub.
Fixes #417
Type of change
Please mark options that are relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
This PR builds and pushes to docker hub the docker image for gateway, node.
The pysyft-grid-gateway image is built and uplodaed to dockerhub.
The pysyft-grid-node image has an error while building.
Checklist:

[x] I did follow the contribution guidelines
[ ] New Unit tests added
[ ] Unit tests pass locally with my changes

@hericlesme ,
Have put in condition that the docker build is triggered only on push to dev. Assuming the appropriate username and password are stored in secret on travis it will push to dockerhub.

@gantir I think docker builds can be triggered in docker hub using webhooks.
"Docker Hub can automatically build images from source code in an external repository and automatically push the built image to your Docker repositories."
You can see more about automated builds here.
Anyway, yes, using Travis could be an alternative.  Now, we just need someone with the needed privileges in our Docker Hub Organization.

By the way, you're doing a great job, thank you! :D

@hericlesme,
Ah didn't know about the docker hub thing. Even otherwise the travis.yml is now refactored to use build stages ;).
Thanks for the good words. !!

I'm so sorry for the delay.
We've finally published our official images at docker hub ( grid-node / grid-gateway )

Hello @gantir!
It would be awesome if we can update our docker images when changes are applied to PySyft's repository too.
It would help to keep the compatibility between the platform and the library. :)

@IonesioJunior,
The best way for what you are asking is to do what was suggested by @hericlesme . If you can add my docker user to openmined organization, I can create the appropriate webhooks.
Pinged you on slack with my docker user id.
Essentially we need to do the following :
[ ] openmined/grid-gateway on pysyft update
[ ] openmined/grid-gateway on syft-proto update
[ ] openmined/grid-gateway on pygrid update
[ ] openmined/grid-node on pysyft update
[ ] openmined/grid-node on syft-proto update
[ ] openmined/grid-node on pygrid update

We migrated from Travis CI to GitHub Actions. As such, these changes no longer apply to our context. Thank you, @gantir. :wink:


API server returns empty completions after a few successful requests

Checklist

[X] 1. I have searched related issues but cannot get the expected help.
[X] 2. The bug has not been fixed in the latest version.

Describe the bug
The OpenAI compatible AI server returns empty completions after answering correctly for a few times.
In my latest test it responded correctly 15 times before it broke. After that it just immediately responds with a 200 OK and an empty completion. The GPU load goes back to zero, so it does not do anything. It can be resolved only by restarting the server, which makes it unusable. My expectation would be that the server handles as many conversations as its batch size allows in parallel and hold up the rest of HTTP connections until a free slot is available.
The server does not print any errors on the console and there is no verbose or debug flag either.
Reproduction
Deploy any Llama2 or derived model, then start the OpenAI API server.
For example in my case of two 4090 GPUs:
python -m lmdeploy.serve.turbomind.deploy llama2 jondurbin/airoboros-l2-13b-gpt4-2.0 --model_format=hf --tp=2
python -m lmdeploy.serve.openai.api_server --tp=2 --server_name=<IP_ADDRESS> ./workspace

Python script to drive the server via the OpenAI async client protocol:
import asyncio
import random

import openai
from aiohttp import ClientSession

openai.api_base = 'http://<IP_ADDRESS>:23333/v1'
openai.api_key = 'xxx'

test_prompts = [
    "Describe the lifecycle of a butterfly.",
    "How do magnets work?",
    "Explain the theory of relativity in simple terms.",
    "What's the significance of the Mona Lisa?",
    "How does photosynthesis work?",
    "Compare and contrast Shakespeare's sonnets.",
    "Why is the sky blue during daytime?",
    "Discuss the importance of the Gutenberg press.",
    "What's the difference between mitosis and meiosis?",
    "How did the pyramids of Egypt get built?",
    "Summarize the plot of 'Moby Dick'.",
    "Describe the key principles of Renaissance art.",
    "How do volcanoes form?",
    "Explain the water cycle.",
    "Why do we dream?",
    "How does the internet work?",
    "Discuss the impact of the industrial revolution.",
    "What causes tides?",
    "Describe the events leading up to World War I.",
    "How does a microwave oven work?",
    "Why do we have leap years?",
    "Explain the concept of black holes.",
    "How are rainbows formed?",
    "Discuss the cultural impact of The Beatles.",
    "What's the role of mitochondria in a cell?",
    "Why is biodiversity important?",
    "Describe the process of fermentation.",
    "How did the Roman Empire fall?",
    "Explain the basics of quantum mechanics.",
    "What are the benefits of reading literature?",
    "How do planes fly?",
    "Why is gold valuable?",
    "Discuss the main tenets of Buddhism.",
    "How does the human eye work?",
    "Explain the process of nuclear fusion.",
    "What are the main causes of global warming?",
    "Describe the plot of 'Pride and Prejudice'.",
    "What's the difference between acids and bases?",
    "How are pearls formed?",
    "Discuss the impact of social media on society.",
    "Why do cats purr?",
    "Explain the concept of supply and demand.",
    "How does a compass work?",
    "What's the significance of the Eiffel Tower?",
    "Describe the history of the English language.",
    "Why do apples turn brown when cut?",
    "How were the Grand Canyons formed?",
    "Explain the principles of democracy.",
    "What are the pros and cons of nuclear energy?",
    "How does a bicycle stay upright?",
    "Discuss the themes in 'To Kill a Mockingbird'.",
    "What's the function of the heart in the human body?",
    "How does electricity get generated?",
    "Describe the moon's effect on Earth.",
    "What is the importance of the Amazon rainforest?",
    "Why do we get goosebumps?",
    "Explain the significance of the Magna Carta.",
    "How do clouds form?",
    "Discuss the legacy of Martin Luther King Jr.",
    "What's the difference between prokaryotic and eukaryotic cells?",
    "Why is the Dead Sea so salty?",
    "Describe the process of digestion.",
    "How did chocolate become popular worldwide?",
    "Explain the basics of artificial intelligence.",
    "Why do we have different time zones?",
    "What's the role of bees in an ecosystem?",
    "How does photosynthesis benefit animals?",
    "Discuss the cultural impact of jazz music.",
    "Explain the concept of gravity.",
    "Why do seasons change?",
    "Describe the symbolism in 'The Great Gatsby'.",
    "How do tsunamis occur?",
    "What are the key principles of communism?",
    "What's the importance of vaccination?",
    "Why does ice float on water?",
    "Explain how the printing press works.",
    "Discuss the history of tea and its global impact.",
    "How does the respiratory system function?",
    "Describe the origins of the Olympic Games.",
    "Why is recycling important?",
    "Explain the phenomenon of aurora borealis.",
    "What are the causes and effects of ozone depletion?",
    "Discuss the themes in 'Romeo and Juliet'.",
    "How do satellites orbit the Earth?",
    "Describe the formation of fossils.",
    "What's the role of the judiciary in a democracy?",
    "Why is the Great Wall of China significant?",
    "Explain the process of osmosis.",
    "What's the impact of the Silk Road in history?",
    "How does a computer's CPU work?",
    "Why do birds migrate?",
    "Discuss the impact of the French Revolution.",
    "What's the significance of Newton's three laws?",
    "Describe the history of the piano.",
    "How do we perceive colors?",
    "Explain the workings of a thermos.",
    "Why do dogs wag their tails?"
]

async def create_chat_completion(prompt: str, index=[1]):
    while 1:
        chat_completion_resp = await openai.ChatCompletion.acreate(model="llama2", messages=[{"role": "user", "content": prompt}])
        output = chat_completion_resp.choices[0].message.content
        if output and output.rstrip():
            break
        await asyncio.sleep(1.0)

    print(f'Q #{index[0]}: {prompt}')
    print(f'A #{index[0]}: {output.lstrip()}')
    print()

    index[0] += 1

    return output

async def main():
    session = ClientSession()
    openai.aiosession.set(session)

    tasks = [create_chat_completion(prompt) for prompt in test_prompts]
    pool = set()
    results = []
    while tasks or pool:
        if len(pool) < 32 and tasks:
            task = tasks.pop()
            pool.add(asyncio.create_task(task))
        done, pool = await asyncio.wait(pool, return_when=asyncio.FIRST_COMPLETED)
        for task in done:
            results.append(task.result())

    await openai.aiosession.get().close()

if __name__ == '__main__':
    asyncio.run(main())

Error traceback
No response

Check https://github.com/InternLM/lmdeploy/issues/375

Thanks

I've tried the "fix" mentioned in #375
It results in CUDA runtime error: out of memory after handling only 6 conversations.
(2x4090, 2x24GB VRAM, 13B model in 16 bits)
So far I cannot get lmDeploy to work reliably.

It is not a fix for your case. What you need is to set renew_session  to True. https://github.com/InternLM/lmdeploy/blob/55764e0b33d8b9298f68b77484bab3832696c010/lmdeploy/serve/openai/api_server.py#L97
If you still want to use the above fix with random instance_id, you should set instance_num with a smaller value for example
python -m lmdeploy.serve.openai.api_server --instance_num 1 --tp=2 --server_name=<IP_ADDRESS> ./workspace


feat: add Integration model

Integration model is meant to be a base model for service integrations (e.g. AWS, GCP, etc.). It serves two main functions:

Manage the service integrations and related credentials within the UI;
Manage the actions/commands to be performed per service integration;
Map assets to service integrations in a similar way as Source model does.

Each service integration has its own particularities and actions, therefore, this model will be inherited and extended per service integration's needs.

I like the idea but not the implementation, in particular the hardcoded values in IntegrationType - it doesn't allow for composability.
IMO it would be better if each "integration" app registered itself on init, similar to how the django apps already work. It could even be an extended version of AppConfig with metadata relevant for integrations.

I like the idea but not the implementation, in particular the hardcoded values in IntegrationType - it doesn't allow for composability.
IMO it would be better if each "integration" app registered itself on init, similar to how the django apps already work. It could even be an extended version of AppConfig with metadata relevant for integrations.

IntegrationType is not an important field, we can rely on the child name

I was thinking to have an extended Integration model part of new applications (if required) and match the application particularities with an extended list of fields. For instance, a new module named django-github, meant to sync data from Github, will extend the Integration model to match Github particularities and actions. Therefore, the credentials will be different for Github compared to other services (e.g. AWS) which requires new fields to be added

I like the idea but not the implementation, in particular the hardcoded values in IntegrationType - it doesn't allow for composability.
IMO it would be better if each "integration" app registered itself on init, similar to how the django apps already work. It could even be an extended version of AppConfig with metadata relevant for integrations.

With this proposal I would like to dynamically manage the integrations at application level at any time after deployment not as an integration config at init phase. I'm trying to move away from the current approach which defines integration credentials as env vars to a more dynamic way of creating and managing integrations via UI

As an example, https://github.com/surface-security/django-github uses Integration model as part of the testapp

somebody has a bit of free time to review this PR?