Emoji (CKEditor & Other)

The mobile experience includes emoji everywhere. AFAIK it's now part of unicode and is effectively standardised worldwide.
I would like it if we could get emoji natively supported in Concrete5-wide, to further increase the mobile quality experience with it.
I know that initially this is likely to include CKEditor 4.10.0 (which I think needs some addons to get Emoji support BTW), but if we could get support elsewhere that CKEditor isn't involved, I think that would be worthwhile in the long-term too.
So, I'm hoping we can start the ball rolling on that getting implemented. Maybe 8.5? ;)
Please? :D

There has to be a ckeditor plug-in for this isn’t there? This doesn’t need to be something we support ourselves surely

@aembler Well, I think perhaps you misread what I was getting at, check the release notes for CKEditor 4.10.0 -> https://ckeditor.com/blog/CKEditor-4.10-released/
The idea is we get c5 to use CKEditor with Emoji support, so, as far as I can tell, work's already done ;)

It might not be as straightforward as you think.
I worked on a social feed plugin recently grabbing data from Facebook, Twitter, and Instagram and I kept getting errors when saving the data to the DB if it contained emojis. Turns out those characters won't save as is, you have to turn them into a Unicode "alias". For instance, for ♫ you will have to save "\u{266b}" in the DB. This means having a huge array of all possible emojis codes which is not necessarily possible as you have the official ones and that some that are specific to the different social networks...
For instance, if you check @Mesuva's Instagram package in the marketplace, it strips all emojis from posts before saving them to the DB because of that.

It might not be as straightforward as you think.
I worked on a social feed plugin recently grabbing data from Facebook, Twitter, and Instagram and I kept getting errors when saving the data to the DB if it contained emojis. Turns out those characters won't save as is, you have to turn them into a Unicode "alias". For instance, for ♫ you will have to save "\u{266b}" in the DB. This means having a huge array of all possible emojis codes which is not necessarily possible as you have the official ones and that some that are specific to the different social networks...
For instance, if you check @Mesuva's Instagram package in the marketplace, it strips all emojis from posts before saving them to the DB because of that.

Perhaps just make it compliant with worldwide unicode emoji only? At least as a baseline... Since it is standardised and all...

Any chance we could get a milestone attached to this please?

@BloodyIron
This branch is closed, but you can still check it out and test it.
https://github.com/concrete5/concrete5/pull/6993

@MrKarlDilkington hmmm lost sight of that, thanks! :D

Hmm, I've been advised by @mlocati to watch this, #6968 for the emoji stuff. Which I'm hoping we can get solved!
He also pointed out the database thing may be resolved in #7158 which I think is the blocker for emoji stuff?

This can be closed, isn't it?

LOL yup!


Broken CSS reference

The CSS file
'components/eonasdan-bootstrap-datetimepicker/build/css/bootstrap-datetimepicker-standalone.css'
references a file which could not be found:
components/eonasdan-bootstrap-datetimepicker/build/fonts/glyphicons-halflings-regular.eot
There is a link to a fonts directory that does not exist!
This happens via '../fonts'.
Please look into it and tell me whether you want the reference removed the the font added.
(then I can create a PR if you wish)
(At the moments this breaks my building process)
Your judgement is appreciated. ;)

unfortunately the standalone css assumes you have a folder where your css lives /css and the folder where your font is is one level above /fonts.  I'm not sure there's a good way to make that work for everyone in ever case since we're not including the fonts.
BS4 has dropped the glyphicons anyway.

I agree.
I am already using BS4 on one project where I also use your datepicker.
Would it be okay to add a comment in the docs for the standalone users telling them to include the fonts?
In any case I think it is good practise not to include broken links.
Until then I will use my patched version.
Thanks for the reply Jonathan.

I have a pull request open to remove the references to these files, which fixed the problem for me.

I hope this gets merged in soon; installing eonasdan-bootstrap-datetimepicker via bower in Django breaks things on ./manage.py collectstatic because Django attempts to resolve file references and fails.

I m having the same issue on Django Bower
whitenoise.storage.MissingFileError: The file 'eonasdan-bootstrap-datetimepicker/build/fonts/glyphicons-halflings-regular.eot' could not be found with <whitenoise.storage.CompressedManifestStaticFilesStorage object at 0x1061cb080>.

The CSS file 'eonasdan-bootstrap-datetimepicker/build/css/bootstrap-datetimepicker-standalone.css' references a file which could not be found:
  eonasdan-bootstrap-datetimepicker/build/fonts/glyphicons-halflings-regular.eot

Please check the URL references in this CSS file, particularly any
relative paths which might be pointing to the wrong location.

Sorry to dig this up but it seems the issue hasnt been tackled yet, right?

@natecox Your PR #1733 got closed because its not targeting dev. Could you change that?


Urgent issue, device completely stuck when resuming player from different activity with different orientation

Hi Tcking,
I added admob ad view to the player, the problem is when user click on Ad while player in fullsceen and Ad open something like Chrome for example in portrait, when user click back button to go back to player, the device completely stuck, buttons stop working, user cant event turn off device, at same time, user can hear the audio in player working, but can't see player,
This also happens if player in portrait, and user open Ad in chrome for example, and change orientation to landscape, then click back button same problem happens,
The app does not throw any error, so i could not provide more about the issue,
Thank you

I solved the issue by adding ".setPortraitWhenFullScreen(false)"


Delete redundant backquotes

remove extra space
fix a typo

fix my own typo

@mh-dm it's so nice of you, noticed that it's git pull --rebase origin development, not git pull --rebase upstream development, why?

^ Commit message/description wasn't updated?

^ Commit message/description wasn't updated?

QwQ ,I used amend option and then force push, it looks well locally. Next time I will double check.


Comments in Markdown support

Would be nice to support comments that do not render.
e.g. Obsidian support this:

Flowershow currently does not but supports MDX comments (?):

Given we have MDX you could just use html/jsx commenting which mdx supports i believe see e.g. https://github.com/mdx-js/mdx/issues/184
My suggestion here is we just add a page to document this e.g. docs/comments
Acceptance

[ ] docs page on comments e.g. docs/comments with example/test

Tasks

[ ] Test that MDX supported comments are indeed supported e.g. html or jsx style
[ ] Put up a docs page

Given we have MDX you could just use html/jsx commenting which mdx supports i believe see e.g. https://github.com/mdx-js/mdx/issues/184
My suggestion here is we just add a page to document this e.g. docs/comments

@rufuspollock @olayway Can you guide me ? I want to contribute. I am new to Opensource

@sahilm-10 what do you understand of the issue for yourself?

@sahilm-10 yes i think the start is to research docs support in mdx and then add a page docs/comments to flowershow.app exclaiming how to add comments to pages ...


WordPressLogin::check

Does the function "WordPressLogin::check" exist at all? Cannot find it anywhere.

If you're referring to the example in this plugin's documentation (http://octobercms.com/plugin/rainlab-user), then no, it doesn't. It's just an example, you would have to create that class and method yourself.


the re-train checkpoints

When can you provide the new training checkpoints?

@woaixueyingying Thanks for your interest.
We will update this by the end of May of early June since I'm busy recently.

OK,thanks.I am looking forward to it.

@woaixueyingying Thanks. Once updated, we will let you know.

@woaixueyingying Thanks for your patience. The updated checkpoints are updated. Mainly two updates: 1) fix a bug mask.sum to sim.sum; 2) conv1x1 to Linear implementation.

Trank you

Closed. Please feel free to re-open it if any further questions or concerns.


Unable to start VM after reboot, "User 'lxd' does not exist" error (but it does exist on the host)

Required information

Distribution: Ubuntu
Distribution version: 18.04
The output of "lxc info":

  driver: lxc
  driver_version: 4.0.6
  firewall: xtables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    netnsid_getifaddrs: "false"
    seccomp_listener: "false"
    seccomp_listener_continue: "false"
    shiftfs: "false"
    uevent_injection: "false"
    unpriv_fscaps: "true"
  kernel_version: 4.15.0-135-generic
  lxc_features:
    cgroup2: "true"
    devpts_fd: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "18.04"
  project: default
  server: lxd
  server_clustered: false
  server_name: opcube
  server_pid: 2372
  server_version: 4.0.4
  storage: zfs
  storage_version: 0.7.5-1ubuntu16.10

Issue description
Unable to start VM after reboot. Recently upgraded nvidia dkms drivers, but not sure if that's related.
$ lxc start wgdev
Error: Failed to run: forklimits limit=memlock:unlimited:unlimited -- /snap/lxd/19032/bin/qemu-system-x86_64 -S -name wgdev -uuid 528c2bf8-9eff-4024-85a4-bcc5b041834e -daemonize -cpu host -nographic -serial chardev:console -nodefaults -no-reboot -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=deny,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/wgdev/qemu.conf -pidfile /var/snap/lxd/common/lxd/logs/wgdev/qemu.pid -D /var/snap/lxd/common/lxd/logs/wgdev/qemu.log -chroot /var/snap/lxd/common/lxd/virtual-machines/wgdev -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd: qemu-system-x86_64: -runas lxd: User "lxd" doesn't exist (and is not :)
: Process exited with a non-zero value
Try lxc info --show-log wgdev for more info
Despite the error message, there is a lxd user and group.
/etc/passwd:lxd:x:105:65534::/var/lib/lxd/:/bin/false
/etc/group:lxd:x:108:op

Information to attach

[X] Any relevant kernel output (dmesg)

Problem with apparmor?
[  430.940055] audit: type=1400 audit(1612036089.523:103): apparmor="DENIED" operation="open" profile="lxd-wgdev_</var/snap/lxd/common/lxd>" name="/var/lib/snapd/hostfs/etc/passwd" pid=5903 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

[X] Container log (lxc info NAME --show-log)

$ lxc info wgdev --show-log
Name: wgdev
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/09/22 21:49 UTC
Status: Stopped
Type: virtual-machine
Profiles: default
Error: open /var/snap/lxd/common/lxd/logs/wgdev/qemu.log: no such file or directory

[X] Container configuration (lxc config show NAME --expanded)

$ lxc config show wgdev --expanded
architecture: x86_64
config:
  boot.autostart: "true"
  image.architecture: amd64
  image.description: Ubuntu bionic amd64 (20200922_07:42)
  image.os: Ubuntu
  image.release: bionic
  image.serial: "20200922_07:42"
  image.type: disk-kvm.img
  limits.cpu: "12"
  limits.memory: 20GiB
  volatile.base_image: 51f2de40ed9c48808327b57daf51354fbfc172114feb4814c71d52f2d5fd1449
  volatile.eth0.hwaddr: 00:16:3e:3f:de:6a
  volatile.last_state.power: STOPPED
  volatile.vm.uuid: 528c2bf8-9eff-4024-85a4-bcc5b041834e
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    size: 32GB
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

[X] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
Same error as when trying to start the VM. Not sure what the unsafe cache I/O error is.

t=2021-01-30T19:43:52+0000 lvl=warn msg="Using unsafe cache I/O with /dev/zd0"
t=2021-01-30T19:43:53+0000 lvl=eror msg="Failed to start instance 'wgdev': Failed to run: forklimits limit=memlock:unlimited:unlimited -- /snap/lxd/19032/bin/qemu-system-x86_64 -S -name wgdev -uuid 528c2bf8-9eff-4024-85a4-bcc5b041834e -daemonize -cpu host -nographic -serial chardev:console -nodefaults -no-reboot -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=deny,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/wgdev/qemu.conf -pidfile /var/snap/lxd/common/lxd/logs/wgdev/qemu.pid -D /var/snap/lxd/common/lxd/logs/wgdev/qemu.log -chroot /var/snap/lxd/common/lxd/virtual-machines/wgdev -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd: qemu-system-x86_64: -runas lxd: User "lxd" doesn't exist (and is not :)\n: Process exited with a non-zero value"
t=2021-01-30T19:44:04+0000 lvl=warn msg="Using unsafe cache I/O with /dev/zd0"
t=2021-01-30T19:45:12+0000 lvl=warn msg="Using unsafe cache I/O with /dev/zd0"
t=2021-01-30T19:48:09+0000 lvl=warn msg="Using unsafe cache I/O with /dev/zd0"

[ ] Output of the client with --debug
[ ] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

I believe we have a fix for this already and it will be rolled out to everyone on Monday.
You could test it with snap refresh lxd --channel=4.0/candidate though if you do that, note that you should then do snap refresh lxd --channel=4.0/stable later in the week so you don't keep getting the candidate builds (but you'll have to wait for it the current candidate to hit stable first).

I believe we have a fix for this already and it will be rolled out to everyone on Monday.
You could test it with snap refresh lxd --channel=4.0/candidate though if you do that, note that you should then do snap refresh lxd --channel=4.0/stable later in the week so you don't keep getting the candidate builds (but you'll have to wait for it the current candidate to hit stable first).

As an alternative you could set raw.apparmor on the instance to /var/lib/snapd/hostfs/etc/passwd r,

As an alternative you could set raw.apparmor on the instance to /var/lib/snapd/hostfs/etc/passwd r,

The candidate version fixed the problem, thanks!

The candidate version fixed the problem, thanks!


will the code be available in the near future?

will the code be available in the near future?

We will release the code after paper acceptance.


Add PHP 8.2 to CI

and the latest php-cs-fixer wants to change the order of some PHPdoc comments, so let it do it.

https://github.com/sabre-io/xml/runs/8069173233?check_suite_focus=true
phpunit OK (111 tests, 141 assertions) with PHP  8.2 - good.


TypeError: undefined is not an object (evaluating 'process.version.split')

After upgrading the RN to 0.62.2 then it failed and then shows the above evaluating 'process.version.split'.
TypeError: undefined is not an object (evaluating 'process.version.split')

Any update or action here? Seeing the same issue.


Track speaker package for conferences

Thanks @mathiasverraes

covers travel cost
travel cost max costs
covers hotel
covers food
other stuff?

This issue was moved to tightenco/symposium#26

 Track speaker package for conferences

Closed by #413


FiltersAggregationDescriptor doesn't support descriptors to specify filters

Elastic.Clients.Elasticsearch version: 8.1.1
Elasticsearch version: 8.8.0
.NET runtime version: .NET 7
Operating system version: Windows 11
Description of the problem including expected versus actual behavior:
There is no overload of FiltersAggregationDescriptor<>.Filters() that supports descriptors.
I see this as a bug, because the entire codebase is successfully using descriptors from the point of connecting to the server to the point of getting a response, the entire flow is working with QueryDescriptor<>, but this class doesn't provide any way to utilize descriptors as filters.
My parameters are all coming in as descriptors and there is no way to provide them to the Filters method, as it only accepts Buckets<Query>.
Previously, using NEST, there was a Filter, instead of Filters method, which was utilized like this:
filterAggregation.Filter(f => firstQuery & secondQuery). Since there is no way of combining queries like this, as far as I'm aware, it should be able to work with multiple descriptors, ultimately achieving the same goal.
Steps to reproduce:

Create new SearchRequestDescriptor<T>()
Use the Aggregations method, which exposes an AggregationDescriptor<>
Use Filters method, which exposes FiltersAggregationDescriptor<>
There is no way to pass descriptors to  FiltersAggregationDescriptor<>.Filters() method

Expected behavior
The class FiltersAggregationDescriptor should have an overload for the Filters method, which either accepts params Action<QueryDescriptor<TDocument>>[] configure or any kind of collection of descriptors, Buckets will also work as long as it accepts QueryDescriptor.

Are there some intentions of fixing this soon? Really needing it :/

Related to: Add fluent descriptor support for union types.


Error trying to open generated scene in Unity.

I tried to follow this guide to view the generated scene in Unity.
Here is what I have in Unity once I open the ai2thor project:

Then, I attempted to run python connect_to_unity.py --scene a_living_room-2024-04-06-11-58-33-174468/a_living_room.json but encountered the following errors.
Any suggestions?
Traceback (most recent call last):
  File "/Users/zhao/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/wsgi_server.py", line 48, in queue_get
    res = que.get(block=True, timeout=queue_get_timeout_per_try)
  File "/Users/zhao/miniconda3/envs/holodeck/lib/python3.9/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zhao/Documents/Holodeck/connect_to_unity.py", line 14, in <module>
    controller = Controller(
  File "/Users/zhao/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/controller.py", line 545, in __init__
    self.start(
  File "/Users/zhao/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/controller.py", line 1548, in start
    self.last_event = self.server.receive(timeout=self.server_start_timeout)
  File "/Users/zhao/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/wsgi_server.py", line 281, in receive
    return queue_get(
  File "/Users/zhao/miniconda3/envs/holodeck/lib/python3.9/site-packages/ai2thor/wsgi_server.py", line 62, in queue_get
    raise TimeoutError(
TimeoutError: Could not get a message from the queue after 600 attempts

got this work now

got this work now

Hello, may I ask how you resolved this issue? I am currently encountering the same error.

Hi, I followed the instructions at https://github.com/allenai/Holodeck#load-the-scene-in-unity closely and managed to get it working. I forgot to do step 3 in my first try.
Zhening

Hi, I followed the instructions at https://github.com/allenai/Holodeck#load-the-scene-in-unity closely and managed to get it working. I forgot to do step 3 in my first try.
Zhening

Thank you for your reply. I have completed everything as instructed, but I am still encountering the error mentioned above.

Hi, I was using a Mac and had no problem following the instructions given by the author. Maybe @ the author team for support?


ui-code component and refactored demo module

:tada: This PR is included in version 1.12.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


replace set-output to use GITHUB_OUTPUT

This is #1320 but for the 1.x branch

You can see the warnings here: https://github.com/apple/swift-protobuf/actions/runs/3957089680


Upgrade install process to install V6 agent.

Upgrade default.rb to install v6 datadog-agent.

@pocket7878 Thank you!
I will release v0.1.2 soon that includes this PR and some fixes.

@pocket7878 Your changes are released in v0.2.0.
Thank you for your contributing!!


append forward slash

Signed-off-by: Dinesh<EMAIL_ADDRESS>Pull Request Template
Description
Please include a summary of the change. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue).

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Additional Context (Please include any Screenshots/gifs if relevant)
...
Checklist:

[x] My code follows the style guidelines of this project.
[x] I have performed a self-review of my own code.

@nehagup review this and merge

LGTM


Replace toml v1 library to v2

Toml v2 is almost stable and is a massive improve in terms of performance and stability over the v1.

#11


Optimize for thousands of subscribers

Hello,
If I flush() more than 200 notifications at a time,  the push fails. Also,  each batch of 200 messages needs over 10 seconds to get sent to endpoints, so for 20k subscribers i would need 20 ~min.
Can this be optimized? What If i filter Chrome endpoints and Firefox endpoints so that each batch has same endpoint?
Did anyone managed to send more than 200 notifications at a time?
I am using php 7.2

Hi,
Sending bulk webpushes heavily uses network and CPU.
Be sure to use latest version of the library (4.x versions)
Be sure to increase memory limit to at least 500 MB. ini_set('memory_limit', '500M');
Be sure to decrease timeout to 6 seconds and disable ssl verify using $push = new WebPush($authenticationData,[],6,['verify'=>false]);
I think you can send 500 webpushes in 5-7 seconds by doing the simple things above. Of course there are many things (like storing and reusing local keys, etc..) that you can do but you have to deep dive to the code to make advanced optimizations.
BTW We send around 1 million webpushes/minute using this library (and lots of servers ) but we did many optimizations to do that.

BTW We send around 1 million webpushes/minute using this library (and lots of servers ) but we did many optimizations to do that.

Could you please share those? At least like a list of things you did above.

Hi,
We use 45 servers (aws t2.micro 1GB RAM) to send WebPushes. We send 500 webpushes in one flush() function and it takes 1 second to send 500 webpushes (if there is no network issue).

Creating Local keys and shared secret gets ~%90 of total process. So when we get new subscription we store Local Keys and shared secret in db. And we change it daily via cron job.

We changed subscription object to get local keys and shared secret . If these fields are empty we create local keys and shared secret like this library does. If subscription object has these fields we use them directly.

Also we cache VAPID headers for each endpoint url path and in one flush call (500 webpushes) we create only a few VAPID headers for each endpoint path in one flush call. (we don't create 500 VAPID headers.)

We use beanstalkd queue system (db queue is slow for large operations)

As I know using same local keys and shared secrets are not recommended by protocol. So we change local keys and shared secrets daily for each subscription.
These are the main things we did to make sending process faster. I hope it helps. Please let me know if you need any extra information.

Great 👍  but 47 seconds is too long as well. I think there must be any other factor on your server. Can you check the things below :

Some Intel CPU's has special thing for encryption (Intel AES New Instructions (AES-NI)). Please check your server if it's support AES-NI
You should look at which process takes how many seconds. (It's been long time but as I remember there are some functions like prepare, encrypt, flush) please look at which function takes how many seconds. I think this is the fastest method that you can find the bottle neck.
Payload size is also important because it makes encryption process slower. We generally use t for title, b for body, etc...  for JSON payload keys. We also don't send image full path we send only file name and add domain name and path in service worker.
Network speed is another important factor. Probably you get many data  from db, send webpushes and delete expired tokens. In this case you have 3 major network operations. Try to seperate them and  check which one takes how many seconds.
Did you decreased guzzle timeout to 6 seconds and set ssl verify to false ? ($push = new WebPush($authenticationData,[],6,['verify'=>false]); ) Library's default timeout is 20 sec. and if any request has problem you have to wait for 20 seconds. Try to decrease it. We use 6 sec. for 500 pushes. But you should find your optimum number.
If it's easy for you,  you can create a new account on AWS for testing. We use t2.micro servers with 1GB RAM and we can send 500 pushes in one second. (it's around 10 $ in a month (If you are a new user it's free for 12 months) you can create new server try sending pushes and close the server. )

Please let me know if these advices works for you.

And the last one :  try to disable auto padding if it's enabled.

@ozgurhangisi, thanks for the answer. I will try it.

@ozgurhangisi Do you have issues with 100% cpu usage ? https://github.com/guzzle/guzzle/issues/756
Also, do you suggest to get a powerful server or split the load in many machines like you? Currently I have a php script and it runs in one server

No I don't have any CPU issue. I prefer to use many small servers instead of 1 powerful server. I use cloud servers and if I want to get powerful server I should spend 400-500 $ in a month. So I can get 40 small servers for this price. If you use 1 server and if there is a problem on the server you can not send webpushes. So if you have many you wouldn't have this problem.

okay I also have to refactor my php script to support multiple process

@ozgurhangisi  Hey does beanstalkd queues allow for example having 2 or more servers consuming the queue ? Like server 1 pop random item, server 2 pop random? I have something with redis and enqueue-dev but the queue is only consumed from the top and that blocks the execution. I might need to use pure redis maybe

Yes it does support multi consumers. we use a few producers and 45 consumers  on beanstalkd. (sometimes we do some critical changes and old and new system has to work at the same time. We use up to 90 consumers at this point. I don't know beanstalkd's limit but it can support 90 consumers at least. And It's enough for us.). As I know it has FIFO rule. It doesn't give you random item. It gives you first waiting item on the queue.

@ozgurhangisi thanks so much for the information. Also do you use php consumers? Do you use something like supervisord to manage the consumers on the servers?

We use crontab and each cron works in every minute and check the queue. All of our servers are php producers and php consumers.
We developed special cron class and we can disable/enable or slow down each server. Also we add last cron status, last startdate, last end date, memory usage, cpu usage to the db and watch them. If something is wrong system send us an automated email.

Thanks,
Do you get the time sending the batches from the cron class stats?

@ozgurhangisi, Did you get better performance with the version 5 or with 4?
@t1gor Version 5 sends the message when I'm iterating over the result... Is this a performance improvement? What If I don't want to iterate over the results. My messages will not be send there.

@fpilee

Version 5 sends the message when I'm iterating over the result... Is this a performance improvement?

In my opinion, there are 2 answers here:

Yes. the \Generator object was introduced as an attempt to save some memory while iterating on a large sets of messages.

As a side effect, which was not intentional BTW, the flush() wouldn't do anything until you start iterating over the results. @Minishlink seems like this is a bug.

What If I don't want to iterate over the results.

You probably want to iterate over the results to clear the expired subs, as mentioned above.

My messages will not be send?

That is something I need to confirm. The way we use the lib in our project, the v5 vs v5 didn't change much, except for the results format, as we are checking the results. If that is true, I guess doing a simple solution like below should hot-fix it, while we're working on a proper solution.
$flushResults = $webpush->flush();
// do not save or process, just loop
iterator_to_array($flushResults);

Thanks for the answer
As the messages are sent in the results foreach code(v5),
v5 is slower (right now ) than v4 right now?

is v5 slower (right now ) than v4 ?

Do you have a benchmark? I don't think this should be relevant as only the place of actually sending requests is changed, not the payload size or smth.

Is 38 notifications sent by second low? To send 80.000 notifications in batches of 200 in a php script it takes 36 minutes approximated. Version 4 timeout set to 6 seconds, verify ssl off, batches of 200. 4 core server 8gb ram

It takes 99 seconds to send 200 web push with the version 5. The server where I ran have xdebug enabled

hey @ozgurhangisi
I can send only 25 notifications by second by server. Are 1core/1gb server on vultr.
$webPush = new WebPush($auth, $defaultOptions,12,['verify'=>false]);
What is this?

Creating Local keys and shared secret gets ~%90 of total process. So when we get new subscription we store Local Keys and shared secret in db. And we change it daily via cron job.
We changed subscription object to get local keys and shared secret . If these fields are empty we create local keys and shared secret like this library does. If subscription object has these fields we use them directly.

Is related to the user subscription endpoint public key /private that we get on subscription event?

It's not the data that you get on subscription. It's the data that the library creates when you send webpushes.
If you have thousands of customers and want to send webpushes so fast I recommend you to send it without payload and you can get payload in service worker. In this way you can send 1000 webpush in under a second and it's gonna be so fast.
We send webpushes with payloads but we spend about 6 months, we developed our own encryption library and we did many things to reach this speed and we use many servers queue systems, etc..
Sending webpushes with payload (to many users) spends so much cpu, ram and network resources. (Because library has to encrypt payload for each subscriber and has to send one by one to each subscriber. There is no bulk sending option for webpush with payload.)
If you need to send your webpushes so fast I recommend you to send it without payload or to use a service provider.

To avoid performance issues use Firebase Cloud Messaging. At least you will avoid the payload encryption delay

Hello, I have similar problem (if not the same). I tried everything written here. I send 1000 notifications for 86-90 seconds with payload and without payload for about 70 seconds which I know that is too slow. I found out that the async sending through Guzzle takes up to 0.2 seconds. So what might be wrong? My server is quite fast as overall but this seems to take way too much time for an async guzzl request, am I right?

Hello, after the update of Minishlink things are way faster. However to make them even better I tried to start asynchronously a few processes. If I send all the notifications (about 30 000 now) at one process it takes about 10 minutes and no problems. If I try to send it in parallel processes some of them get this error:
PHP Fatal error:  Uncaught Error: Class 'GuzzleHttp\Psr7\Response' not found in /home/www/client2/web/tools/composer-php72/vendor/guzzlehttp/guzzle/src/Handler/EasyHandle.php:76
Stack trace:
#0 /home/www/client2/web/tools/composer-php72/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(545): GuzzleHttp\Handler\EasyHandle->createResponse()
#1 [internal function]: GuzzleHttp\Handler\CurlFactory->GuzzleHttp\Handler{closure}(Resource id #3086, '\r\n')
#2 /home/www/client2/web/tools/composer-php72/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(108): curl_multi_exec(Resource id #3089, 999)
#3 /home/www/client2/web/tools/composer-php72/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(125): GuzzleHttp\Handler\CurlMultiHandler->tick()
#4 /home/www/client2/web/tools/composer-php72/vendor/guzzlehttp/promises/src/Promise.php(246): GuzzleHttp\Handler\CurlMultiHandler->execute(true)
#5 /home/www/client2/web/tools/composer-php72/vendor/guzzl in /home/www/client2/web/tools/composer-php72/vendor/guzzlehttp/guzzle/src/Handler/EasyHandle.php on line 76

We use crontab and each cron works in every minute and check the queue. All of our servers are php producers and php consumers.
We developed special cron class and we can disable/enable or slow down each server. Also we add last cron status, last startdate, last end date, memory usage, cpu usage to the db and watch them. If something is wrong system send us an automated email.

do you want to share with us ?😄

Hello @ozgurhangisi
You talked about "Creating Local keys and shared secret gets ~%90 of total process. ".
Din you modify the Encryption class so it draws the local keys and shared secret from the DB or am I missing some parameters to send those from the WebPush object?
Thanks!

Hello
If I update local key daily,
Is it safe to use same local key for all subscribers ?
Or should I use different local keys for each subscribers ?

Hi, I am so new to push.
can anyone can explain this term in more details?
""Creating Local keys and shared secret gets ~%90 of total process. "."
Thanks

@agron2017 if you're trying to look smarter, it did not work. This lib does not run on page hits, at least it should not.

Related to #367 (implemented in #401)


revise finding the maximum energy of the label code

I think that updating 'probability' instance is missed while finding the maximum "energy" of the label

thanks for fixing this!


Issue with feature finding and proteomics_4d.config file on linux

Hi,
Thank you for Alphapept! It looks really good.
Im trying to get it running on a containerized linux environment but run into some issues with feature_finding.py. At line 1761 in feature_finding.py it executes the uff-cmdline2 tool, reading the proteomics_4d.config file. Something seems to go wrong here; it would not read the bool 0 or 1 properly. I did fix that by changing to false/true, respectivly.
Now, it doesnt read the floats properly;

Any idea how I could resolve this? Perhaps a .config file is slightly different between Windows and Linux environments? Or maybe eaiser to get in touch with the Bruker people?
Best,
Patrick

Hi Patrick,
Thanks for your kind words.
As for the Linux version, this is not heavily tested. We have a work-in-progress draft for containerization in docker for Bruker here.
As I don't have a Linux system at hand, I have just made a quick test on Ubuntu 20.04 LTS on WSL. I used our small Bruker IRT file for testing. When just calling the feature finder, it runs through but gets a permission error writing the file. I tested changing the access rights for the feature finder with chmod but was not successful. Unfortunately, I am not very proficient in Linux and didn't proceed with troubleshooting the access rights. When I call the feature finder in the terminal directly as sudo with the same arguments, it runs through. Therefore I would assume that the config file is okay.
From the bug, I could imagine that might be a parsing error (e.g., different settings for comma separator in system etc.).
Attached are some screenshots. I hope his helps

.


v2 ETA

Sorry, I hate raising tickets like this, I know you're all doing a wonderful job and this isn't a demand or anything, merely a question.
When do you think v2 will likely be ready? Or is there a beta release or anything I can try out?
The reason I ask is because I'm really keen to try out timber but am blocked with lack of twig namespace support which has been added to the v2 branch.
I'd just like to know if I can try this out on a new project I'm about to start or if it's still some time away?
Alternatively, I could re-do the PR above backported to v1?
Thanks for your time :)

Elephants do it 95 weeks, we supersede elephants!

@jenkoian thanks for your note! I appreciate that you recognize we're doing the best we can to find time and make this happen.
I really think that we will have 2.x out on by the end of June. We've got one last "feature" to complete and then we can start with last-step code cleanup and other testing/release steps.
In regards to re-doing a back-ported PR, I'd of course be happy to take a look — but I'm trying to be judicious with 1.x PRs to help manage merge conflicts. Is there a lot of code changes for this particular issue?

Thanks for the update @jarednova - don't want to distract from the excellent work you're doing so will hold off on a backporting PR 👍


phc2sys stucks in "Waiting for ptp4l..."

Hello, I have a host PC as follow with Intel i210 adapter:
@ubuntu:~$ uname -a
Linux ubuntu 5.4.0-48-generic #52~18.04.1-Ubuntu

I follow tsn-doc tutorial, setting-up vlan, disabling network manager for pointed device, disabling NTP, etc.
then using:
art@ubuntu:~$ sudo ptp4l -i enp1s0 -f configs/gPTP.cfg --step_threshold=1 -m
[sudo] password di art:
ptp4l[5663.757]: selected /dev/ptp0 as PTP clock
ptp4l[5663.788]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[5663.788]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[5663.788]: port 1: link down
ptp4l[5663.788]: port 1: LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[5663.819]: selected local clock 6cb311.fffe.523b81 as best master
ptp4l[5663.819]: port 1: assuming the grand master role
ptp4l[5692.030]: selected local clock 6cb311.fffe.523b81 as best master
ptp4l[5692.030]: port 1: assuming the grand master role

art@ubuntu:~$ sudo pmc -u -b 0 -t 1 "SET GRANDMASTER_SETTINGS_NP clockClass 248 clockAccuracy 0xfe offsetScaledLogVariance 0xffff currentUtcOffset 37 leap61 0 leap59 0 currentUtcOffsetValid 1 ptpTimescale 1 timeTraceable 1 frequencyTraceable 0 timeSource 0xa0"
sending: SET GRANDMASTER_SETTINGS_NP
	6cb311.fffe.523b81-0 seq 0 RESPONSE MANAGEMENT GRANDMASTER_SETTINGS_NP
		clockClass              248
		clockAccuracy           0xfe
		offsetScaledLogVariance 0xffff
		currentUtcOffset        37
		leap61                  0
		leap59                  0
		currentUtcOffsetValid   1
		ptpTimescale            1
		timeTraceable           1
		frequencyTraceable      0
		timeSource              0xa0
art@ubuntu:~$ sudo phc2sys -w -m -s enp1s0 -c CLOCK_REALTIME --step_threshold=1 –transportSpecific=1
phc2sys[1374.742]: Waiting for ptp4l...
phc2sys[1375.743]: Waiting for ptp4l...
phc2sys[1376.743]: Waiting for ptp4l...
...

phc2sys never syncs.
If I run
sudo ptp4l -i enp1s0  --step_threshold=1 -m
it seems to work, but clock_check  gives system is not syncronized:
phc-rt delta is not 37 sec !
phc-tai delta is greater than 50 usec !
TAI offset set in kernel is not correct !
Please verify ptp4l and phc config and restart them if necessary to synchronize the clocks !

Any hints, please?
Thanks a lot

Hi @elabbing,
Is the i210 interface up? Is it connected to something? What does ip link show enp1s0 show?
Another hint is that you can use -l [num] to make the output of phc2sys/ptp4l more verbose - 7 is the most verbose.

Thank you Ederson.
Here ip link output:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 6c:b3:11:52:3b:81 brd ff:ff:ff:ff:ff:ff
3: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 00:19:99:85:71:3a brd ff:ff:ff:ff:ff:ff
4: enp1s0.5@enp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 6c:b3:11:52:3b:81 brd ff:ff:ff:ff:ff:ff

Here ptp4l and phc2sys output, running in 2 different shell (terminal emulator):
sudo ptp4l -i enp1s0 -f configs/gPTP.cfg --step_threshold=1 -m
ptp4l[2762.729]: selected /dev/ptp0 as PTP clock
ptp4l[2762.768]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[2762.768]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[2762.768]: port 1: link down
ptp4l[2762.768]: port 1: LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[2762.796]: selected local clock 6cb311.fffe.523b81 as best master
ptp4l[2762.796]: port 1: assuming the grand master role

sudo phc2sys -w -m -s enp1s0 -c CLOCK_REALTIME --step_threshold=1 –transportSpecific=1 -l 7
phc2sys[3057.535]: config item (null).clock_servo is 0
phc2sys[3057.535]: config item (null).kernel_leap is 1
phc2sys[3057.535]: config item (null).sanity_freq_limit is 200000000
phc2sys[3057.535]: config item (null).pi_proportional_const is 0.700000
phc2sys[3057.535]: config item (null).pi_integral_const is 0.300000
phc2sys[3057.535]: config item (null).pi_proportional_scale is 0.000000
phc2sys[3057.535]: config item (null).pi_proportional_exponent is -0.300000
phc2sys[3057.535]: config item (null).pi_proportional_norm_max is 0.700000
phc2sys[3057.535]: config item (null).pi_integral_scale is 0.000000
phc2sys[3057.535]: config item (null).pi_integral_exponent is 0.400000
phc2sys[3057.535]: config item (null).pi_integral_norm_max is 0.300000
phc2sys[3057.535]: config item (null).step_threshold is 1.000000
phc2sys[3057.535]: config item (null).first_step_threshold is 0.000020
phc2sys[3057.535]: config item (null).max_frequency is 900000000
phc2sys[3057.535]: config item (null).servo_offset_threshold is 0
phc2sys[3057.535]: config item (null).servo_num_offset_values is 10
phc2sys[3057.535]: PI servo: sync interval 1.000 kp 0.700 ki 0.300000
phc2sys[3057.535]: ioctl PTP_SYS_OFFSET_PRECISE: Operation not supported
phc2sys[3057.535]: config item (null).pi_proportional_const is 0.700000
phc2sys[3057.535]: config item (null).pi_integral_const is 0.300000
phc2sys[3057.535]: config item (null).pi_proportional_scale is 0.000000
phc2sys[3057.535]: config item (null).pi_proportional_exponent is -0.300000
phc2sys[3057.535]: config item (null).pi_proportional_norm_max is 0.700000
phc2sys[3057.535]: config item (null).pi_integral_scale is 0.000000
phc2sys[3057.535]: config item (null).pi_integral_exponent is 0.400000
phc2sys[3057.535]: config item (null).pi_integral_norm_max is 0.300000
phc2sys[3057.535]: config item (null).step_threshold is 1.000000
phc2sys[3057.535]: config item (null).first_step_threshold is 0.000020
phc2sys[3057.536]: config item (null).max_frequency is 900000000
phc2sys[3057.536]: config item (null).servo_offset_threshold is 0
phc2sys[3057.536]: config item (null).servo_num_offset_values is 10
phc2sys[3057.536]: PI servo: sync interval 1.000 kp 0.700 ki 0.300000
phc2sys[3057.536]: config item (null).transportSpecific is 0
phc2sys[3057.536]: config item (null).domainNumber is 0
phc2sys[3057.536]: config item (null).uds_address is '/var/run/ptp4l'
phc2sys[3058.537]: Waiting for ptp4l...
phc2sys[3059.538]: Waiting for ptp4l...
phc2sys[3060.539]: Waiting for ptp4l...

I 'm following TSN-doc obviously.
Maybe it's mandatory using igb-avb driver, instead of igb standard driver?
Thanks a lot for your help

Hi @elabbing,
It appears that the network interface is actually down, from this:

enp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000

Is the interface connected to a switch or another machine? I'd suggest to connect it, so that ptp4l startup can complete and phc2sys connect to ptp4l.
And no, there's no need for the igb-avb driver.

I run some test with 2 PC linked back to back.
First test with any config file,
second one using config file gPTP.cfg.
Test without config file goes straight, but it needs link up in order to phc2sys starts to sync. After sync , if disconnect ethernet cable phc2sys seems to go on normally.
Test with gPTP.cfg doesn't work.
Here output, if could help to solve.

 Can you confirm which value is on configs/gPTP.cfg? The default should be 1, but as long as they agree (the config and the option passed to phc2sys), it should work.

Ok, many thanks.
gPTP.cfg I used had misconfiguration.
Now it works.

Now, another issue.
Running the sequence:
sudo tc qdisc add dev enp1s0 parent root handle 6666 mqprio num_tc 2 map 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 queues 1@0 1@1 hw 0
sudo tc qdisc replace dev enp1s0 parent 6666:2 handle 8888 cbs idleslope 3648 sendslope -996352 hicredit 12 locredit -113 offload 1

sudo tc qdisc add dev enp1s0 parent 8888:1 etf clockid CLOCK_TAI delta 500000 offload

sudo ptp4l -i enp1s0 -f configs/gPTP.cfg --step_threshold=1 -m

sudo pmc -u -b 0 -t 1 "SET GRANDMASTER_SETTINGS_NP clockClass 248 clockAccuracy 0xfe offsetScaledLogVariance 0xffff currentUtcOffset 37 leap61 0 leap59 0 currentUtcOffsetValid 1 ptpTimescale 1 timeTraceable 1 frequencyTraceable 0 timeSource 0xa0"
sudo phc2sys -w -m -s enp1s0 -c CLOCK_REALTIME --step_threshold=1 --transportSpecific=1

ptp4l output stucks in:
ptp4l[143990.630]: port 1: FAULTY to LISTENING on INIT_COMPLETE
ptp4l[143990.630]: port 1: received link status notification
ptp4l[143990.630]: interface index 2 is up
ptp4l[143991.577]: port 1: peer port id set to 089e01.fffe.55da81-1
ptp4l[143991.577]: send failed: 105 No buffer space available
ptp4l[143991.578]: port 1: send peer delay response failed
ptp4l[143991.578]: port 1: LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[143991.608]: waiting 2^{4} seconds to clear fault on port 1

and phc2sys stucks in:
phc2sys[143982.793]: config item (null).uds_address is '/var/run/ptp4l'
phc2sys[143983.797]: Waiting for ptp4l...
phc2sys[143984.798]: Waiting for ptp4l...
phc2sys[143985.799]: Waiting for ptp4l...

I hope you could help me.

Then I type:
sudo tc qdisc add dev enp1s0 parent 8888:1 etf clockid CLOCK_TAI delta 500000 offload
and ptp4l stucks,
but phc2sys continues to run.
Tricky for me...

Hi @elabbing,
To get more information about transportSpecific (and other configuration options), I'll refer you to the IEEE-1588 Standard.
About your second issue, what happens is that ptp4l can't send data through an ETF qdisc. To send data through ETF, an application needs to timestamp each packet with a desired launch time. As ptp4l doesn't do that, I'd suggest you to direct ptp4l traffic to another qdisc.

Hi @elabbing,
To get more information about transportSpecific (and other configuration options), I'll refer you to the IEEE-1588 Standard.

Ok, thanks.

About your second issue, what happens is that ptp4l can't send data through an ETF qdisc. To send data through ETF, an application needs to timestamp each packet with a desired launch time. As ptp4l doesn't do that, I'd suggest you to direct ptp4l traffic to another qdisc.

In TSN-doc is stated:
Three qdiscs are required to set up an AVB system: MQPRIO, CBS and ETF.
and further is exaplained as to set Time Synchronization through ptp4l
please, see https://tsn.readthedocs.io/avb.html#system-setup
Could you clarify that?

The AVB application - in our documentation, the one using  GStreamer AVTP plugin or Alsa AAF plugin - is the one that uses the CBS and ETF qdisc. Not ptp4l. In fact, ptp4l messages could be delayed by CBS - if it tries to send a message when credit is over, which is undesirable.
So, the documentation sets up the qdiscs for the application. You can verify that it defines Linux socket priority 3 as the one that will go over the TSN qdiscs (CBS and ETF). The applications are then instructed to also use socket priority 3. However, ptp4l doesn't use socket priority 3. So, it will be best effort.

The AVB application - in our documentation, the one using GStreamer AVTP plugin or Alsa AAF plugin - is the one that uses the >CBS and ETF qdisc. Not ptp4l. In fact, ptp4l messages could be delayed by CBS - if it tries to send a message when credit is >over, which is undesirable.

So, if I understand well, in TSN-doc is described a system set-up don't work for ptp4l and QDisc running at the same time?
In other word, if I set-up ptp4l first and then QDisc on the same host PC, this breaks the host clocks synchronization, LAN sync, or both ?
But, on the other hand, I need running both ptp4l and QDisc to build a TSB-AVB compliant system?

So, the documentation sets up the qdiscs for the application. You can verify that it defines Linux socket priority 3 as the one that >will go over the TSN qdiscs (CBS and ETF). The applications are then instructed to also use socket priority 3. However, ptp4l >doesn't use socket priority 3. So, it will be best effort.

For ALSA and GSreamer, in TSN-doc is reported priority set to 2 as follow:
pcm.aaf0 {
        type aaf
        ifname eth0.5
        addr 01:AA:AA:AA:AA:AA

                                       prio 2

        streamid AA:BB:CC:DD:EE:FF:000B
        mtt 50000
        time_uncertainty 1000
        frames_per_pdu 12
        ptime_tolerance 100
}

gst-launch-1.0 clockselect. \( clock-id=realtime \
    filesrc location=piano2.wav ! wavparse ! audioconvert ! \
    audiobuffersplit output-buffer-duration=12/48000 ! \
    avtpaafpay mtt=50000000 tu=1000000 streamid=0xAABBCCDDEEFF000B processing-deadline=0 ! \
    avtpsink ifname=eth0.5 address=01:AA:AA:AA:AA:AA

                                priority=2

processing-deadline=0 \)

It is reported as working configuration.
Maybe I fail to understand anything?
Many  thanks

Hi,
About the priority 2 vs 3: well spotted! I just thought of the video example (which uses priority 3).
About qdiscs vs ptp4l, they do work together. If you follow the examples, you should get everything working fine: ptp4l, TSN traffic (audio and/or video) and even some other best-effort traffic as well, try it! If you run into any issues, please ask!
I noticed that you use a slightly different MQPRIO configuration than that in the TSN-doc examples, so let's take a step back and look at MQPRIO configuration there - as is the one which deals with priorities. I'll try to break it down so that you can do the changes needed for your case in a more informed way. There, first command is:
sudo tc qdisc add dev eth0 parent root handle 6666 mqprio \
        num_tc 3 \
        map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
        queues 1@0 1@1 2@2 \
        hw 0

Taking a closer look at the map, which is basically an array from with 16 items, indexed from 0 to 15, we see that index 2 has value 1:

2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2

And index 3 has value 0:

2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2

Which means that priority 2 is mapped to traffic class 1, and priority 3 to traffic class 0. All other priorities (0-1, 4-15) are mapped to traffic class 2. Note that, if an application doesn't set the priority of a socket, it will be 0. While ptp4l can be configured to use a given socket priority, that is not done in the documentation, so it will use 0.
After you run the mqprio command, if you check the qdiscs for the NIC, you'll get something like:
$ tc qdisc show dev enp6s0
qdisc mqprio 6666: root  tc 3 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2
             queues:(0:0) (1:1) (2:3)
qdisc fq 0: parent 6666:4 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40.0ms
qdisc fq 0: parent 6666:3 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40.0ms
qdisc fq 0: parent 6666:2 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40.0ms
qdisc fq 0: parent 6666:1 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40.0ms

Note that there are 4 queues under the mqprio (note that their "parent" has the handle of the mqprio one). That's basically because the NIC I'm using has 4 hardware queues. A question that may arise here is: which traffic goes to each queue?
The answer is in the mqprio command:

queues 1@0 1@1 2@2

Think of this as an array whose size is the number of traffic queues (in our example, 3) in which each position there's a set of hardware queues. Something like:

queues [hw queues for traffic class 0] [hw queues for traffic class 1] [hw queues for traffic class 2]

And the x@y is the count@offset of the hardware queues that will be used for the traffic class. So, for traffic class 0, we have:

1@0

Meaning 1 hardware queue, at offset 0. Again, think of the 4 hardware queues as an array indexed from 0 to 3.
Next, for traffic class 1:

1@1

Meaning 1 hardware queue, at offset 1. Finally, for traffic class 2:

2@2

Meaning 2 hardware queues, starting at offset 2. So, in the end, from hardware queues perspective, the mapping is something like:

[Traffic class 0] [Traffic class 1] [Traffic class 2] [Traffic class 2]

So, back to the result of mqprio command, we have:
$ tc qdisc show dev enp6s0
qdisc mqprio 6666: root  tc 3 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2
             queues:(0:0) (1:1) (2:3)
qdisc fq 0: parent 6666:4 limit (...) # This queue is for traffic class 2
qdisc fq 0: parent 6666:3 limit (...) # This queue is for traffic class 2
qdisc fq 0: parent 6666:2 limit (...) # This queue is for traffic class 1
qdisc fq 0: parent 6666:1 limit (...) # This queue is for traffic class 0

Now that we got the MQPRIO side, let's go to CBS and ETF. Back in the qdiscs, you will note that the qdiscs for traffic class 0 and 1 are generic fq qdiscs. We don't want that, we want to use our TSN capable ones. So next step is:
sudo tc qdisc replace dev eth0 parent 6666:1 handle 7777 cbs \
        idleslope 98688 sendslope -901312 hicredit 153 locredit -1389 \
        offload 1

sudo tc qdisc replace dev eth0 parent 6666:2 handle 8888 cbs \
        idleslope 3648 sendslope -996352 hicredit 12 locredit -113 \
        offload 1

After running those, if we check the qdisics we'll have something like:
$ tc qdisc show dev enp6s0
qdisc mqprio 6666: root  tc 3 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2
             queues:(0:0) (1:1) (2:3)
qdisc cbs 8888: parent 6666:2 hicredit 12 locredit -113 sendslope -996352 idleslope 3648 offload 1
qdisc pfifo 0: parent 8888: limit 1000p
qdisc pfifo 0: parent 7777: limit 1000p
qdisc fq 0: parent 6666:4 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40.0ms
qdisc fq 0: parent 6666:3 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40.0ms
qdisc cbs 7777: parent 6666:1 hicredit 153 locredit -1389 sendslope -901312 idleslope 98688 offload 1

Wow, looks messy. But we can navigate through this. There are still the two fq related to traffic class 2:
(...)
qdisc fq 0: parent 6666:4 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40.0ms
qdisc fq 0: parent 6666:3 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40.0ms
(...)

There are the two CBS ones that we added:
(...)
qdisc cbs 8888: parent 6666:2 hicredit 12 locredit -113 sendslope -996352 idleslope 3648 offload 1
(...)
qdisc cbs 7777: parent 6666:1 hicredit 153 locredit -1389 sendslope -901312 idleslope 98688 offload 1

And finally, under each CBS, there's a pfifo qdisc (note their parents handle mach the CBS ones):
(...)
qdisc pfifo 0: parent 8888: limit 1000p
qdisc pfifo 0: parent 7777: limit 1000p
(...)

Now, packets that are sent via a socket with priority 2 will be mapped to traffic class 1. Traffic class 1 traffic go to the second hardware queue. But before getting there, it will pass through the CBS queue with handle 8888. Something similar happens to packets sent to a socket with priority 3, but those will end up going to the CBS queue with handle 7777. All other packets will go to either of the remaining fq qdiscs under MQPRIO.
So, in the documentation example, only the audio and video traffic will be sent through the TSN qdiscs. Packets generated by ptp4l will go to either of the remaining fq qdiscs under MQPRIO.
Note that in the example, those pfifo qdiscs will be replaced with ETF ones:
sudo tc qdisc add dev eth0 parent 7777:1 etf \
        clockid CLOCK_TAI \
        delta 500000 \
        offload

sudo tc qdisc add dev eth0 parent 8888:1 etf \
        clockid CLOCK_TAI \
        delta 500000 \
        offload

The final look of the qdiscs will be something like:
$ tc qdisc show dev enp6s0
qdisc mqprio 6666: root  tc 3 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2
             queues:(0:0) (1:1) (2:3)
qdisc cbs 8888: parent 6666:2 hicredit 12 locredit -113 sendslope -996352 idleslope 3648 offload 1
qdisc fq 0: parent 6666:4 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40.0ms
qdisc fq 0: parent 6666:3 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40.0ms
qdisc etf 804e: parent 8888:1 clockid TAI delta 500000 offload on deadline_mode off skip_sock_check off
qdisc cbs 7777: parent 6666:1 hicredit 153 locredit -1389 sendslope -901312 idleslope 98688 offload 1
qdisc etf 804d: parent 7777:1 clockid TAI delta 500000 offload on deadline_mode off skip_sock_check off

However, a packet that goes to an ETF qdisc needs to have a timestamp associated to it, indicating when it should be transmitted. Both the Gstreamer AVTP plugin as well as the Alsa AAF plugin take care of that, but ptp4l doesn't do that, so it can't send packets if an ETF qdisc is in the way.
With that, I hope you can tweak the configuration as needed - for instance, if your NIC has a different number of HW queues or so.

Thanks a lot, great reply!
Now, I take some time to study more deeply.

Hi @elabbing,
Can this issue be closed now?

yes, many thanks

Hello,
I got stuck with the same issue, the phc2sys stucks in "Waiting for ptp4l...", and I don't have a clue now what to change.
I tried to follow the guide, but I could make the qdisc part work with the following way:
sudo tc qdisc add dev enp2s0 parent root handle 6666 mqprio num_tc 2 map 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 queues 1@0 1@1 hw 0
sudo tc qdisc replace dev enp2s0 parent 6666:1 handle 8888 cbs idleslope 3648 sendslope -996352 hicredit 12 locredit -113 offload 1
sudo tc qdisc add dev enp2s0 parent 8888:1 etf clockid CLOCK_TAI delta 500000 offload
sudo ptp4l -i enp2s0 -f /usr/share/doc/linuxptp/gPTP.cfg --step_threshold=1 -m
sudo pmc -u -b 0 -t 1 "SET GRANDMASTER_SETTINGS_NP clockClass 248 clockAccuracy 0xfe offsetScaledLogVariance 0xffff currentUtcOffset 37 leap61 0 leap59 0 currentUtcOffsetValid 1 ptpTimescale 1 timeTraceable 1 frequencyTraceable 0 timeSource 0xa0"
sudo phc2sys -w -m -s enp2s0 -c CLOCK_REALTIME -S 1

I could not find a way to substitute the --transportSpecific=1 parameter in the last command, it seems to be an invalid argument.
My system is a Debian 10 with kernel 4.19.0-18-amd64 and would like to set up the ALSA streaming with 2 identical computers with Intel I210 controller. The num_tc 3 parameter did not work, so I changed it to num_tc 2 and the map to map 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1.
Any idea how to proceed from this point?
Thank you!

@petipok
It seems you are having issues on some previous steps: what did happen when you tried to use num_tc 3 for mqprio? Also, what is the linuxptp version in your systems? Trying with more verbose log doesn't yield more clues?

@edersondisouza
I got this with num_tc 3:
sudo tc qdisc add dev enp2s0 parent root handle 6666 mqprio num_tc 3 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@0 1@1 2@2 hw 0 RTNETLINK answers: Invalid argument

I have the following version of ptp4l:
sudo ptp4l -v 1.92


Create pt.json

add portuguese language in lokalise pls

Done!
Your translations will be included in next release of the integration.

@ViPeR5000 The portuguese translations are still empty in Lokalise. Are you interested in contributing?

@ViPeR5000 The portuguese translations are still empty in Lokalise. Are you interested in contributing?

yes but...No access to resource

Sorry, should work now.


Fix: broken rendering in create new scaler markdown file

Hi team,
This MR fixes the broken rendering in markdown file to create a new scaler. Let me know if I need to update CHANGELOG file if it is for Document changes
Checklist

[x] I have verified that my change is according to the deprecations & breaking changes policy
[x] Commits are signed with Developer Certificate of Origin (DCO - learn more)

Thanks @tomkerkhove ! It should be fixed now.


feat: Avoid crashing on unsopported platforms

Added a platform-agnostic entry point (except for Android/iOS) that will show a warning for unsupported platforms (instead of making them crash).

:tada: This PR is included in version 8.7.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


[SPARK-34607][SQL][2.4] Add Utils.isMemberClass to fix a malformed class name error on jdk8u

What changes were proposed in this pull request?

 In general, Scala generates a class name for a nested class by inserting the dollar-sign ( $ ) in between each level of class nesting. The problem is that this format can concatenate into a very long string that goes beyond certain limits, so Scala will change the class name format beyond certain length threshold.
For the example above, we can see that the first two levels of class nesting have class names that look like this:
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassName1$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassName1$OuterLevelWithVeryVeryVeryLongClassName2$

If we leave out the fact that Scala uses a dollar-sign ( $ ) suffix for the class name of the companion object, OuterLevelWithVeryVeryVeryLongClassName1's full name is a prefix (substring) of OuterLevelWithVeryVeryVeryLongClassName2.
But if we keep going deeper into the levels of nesting, you'll find names that look like:
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$2a1321b953c615695d7442b2adb1$$$$ryVeryLongClassName8$OuterLevelWithVeryVeryVeryLongClassName9$OuterLevelWithVeryVeryVeryLongClassName10$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$2a1321b953c615695d7442b2adb1$$$$ryVeryLongClassName8$OuterLevelWithVeryVeryVeryLongClassName9$OuterLevelWithVeryVeryVeryLongClassName10$OuterLevelWithVeryVeryVeryLongClassName11$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$85f068777e7ecf112afcbe997d461b$$$$VeryLongClassName11$OuterLevelWithVeryVeryVeryLongClassName12$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$85f068777e7ecf112afcbe997d461b$$$$VeryLongClassName11$OuterLevelWithVeryVeryVeryLongClassName12$OuterLevelWithVeryVeryVeryLongClassName13$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$85f068777e7ecf112afcbe997d461b$$$$VeryLongClassName11$OuterLevelWithVeryVeryVeryLongClassName12$OuterLevelWithVeryVeryVeryLongClassName13$OuterLevelWithVeryVeryVeryLongClassName14$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$5f7ad51804cb1be53938ea804699fa$$$$VeryLongClassName14$OuterLevelWithVeryVeryVeryLongClassName15$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$5f7ad51804cb1be53938ea804699fa$$$$VeryLongClassName14$OuterLevelWithVeryVeryVeryLongClassName15$OuterLevelWithVeryVeryVeryLongClassName16$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$5f7ad51804cb1be53938ea804699fa$$$$VeryLongClassName14$OuterLevelWithVeryVeryVeryLongClassName15$OuterLevelWithVeryVeryVeryLongClassName16$OuterLevelWithVeryVeryVeryLongClassName17$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$69b54f16b1965a31e88968df1a58d8$$$$VeryLongClassName17$OuterLevelWithVeryVeryVeryLongClassName18$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$69b54f16b1965a31e88968df1a58d8$$$$VeryLongClassName17$OuterLevelWithVeryVeryVeryLongClassName18$OuterLevelWithVeryVeryVeryLongClassName19$
org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassNam$$$$69b54f16b1965a31e88968df1a58d8$$$$VeryLongClassName17$OuterLevelWithVeryVeryVeryLongClassName18$OuterLevelWithVeryVeryVeryLongClassName19$OuterLevelWithVeryVeryVeryLongClassName20$

with a hash code in the middle and various levels of nesting omitted.
The java.lang.Class.isMemberClass method is implemented in JDK8u as:
http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/tip/src/share/classes/java/lang/Class.java#l1425
    /**
     * Returns {@code true} if and only if the underlying class
     * is a member class.
     *
     * @return {@code true} if and only if this class is a member class.
     * @since 1.5
     */
    public boolean isMemberClass() {
        return getSimpleBinaryName() != null && !isLocalOrAnonymousClass();
    }

    /**
     * Returns the "simple binary name" of the underlying class, i.e.,
     * the binary name without the leading enclosing class name.
     * Returns {@code null} if the underlying class is a top level
     * class.
     */
    private String getSimpleBinaryName() {
        Class<?> enclosingClass = getEnclosingClass();
        if (enclosingClass == null) // top level class
            return null;
        // Otherwise, strip the enclosing class' name
        try {
            return getName().substring(enclosingClass.getName().length());
        } catch (IndexOutOfBoundsException ex) {
            throw new InternalError("Malformed class name", ex);
        }
    }

and the problematic code is getName().substring(enclosingClass.getName().length()) -- if a class's enclosing class's full name is longer than the nested class's full name, this logic would end up going out of bounds.
The bug has been fixed in JDK9 by https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8057919 , but still exists in the latest JDK8u release. So from the Spark side we'd need to do something to avoid hitting this problem.
This is the backport of #31733.
Why are the changes needed?

Bugfix on jdk8u.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Added tests.

Test build #135782 has started for PR 31747 at commit d0d049c.

It seems the test failed and I'll check it.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40364/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40364/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40364/

See https://github.com/apache/spark/pull/31745#issuecomment-791100327 for the failure reason.

Test build #135782 has finished for PR 31747 at commit d0d049c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
   * inner class (See SPARK-34607 for details). This issue has already been fixed in jdk9+, so
       //getSimpleBinaryName() returns null if a given class is a top-level class,

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135782/

Could you make a PR to branch-2.4, @rednaxelafx ? Although we know that will fails, we can merge this PR and the original PR together.

Let me briefly describe the situation in a new comment in my original PR: https://github.com/apache/spark/pull/31709#issuecomment-791275810

I made a relevant test suite fix for this.

https://github.com/apache/spark/pull/31766

Should this not go into master + 3.x as well? or is it already fixed there?

@srowen . This is a backport of https://github.com/apache/spark/pull/31733 .
This has been blocked due to the UT failures of the relevant prior patches and the test suite bug ( https://github.com/apache/spark/pull/31766 ).

Jenkins retest this please

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135981/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40565/

Jenkins retest this please

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40593/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136009/

Is Jenkins unstable?

retest this please

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40608/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40608/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136024/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40608/

retest this please

I suppose this should be in 2.4.8, right? @maropu

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136045/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40628/

retest this please

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40663/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136080/

There is a test error in the GA:
[info]   Cause: java.lang.StringIndexOutOfBoundsException: String index out of range: -83
[info]   at java.lang.String.substring(String.java:1931)
[info]   at java.lang.Class.getSimpleBinaryName(Class.java:1448)
[info]   at java.lang.Class.getSimpleName(Class.java:1309)
[info]   at org.apache.spark.sql.catalyst.expressions.objects.NewInstance$$anonfun$11.apply(objects.scala:490)
...

I suppose this should be in 2.4.8, right? @maropu

Yea, I think so. I'll check the test failure.

I noticed that branch-2.4 does not have an interpreted implementation of SafeProjection, so the fallback cannot happen in some code path. For example, ExpressionEncoder directly uses GenerateSafeProjection (see the code below), so it throws an exception no matter what value we set to spark.sql.codegen.factoryMode.
https://github.com/apache/spark/blob/3c627ad6f6392b085e05475a7475f7471190d0a8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala#L271
I think we have two options;

give up this PR and #31709 in branch-2.4
merge this PR and #31709 into branch-2.4 for changing the thrown exception from JVM StringIndexOutOfBoundsException into a Janino compilation error

I feel there is no much difference between them for users though, the second one looks better cuz StringIndexOutOfBoundsException gives users no error context, I think. WDHY? @viirya @dongjoon-hyun @HyukjinKwon

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40675/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136092/

SQL tests seem all passed in GA? I only saw unrelated R install error. Triggered GA again.

retest this please

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40708/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136126/

Yea, in the latest commit, I modified the test itself and applied the #31709 fix so that the GA tests can pass. The modified test (checkCompilationError) just checks that the compilation fails with codegen enabled/disabled. If we merge this fix into branch-2.4, I think we need to apply the #31709 fix first, and then merge this in.

I think we have two options;
* give up this PR and #31709 in branch-2.4

* merge this PR and #31709 into branch-2.4 for changing the thrown exception from JVM `StringIndexOutOfBoundsException` into a Janino compilation error

I feel there is no much difference between them for users though, the second one looks better cuz StringIndexOutOfBoundsException gives users no error context, I think. WDHY? @viirya @dongjoon-hyun @HyukjinKwon

Hmm, so even with this and #31709, there is still compilation error? Do we know what the compilation error is?

Hmm, so even with this and #31709, there is still compilation error? Do we know what the compilation error is?

Yea, right. All the branches have the same compilation error if branch-2.4 has #31709 and this PR. The only difference is that master/branch-3.1/branch-3.2 can fall back into the interpreted one, but branch-2.4 does not.

Yea, right. All the branches have the same compilation error if branch-2.4 has #31709 and this PR. The only difference is that master/branch-3.1/branch-3.2 can fall back into the interpreted one, but branch-2.4 does not.

Okay, then a compilation error instead of mystical StringIndexOutOfBoundsException seems better.

I'm fine to have this and #31709 to branch-2.4.

Thanks for the check, @viirya ! okay, I'll open a new PR to backport #31709 first.

@maropu #31709 was merged. Can you sync up this? If no problem, I think we can merge this.

Sure.

done

retest this please

Test build #136484 has started for PR 31747 at commit a9c3188.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41068/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41068/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41068/

Test build #136484 has finished for PR 31747 at commit a9c3188.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136484/

Note that this is backport PR.
I will merge this to branch-2.4 tomorrow if no more comments.
cc @dongjoon-hyun @rednaxelafx @cloud-fan

Thanks all. Merging to branch-2.4.

Thanks for the reviews, @viirya @cloud-fan !


docker: fix listenOptions

Description of changes
listenOptions exists but doesn't take effect.
Things done
Pass the value of listenOptions to daemon.settings

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

This already works as intended. Same as upstream, we use systemd socket activation. Docker doesn't manage its own sockets at that point. The comment for listenOptions explains how to configure it.
See https://docs.docker.com/engine/reference/commandline/dockerd/ and links therein for a deeper explanation.

On Systemd based systems, you can communicate with the daemon via Systemd socket activation, use dockerd -H fd://. Using fd:// will work perfectly for most setups but you can also specify individual sockets: dockerd -H fd://3. If the specified socket activated files aren’t found, then Docker will exit. You can find examples of using Systemd socket activation with Docker and Systemd in the Docker source tree.