Can wifi be activiated ?

I know Jamulus works on Lan, but for whatever reason, we may want to operate the PI when there is only wifi accessible. Can it be built-in as well, or an option provided?
If I want to manually activate wifi, what would be the procedure?

Just access the desktop via the integrated VNC capability, and turn WiFi on
(assuming you are using an rpi4, btw)
If you are using a different model of pi, you may need a WiFi ‘hat’
On Thu, 14 Jan 2021 at 19:54, Goldy (Gordon<EMAIL_ADDRESS>wrote:

I know Jamulus works on Lan, but for whatever reason, we may want to
operate the PI when there is only wifi accessible. Can it be built-in as
well, or an option provided?
If I want to manually activate wifi, what would be the procedure?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/kdoren/jambox-pi-gen/issues/21, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXXXG6BNLFAW5SZVVVPDTTSZ5DWPANCNFSM4WC64XIQ
.

Just access the desktop via the integrated VNC capability, and turn WiFi on
(assuming you are using an rpi4, btw)
If you are using a different model of pi, you may need a WiFi ‘hat’
On Thu, 14 Jan 2021 at 19:54, Goldy (Gordon<EMAIL_ADDRESS>wrote:

I know Jamulus works on Lan, but for whatever reason, we may want to
operate the PI when there is only wifi accessible. Can it be built-in as
well, or an option provided?
If I want to manually activate wifi, what would be the procedure?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/kdoren/jambox-pi-gen/issues/21, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXXXG6BNLFAW5SZVVVPDTTSZ5DWPANCNFSM4WC64XIQ
.

How to turn wifi on? I always got "no wireless interface found". I am using rpi4 module B. I am assuming this module always comes with a wireless interface?

How to turn wifi on? I always got "no wireless interface found". I am using rpi4 module B. I am assuming this module always comes with a wireless interface?

Oh that's odd; As far as I'm aware that model (the same as mine) always
comes with Wifi - but I could be wrong!
Have you tried it using a standard copy of Raspian, just to see if it
behaves differently?
There are some useful tips here:
https://www.raspberrypi.org/forums/viewtopic.php?t=259530
I hope it helps; sounds like a PI issue rather than Jambox-pi...
On Thu, Jan 14, 2021 at 8:13 PM Goldy (Gordon<EMAIL_ADDRESS>wrote:

How to turn wifi on? I always got "no wireless interface found". I am
using rpi4 module B. I am assuming this module always comes with a wireless
interface?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kdoren/jambox-pi-gen/issues/21#issuecomment-760449155,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXXXGZZSWHW5RCKWLIQVD3SZ5F4XANCNFSM4WC64XIQ
.

Oh that's odd; As far as I'm aware that model (the same as mine) always
comes with Wifi - but I could be wrong!
Have you tried it using a standard copy of Raspian, just to see if it
behaves differently?
There are some useful tips here:
https://www.raspberrypi.org/forums/viewtopic.php?t=259530
I hope it helps; sounds like a PI issue rather than Jambox-pi...
On Thu, Jan 14, 2021 at 8:13 PM Goldy (Gordon<EMAIL_ADDRESS>wrote:

How to turn wifi on? I always got "no wireless interface found". I am
using rpi4 module B. I am assuming this module always comes with a wireless
interface?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kdoren/jambox-pi-gen/issues/21#issuecomment-760449155,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXXXGZZSWHW5RCKWLIQVD3SZ5F4XANCNFSM4WC64XIQ
.

Good to know. I will try the standard image.  From what I search all rsp4b should have built in wireless.
https://www.raspberrypi.org/documentation/faqs/#:~:text=The Raspberry Pi 4 Model,wireless LAN%2C and Bluetooth 5.0.

Good to know. I will try the standard image.  From what I search all rsp4b should have built in wireless.
https://www.raspberrypi.org/documentation/faqs/#:~:text=The Raspberry Pi 4 Model,wireless LAN%2C and Bluetooth 5.0.

I believe wireless is disabled in /boot/config.txt, you need to comment or
remove this line:
dtoverlay=disable-wifi
Reboot, then try enabling wifi.
Carlos
On Thu, Jan 14, 2021 at 12:25 PM Goldy (Gordon<EMAIL_ADDRESS>wrote:

Good to know. I will try the standard image. From what I search all rsp4b
should have built in wireless.
https://www.raspberrypi.org/documentation/faqs/#:~:text=The Raspberry Pi 4 Model,wireless LAN%2C and Bluetooth 5.0
.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/kdoren/jambox-pi-gen/issues/21#issuecomment-760455496,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKJTE3JT6YAM3YY23UC6QDSZ5HMXANCNFSM4WC64XIQ
.

I believe wireless is disabled in /boot/config.txt, you need to comment or
remove this line:
dtoverlay=disable-wifi
Reboot, then try enabling wifi.
Carlos
On Thu, Jan 14, 2021 at 12:25 PM Goldy (Gordon<EMAIL_ADDRESS>wrote:

Good to know. I will try the standard image. From what I search all rsp4b
should have built in wireless.
https://www.raspberrypi.org/documentation/faqs/#:~:text=The Raspberry Pi 4 Model,wireless LAN%2C and Bluetooth 5.0
.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/kdoren/jambox-pi-gen/issues/21#issuecomment-760455496,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKJTE3JT6YAM3YY23UC6QDSZ5HMXANCNFSM4WC64XIQ
.

@CAguayo  yes that works!  Thanks!  I have interface now though I can not connect yet.
I tried to search wifi or wireless in the whole repository but found nothing.  This is better documented.

@CAguayo  yes that works!  Thanks!  I have interface now though I can not connect yet.
I tried to search wifi or wireless in the whole repository but found nothing.  This is better documented.

There is an expectation that you would not want to use wireless, as that's
really bad for latency and jitter, so it's not surprising that you don't
see it mentioned in the source tree.  Here's a guide for enabling wifi from
the command line:
https://www.raspberrypi.org/documentation/configuration/wireless/wireless-cli.md
On Thu, Jan 14, 2021 at 12:45 PM Goldy (Gordon<EMAIL_ADDRESS>wrote:

@CAguayo https://github.com/CAguayo yes that works! Thanks! I have
interface now though I can not connect yet.
I tried to search wifi or wireless in the whole repository but found
nothing. This is better documented.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kdoren/jambox-pi-gen/issues/21#issuecomment-760465429,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKJTEZZ4YWESYYV3VWWGPDSZ5JYDANCNFSM4WC64XIQ
.

There is an expectation that you would not want to use wireless, as that's
really bad for latency and jitter, so it's not surprising that you don't
see it mentioned in the source tree.  Here's a guide for enabling wifi from
the command line:
https://www.raspberrypi.org/documentation/configuration/wireless/wireless-cli.md
On Thu, Jan 14, 2021 at 12:45 PM Goldy (Gordon<EMAIL_ADDRESS>wrote:

@CAguayo https://github.com/CAguayo yes that works! Thanks! I have
interface now though I can not connect yet.
I tried to search wifi or wireless in the whole repository but found
nothing. This is better documented.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kdoren/jambox-pi-gen/issues/21#issuecomment-760465429,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKJTEZZ4YWESYYV3VWWGPDSZ5JYDANCNFSM4WC64XIQ
.

The readme file did mentioned options to even auto connect certain wifi AP so I think there is a need to clarify how to activate it.  How is that config.txt configured in this GIt repository or is it default from some external source?

The readme file did mentioned options to even auto connect certain wifi AP so I think there is a need to clarify how to activate it.  How is that config.txt configured in this GIt repository or is it default from some external source?

@goldyliang I disabled wifi to keep people out of trouble; WiFi can add a lot of jitter so you're not likely to be happy using it.  (I've heard people say "I have gigabit internet, so I can use WiFi" but it doesn't work like that).
It's disabled on line 18 here when the image is generated:  https://github.com/kdoren/jambox-pi-gen/blob/jambox/stage3/04-install-sw/08-run.sh
However, if you want to use WiFI, as Carlos said, first you need to stop disabling in on boot:
sudo sed -i 's/^dtoverlay=disable-wifi/#dtoverlay=disable-wifi/' /boot/config.txt

Then, after first reboot, you need to set your country (it won't enable until you do that).
This is done by clicking the network up/down arrow icon on the taskbar at the very upper right.
Then after turning on WiFi, you can select a network and login.
You say you may want to use it where only WiFi is available.
However, in order log your Raspberry Pi into WiFi, you need to be connected to your Raspberry Pi.
So you would need to either:

hook a display and keyboard to you RPi so you can login to WiFI
Set it up so that you can get to the desktop via Bluetooth.  This is a bit tricky to set up; I've done it in the past but this seems to not be working now, not sure why.
Set the WFi login credentials in advance and hope that it works when you get there:

WiFi login credentials are set in the file "/etc/wpa_supplicant/wpa_supplicant.conf":
cttrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=US

network={
        ssid="<ssid>"
        psk="<password>"
        key_mgmt=WPA-PSK
}

@goldyliang I disabled wifi to keep people out of trouble; WiFi can add a lot of jitter so you're not likely to be happy using it.  (I've heard people say "I have gigabit internet, so I can use WiFi" but it doesn't work like that).
It's disabled on line 18 here when the image is generated:  https://github.com/kdoren/jambox-pi-gen/blob/jambox/stage3/04-install-sw/08-run.sh
However, if you want to use WiFI, as Carlos said, first you need to stop disabling in on boot:
sudo sed -i 's/^dtoverlay=disable-wifi/#dtoverlay=disable-wifi/' /boot/config.txt

Then, after first reboot, you need to set your country (it won't enable until you do that).
This is done by clicking the network up/down arrow icon on the taskbar at the very upper right.
Then after turning on WiFi, you can select a network and login.
You say you may want to use it where only WiFi is available.
However, in order log your Raspberry Pi into WiFi, you need to be connected to your Raspberry Pi.
So you would need to either:

hook a display and keyboard to you RPi so you can login to WiFI
Set it up so that you can get to the desktop via Bluetooth.  This is a bit tricky to set up; I've done it in the past but this seems to not be working now, not sure why.
Set the WFi login credentials in advance and hope that it works when you get there:

WiFi login credentials are set in the file "/etc/wpa_supplicant/wpa_supplicant.conf":
cttrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=US

network={
        ssid="<ssid>"
        psk="<password>"
        key_mgmt=WPA-PSK
}

I've actually wondered why we leave bluetooth enabled, maybe that should be
shut off unless it's needed as well.  At very least, it takes power to run,
and at worst, it introduces more background activity in a system that we
really want to be dedicated to audio processing.
Carlos
On Sat, Jan 16, 2021 at 10:56 AM Kevin Doren<EMAIL_ADDRESS>wrote:

@goldyliang https://github.com/goldyliang I disabled wifi to keep
people out of trouble; WiFi can add a lot of jitter so you're not likely to
be happy using it. (I've heard people say "I have gigabit internet, so I
can use WiFi" but it doesn't work like that).
It's disabled on line 18 here when the image is generated:
https://github.com/kdoren/jambox-pi-gen/blob/jambox/stage3/04-install-sw/08-run.sh
However, if you want to use WiFI, as Carlos said, first you need to stop
disabling in on boot:
sudo sed -i 's/^dtoverlay=disable-wifi/#dtoverlay=disable-wifi/' /boot/config.txt
Then, after first reboot, you need to set your country (it won't enable
until you do that).
This is done by clicking the network up/down arrow icon on the taskbar at
the very upper right.
Then after turning on WiFi, you can select a network and login.
You say you may want to use it where only WiFi is available.
However, in order log your Raspberry Pi into WiFi, you need to be
connected to your Raspberry Pi.
So you would need to either:

hook a display and keyboard to you RPi so you can login to WiFI
Set it up so that you can get to the desktop via Bluetooth. This is
a bit tricky to set up; I've done it in the past but this seems to not be
working now, not sure why.
Set the WFi login credentials in advance and hope that it works
when you get there:

WiFi login credentials are set in the file
"/etc/wpa_supplicant/wpa_supplicant.conf":
cttrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=US
network={
ssid=""
psk=""
key_mgmt=WPA-PSK
}
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kdoren/jambox-pi-gen/issues/21#issuecomment-761615676,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKJTE6R2FCQ2XPWIWOIL6TS2HONDANCNFSM4WC64XIQ
.

I've actually wondered why we leave bluetooth enabled, maybe that should be
shut off unless it's needed as well.  At very least, it takes power to run,
and at worst, it introduces more background activity in a system that we
really want to be dedicated to audio processing.
Carlos
On Sat, Jan 16, 2021 at 10:56 AM Kevin Doren<EMAIL_ADDRESS>wrote:

@goldyliang https://github.com/goldyliang I disabled wifi to keep
people out of trouble; WiFi can add a lot of jitter so you're not likely to
be happy using it. (I've heard people say "I have gigabit internet, so I
can use WiFi" but it doesn't work like that).
It's disabled on line 18 here when the image is generated:
https://github.com/kdoren/jambox-pi-gen/blob/jambox/stage3/04-install-sw/08-run.sh
However, if you want to use WiFI, as Carlos said, first you need to stop
disabling in on boot:
sudo sed -i 's/^dtoverlay=disable-wifi/#dtoverlay=disable-wifi/' /boot/config.txt
Then, after first reboot, you need to set your country (it won't enable
until you do that).
This is done by clicking the network up/down arrow icon on the taskbar at
the very upper right.
Then after turning on WiFi, you can select a network and login.
You say you may want to use it where only WiFi is available.
However, in order log your Raspberry Pi into WiFi, you need to be
connected to your Raspberry Pi.
So you would need to either:

hook a display and keyboard to you RPi so you can login to WiFI
Set it up so that you can get to the desktop via Bluetooth. This is
a bit tricky to set up; I've done it in the past but this seems to not be
working now, not sure why.
Set the WFi login credentials in advance and hope that it works
when you get there:

WiFi login credentials are set in the file
"/etc/wpa_supplicant/wpa_supplicant.conf":
cttrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=US
network={
ssid=""
psk=""
key_mgmt=WPA-PSK
}
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kdoren/jambox-pi-gen/issues/21#issuecomment-761615676,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKJTE6R2FCQ2XPWIWOIL6TS2HONDANCNFSM4WC64XIQ
.


Set RSA key size to 4096 (defaults to 2048 which is debatably insecure)

If nothing else is specified the RSA key requested from letsencrypt is 2048 bits. This is possibly insecure and 4096 bit keys should be a negligible difference in processing speed.

This is an automated message from CI:
Docker Image for build 1 is available on DockerHub as jc21/nginx-proxy-manager:github-pr-1765
Note: ensure you backup your NPM instance before testing this PR image! Especially if this PR contains database changes.

If nothing else is specified the RSA key requested from letsencrypt is 2048 bits. This is possibly insecure and 4096 bit keys should be a negligible difference in processing speed.
I am aware NPM defaults to using elliptic curve certificates, but RSA certificates are also issued for compatibility reasons. This ensures that those certificates are equally secure.


Showing every website safe for my own random_forest.pkl

Hi @rohitnaik246, I have created my own random_forest.pkl using train.py without any changes but still I am getting every website as safe. Why? How to get the output as Phishing for Phishing website? What changes I must do in the project? Please suggest the ideas.

If you created random_forest.pkl using the exact same train.py file, how is it anything new from what is already there in the model?
In this case, my suggestion would be what I discussed in this issue
Thanks!
Closing the issue as the solution is clear. If anything is unclear, please feel free to reopen the issue.


DataSetIterator: duplicate method

It seems like the methods totalExamples() and numExamples() in the DataSetIterator interface are duplicates. According to the Java doc, they are supposed to return the

Total examples in the iterator

and the

Total number of examples in the dataset

respectively.
Not sure whether this is just a documentation issue.

They return the same value. The docs are correct.

In that case, I think that one of the method should be removed to avoid confusion, if not already done.


fix: remove branch protection when user downgraded from ee

Description
The list of protected branches should be reset when user downgrades from EE to CE
PR fixes following issue(s)
Fixes #28734
Media
Type of change

Bug fix (non-breaking change which fixes an issue)

Testing
How Has This Been Tested?

[x] Manual
[x] JUnit

Test Plan
Issues raised during DP testing
Checklist:
Dev activity

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[ ] PR is being merged under a feature flag

QA activity:

[ ] Speedbreak features have been covered
[ ] Test plan covers all impacted features and areas of interest
[ ] Test plan has been peer reviewed by project stakeholders and other QA members
[ ] Manually tested functionality on DP
[ ] We had an implementation alignment call with stakeholders post QA Round 2
[ ] Cypress test cases have been added and approved by SDET/manual QA
[ ] Added Test Plan Approved label after Cypress tests were reviewed
[ ] Added Test Plan Approved label after JUnit tests were reviewed

/ok-to-test

/ok-to-test sha=1de983deefed34f5024d852bc950c83c908722ef


fix: nix develop is inheriting CARGO_PROFILE from nix release build

was this introduced in scope refactor?

Yes.

Does this close https://github.com/fedimint/fedimint/issues/2424?


Refactor test_sox_effects

Move misplaced sox compatibility tests (T,Fade, T.Vol, T.Vad) from test_sox_effects.py to test_sox_compatibility.py
Move test_sox_effects.py to test/sox_effect/ where all the other functionalities from torchaudio.sox_effects are tested

Thanks


Use variables inside tasks.json inputs config

I'd love to be able to create an input variable in tasks.json which defaults to a predefined variable:
"inputs": [
  {
    "type": "promptString",
    "id": "filePath",
    "description": "Enter file path",
    "default": "${relativeFile}"
  }
]

however currently the default is literally ${relativeFile}

Duplicate of #64358


SourceReportKind can't be constructed outside this lib

I want to call VmService.getSourceReport, but I can't provide anything for the list of SourceReportKind because its constructor is internal.
It seems like it should offer some static getters or a factory of some sort.

That looks like a bug in the generator. We use those ~enum classes more for documentation and a way to discover the valid enum values. The param should be taking a String -
Future<SourceReport> getSourceReport(
      String isolateId, List<String> reports,
      {String scriptId, int tokenPos, int endTokenPos, bool forceCompile});

Thanks for the report! 3.14.2 is published and should get you unblocked.


fix clear snapshot info for cancelled backup job to reduce log size

Cherry-pick #32604

run buildall


Cannot view $SYS topics using "./emqttd_ctl topics list" command

Environment

OS: Docker alpine:3.4
Erlang/OTP: default installed from this alpine version repos
EMQ: v2.0.7p1

Description
We try to build a docker container starting from the official Dockerfile of the 2.0.7 release
https://github.com/emqtt/emq-docker/releases/tag/v2.0.7
Service starts up correctly, we can access to the dashboard and see statics, clients etc... When we try to list topics using the command line
./bin/emqttd_ctl topics list
inside the running docker container the command doesn't list $SYS topics, only topics created from pub/sub operations. We try to change the default acl.conf commenting out some lines:
(...)
{allow, {user, "dashboard"}, subscribe, ["$SYS/#"]}.
%% {allow, {ipaddr, "<IP_ADDRESS>"}, pubsub, ["$SYS/#", "#"]}.
%% {deny, all, subscribe, ["$SYS/#", {eq, "#"}]}.
{allow, all}.
but nothing changes. We check the command from official docs
http://emqtt.io/docs/v2/commands.html#topics-show-topic
and it seems it should list also $SYS topics. What is wrong? Is a permission related problem?
Thanks.

Problem is solved in release 2.2:
/opt/emqttd # ./bin/emqttd_ctl broker
sysdescr  : Erlang MQTT Broker
version   : 2.2
uptime    : 15 minutes, 24 seconds
datetime  : 2017-07-25 15:39:35
/opt/emqttd # ./bin/emqttd_ctl topics list
$SYS/brokers/+/clients/#

This issue can be closed.


Unofficial server appears to be down

Never used node before, and it seems excessive to learn for a one off usage, but the online server appears to not be working right now. Does anyone know if this is a temporary or permanent thing?

The online server is not officially supported, though we listed the server in the README for the convenience of folks like you. @simi , do you know the status of the server?

Do you mean https://uncss-online.com/? That's up for me. Feel free to ping @pajasevi, he's author and maintainer of this project if you'll see any problems.

Fixed!

Thanks!

Dear @pajasevi https://uncss-online.com/ seems to be offline again. I was doing some tests to my pages with amazing results but I cannot reach the website. Thank you!

Up for me

It's already fixed, my VPS provider had some technical issues with the server.


Half of session 3 mailroom.py; older assignments

Uploading older session assignments. Need to revisit Mailroom, as I have the first half finished.

I won't comment on the older stuff -- but let me know if you have questions.


error: named cookie not present

I'm seeing a lot of this in my logs:
time="2017-12-04T15:10:47Z" level=error msg="http: named cookie not present"

I think it's coming from this line where it's logging the error and the return statement is commented out. That would suggest to me that it's not actually an error.

I changed this to a debug in https://github.com/bnfinet/lasso/pull/11/commits/90d140c2ee3483fb1cfe5af496a8c89e60a062be


Change progress for tasks of type "project"

This project looks great!
Is it possible to change progress for tasks of type "project"?

Hi Bjorn,
Yes. You may use same progress parameter.


Mini Cart Question

When you add an item to the cart, it brings up the mini cart and your options are to edit the cart or to checkout. Shouldn't there be a "continue shopping?" option?

I'll add in a continue option as a link below the buttons.

A continue shopping link has been added below the CTA section.


Timeline rendering is too heavy.

Comment

https://tips.hecomi.com/entry/2022/01/30/152519

fixed https://github.com/hecomi/uLipSync/commit/83668bd363a514a8c4f1ba1924a25b350830074f


ConfigParser.write takes 1 positional argument but 2 were given

Software Versions

Python: 3.9.18
OS: Ubuntu 22.04
Kivy: 2.3.0
Kivy installation method: pip install

Describe the bug
https://github.com/kivy/kivy/blob/c0d4894384abb81cf4729984a1eaa4c437d266da/kivy/__init__.py#L482-L487
Config is object of ConfigParser and ConfigParser.write method does not require any positional arguments.
Thus, it always raises TypeError: write() takes 1 positional argument but 2 were given at line 484.
I believe setting self.filename as kivy_config_fn is more accurate like this:
Config.filename = kivy_config_fn
Config.write()

I wonder it is the right behavior.
Expected behavior
Config.filename = kivy_config_fn
Config.write()

To Reproduce
from kivy.config import Config

with open("abc", 'w') as fd:
        Config.write(fd)

Code and Logs and screenshots
https://github.com/kivy/kivy/blob/c0d4894384abb81cf4729984a1eaa4c437d266da/kivy/__init__.py#L482-L487

Nice catch!
I can confirm the issue is reproducible with python examples/demo/touchtracer/main.py -m monitor --save
I guess no one used the --save option for the last  ~12 years? 😅
Would you like to make a PR for it?

Fixed via #8681


Remove duplicated items from vlab

Some items are now on both vlab and github. In addition, many items are manually uploaded to vlab, which takes a lot of time.
For items that are maintained on github, provide links to the github version so that users can download the materials there instead.

@Rick-Methot-NOAA , could you take a look at the Stock Synthesis website home page and see if it looks good to you?
For now, I have left the exes on github, but I think we discussed just using github releases?

Looks like changes have already been made to the public VLAB page?
I see several places where the viewer gets referred to the github
page, which may create confusion.  I envisioned a cleaner separation that
simply put all aspects of code management at github and kept user examples
at VLAB.
Will try to comment more later.
On Wed, Sep 15, 2021 at 10:19 AM Kathryn Doering @.***>
wrote:

@Rick-Methot-NOAA https://github.com/Rick-Methot-NOAA , could you take
a look at the Stock Synthesis website home page
https://vlab.noaa.gov/web/stock-synthesis/home and see if it looks good
to you?
For now, I have left the exes on github, but I think we discussed just
using github releases
https://github.com/nmfs-stock-synthesis/stock-synthesis/releases?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nmfs-stock-synthesis/stock-synthesis/issues/208#issuecomment-920215137,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABPV4IGTJLMJ4DQ3A3EN7VLUCDIRJANCNFSM5DUYTLNQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Sorry about that, I should have saved it as a draft instead!
My thinking was that we are actually maintaining user examples using the github repository, so it could actually be less confusing than having them duplicated on github, but I understand how having links away from vlab could be confusing also.
I'll wait for your additional comments until changing anything else.

Rick and I had a look at  the website and figured out what should stay on vlab and what to link to on github. The basic strategy was not to provide 2 links to the same place. I think we are all set now, and most things will now be linked to directly to where they are maintained, so no need to duplicate uploading to vlab!


Load sprintf.js via npm

This requires the sprintf.js npm module instead of inlining it and updates the library from 0.7-beta1 to version 1.0.3. A few changes to the tests were required to accommodate for the change.
See #41.

What do you think about throwing an error instead of just turning undefined into an empty string? I think it would be better to make people aware of their errors instead of suppressing them.
I'd be happy to add support for this to the sprintf library if that sounds good to you.

Is it really an error to have empty values? I think runtime errors in
translation are never more helpful than just outputting empty strings. I'd
be ok with adding a hook to be notified of these problems, but just
erroring out and not rendering someone's text seems like an extreme
punishment for a missing variable.
On Fri, Sep 23, 2016 at 1:34 PM Pascal Birchler<EMAIL_ADDRESS>wrote:

What do you think about throwing an error instead of just turning
undefined into an empty string? I think it would be better to make people
aware of their errors instead of suppressing them.
I'd be happy to add support for this to the sprintf library if that sounds
good to you.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/SlexAxton/Jed/pull/46#issuecomment-249269405, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAF5Kl2ZgnIK5EybJWz9x_WcTHunsG1Cks5qtBvCgaJpZM4KE7s0
.

@SlexAxton Just stumbled upon this again. What do you think about using the latest version of sprintf from npm (see also #56) and then wrap a little helper function around them to handle undefined? Happy to submit a PR.

Happy to take a PR for that.
On Fri, Aug 10, 2018 at 1:36 AM Pascal Birchler<EMAIL_ADDRESS>wrote:

@SlexAxton https://github.com/SlexAxton Just stumbled upon this again.
What do you think about using the latest version of sprintf from npm (see
also #56 https://github.com/messageformat/Jed/issues/56) and then wrap
a little helper function around them to handle undefined? Happy to submit
a PR.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/messageformat/Jed/pull/46#issuecomment-411989724, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAF5Km0csLK_t3uHpEUmNk1XGXlYF7qsks5uPSntgaJpZM4KE7s0
.


[WIP] Run additional bundle validators

Validate a bundle against the entire suite of validators for Operator Framework as described here.

/lgtm
Seems reasonable. @tedhtchang is it worth making a github action for this?

Yes reasonable. make bundle would fail with ERRO[0000] Error: Value : (codeflare-operator.v0.0.1) csv.Spec.Icon elements should contain both data and mediatype. So I need to add those static fields(+a few more) into the base manifest.

/cc @anishasthana I added validate-bundle to the precommit workflow. Please review the changes.
How to test:

Checkout this PR and run make bundle and inspect files in the files bundle/manifests folder.

base64data and mediatype are populated (to pass the additional validators --select-optional suite=operatorframework flag).
displayName: InstaScale should be w/o space.
static fields: annotions.container.(Image, reposiotry, support, operatorframework.io/suggested-namespace),links are populated in the generated csv.

Build bundle img locally podman build -f bundle.Dockerfile -t ttl.sh/testmybundle:10m .
Push temporary bundle image podman push ttl.sh/testmybundle:10m
Install bundle to your OCP operator-sdk run bundle ttl.sh/testmybundle:10m and verify the installed operator from OCP console.
Remove bundle operator-sdk cleanup codeflare-operator


[SDK] Add file-based and environment-based Secrets support

Environment-based secrets:
https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L2029

Protos.Secret secret = Protos.Secret.newBuilder()
                .setType(Protos.Secret.Type.REFERENCE)
                .setReference(Protos.Secret.Reference.newBuilder()
		 .setName(secretName).build()).build();

https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L1963
executorCommandBuilder.getEnvironmentBuilder().addVariablesBuilder()
                .setName(environmentName)
                .setType(Protos.Environment.Variable.Type.SECRET)
                .setSecret(secret);

File-based secrets:
https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L2221

Protos.Volume secretVolume = Protos.Volume.newBuilder().setSource(
                Protos.Volume.Source.newBuilder()
                        .setType(Protos.Volume.Source.Type.SECRET)
                        .setSecret(secret).build())
                        .setContainerPath(secretFile)
                        .setMode(Protos.Volume.Mode.RO)
                        .build();

executorInfoBuilder.setContainer(executorInfoBuilder.getContainerBuilder()
                    .setType(Protos.ContainerInfo.Type.MESOS)
                    .addVolumes(secretVolume).build());

YAML Configuration

> pods:
>   hello:
>     secrets:
>       s1:
>         secret: hello-world/secret1
>         env-key: SECRET1_ENV
>         file: SECRET1_FILE
>       s2:
>         secret: hello-world/secret2
>         file: SECRET2_FILE
>       s3:
>         secret: hello-world/secret3
>         env-key: SECRETe_ENV
>       s4:
>         secret: hello-world/secret4

If file path and env-key are both empty, we use secret path as file path

"File-based Secrets" code is tested with:
 * https://downloads.mesosphere.com/libmesos-bundle/libmesos-bundle-file-based-secrets.tar.gz

 * testing/pull/871 CCM channel

CI tests can not be added now. Feature is not in testing/master yet!

retest this please

retest this please

LGTM

This cannot be merged until testing/master has secrets support merged (mesos changes).
Once those land (they may have already?) it can be merged.

@bwood why?

It has integration tests right? Those tests will not pass without the changes landing.

Could you please read design doc, project slack channel, and comments in this PR before writing a blocking comment like that.

secrets support has already landed in testing/master

This PR does not depend on testing/master to support Secrets.   It does have examples/secrets.yml in hello-world. All had been manually tested (Please see design doc and JIRA ticket to implement integration test).
It is ~20 days old!

testing is broken

Chicken or egg: We want to have PRs small in size - we do not like to review PRs with 1K lines. We want to put all related tests and checks in the same PR. We do not want to keep a PR open more than a week.

@karya0 testing/master is same as 871? Can you please verify? Do we have space label also?

@benclarkwood and @mbdcos:  This design does not appear to address the possibility of per pod instance secrets.  Are we sure we don't need that?

Is something like the below a possibility?  Some way to refer to a particular instance of a pod is probably desirable.
pods:
  hello:
    secrets:
      s1:
        secret: hello-world/secret$POD_INSTANCE_INDEX
        env-key: SECRET1_ENV
        file: SECRET1_FILE

retest this please

  hello:
    secrets:
      s1:
        secret: hello-world/secret$POD_INSTANCE_INDEX
        env-key: SECRET1_ENV
        file: SECRET1_FILE

@gabrielhartmann  You can do this but it is just naming.
Did you mean something like?
  hello:
    secrets:
      s1:
        secret: hello-world/secret_name
        pod-instance: 1
        env-key: SECRET1_ENV
        file: SECRET1_FILE

If so, we do not specify pod instance number in any Spec in our YAML format AFAIK.
For example, we do not give pod instance in image-name:
  hello:
    container:
      image-name:  docker-image-1
        pod-instance: 1

Instead of that, we define a new pod, with new settings!
If you mean, better granularity for Secrets, for example task level Secrets, that has been discussed already.
In any case, it is a substantial design change and should be discussed in a separate design doc, outside of this PR.

I just tried testing/master to make sure it has Secret support.  It is still same old one.  @gabrielhartmann  who told you that it has Secret support?

Sorry, I gave misinformation.  Yea, you need to use testing/pull/871.

@mbdcos:  Nevermind about the secret per pod instance.  We'll address that later if necessary.

Thank you @gabrielhartmann

testing/master is merged today. So I pushed integration/tests.

Are you going to support DCOS_SECRETS_DIRECTIVE?

@mgummelt Yes SDK will support DCOS_SECRETS_DIRECTIVE, but not in this PR.  If DCOS_SPACE environment variable is not set by Marathon in 1.9, please use MARATHON_APP_ID environment variable.

def test_secrets_basic():
# 1) create Secrets
# 2) install examples/secrets.yml
# 3) if secret file is not created, tasks will fail
# 4) wait till deployment finishes
# 5) do replace operation
# 6) ensure all tasks are running
# 7) delete Secrets
def test_secrets_verify():
# 1) create Secrets
# 2) install examples/secrets.yml
# 3) verify Secrets content
# 4) delete Secrets
def test_secrets_update():
# 1) create Secrets
# 2) install examples/secrets.yml
# 3) update Secrets
# 4) restart task
# 5) verify Secrets content (updated after restart)
# 6) delete Secrets
def test_secrets_config_update():
# 1) install examples/secrets.yml
# 2) create new Secrets, delete old Secrets
# 2) update configuration with new Secrets
# 4) verify secret content (using new Secrets after config update)
def test_secrets_dcos_space():
# 1) create secrets in hello-world/somePath, i.e. hello-world/somePath/secret1 ...
# 2 ) Tasks with DCOS_SPACE hello-world/somePath
#       or some DCOS_SPACE path under hello-world/somePath
#               (for example hello-world/somePath/anotherPath/)
#

Needs an update to reflect the new fields: https://github.com/mesosphere/dcos-commons/blob/master/docs/pages/yaml-reference.md

@nickbp PR #1093

:+1:


TypeError: Cannot read properties of undefined (reading 'methods')

We see this crash on Faucet in Westend:

Currently running on 1.3.2
Also not happenning on Paseo

This was solved in #794 (released on v1.6.1). Feel free to comment if you see this error again. Thanks!


Fix #1367, Add cfe functional tests to CI

Describe the contribution
Fixes #1367
Add the functional tests to the CI workflow.
Testing performed
Changed the tests to be wrong and force a failure.
Changed the tests to have a delay to simulate a timeout.
Expected behavior changes
Will now fun the cFE functional tests on every pull request.
System(s) tested on
Ubuntu 20.04
Contributor Info - All information REQUIRED for consideration of pull request
Alex Campbell GSFC

CCB:2021-06-16 APPROVED

Should we modify the cFE assert so that the test log is dumped directly and doesn't depend on the Operating System?

@jphickey open issue to do this

Could also set standard output to be synchronous. This only works on Linux


How to pass an argument to the current script through a click?

Hey! :)
Is it possible to have an clickable item, that refreshes the script with a given argument?
And how to access it inside of the script?
So for example in Ruby it would be enough for me if the argument would be available in ARGV or ENV.
I think I don't understand the documentation, how to achieve that with the bash or param options.

You pass the absolute path to the script to bash=.. and param1=.., param2.. and so on are the arguments. Maybe look at one of the Ruby examples in matryer/bitbar-plugins.

hey, thanks for the quick response! :)
even after reading all the example plugins... (and trying a couple)
I don't get the argument into the script output to show it again in the plugin menu.
Even in the examples.
The interesting thing is that I see the arguments in the terminal (when I use terminal=true), but they are not available in the script.
so maybe I'm holding it wrong?
#!/usr/bin/env ruby

puts [
  'test',
  '---',
  "click me | bash=#{$0} param1=foo param2=bar terminal=false",
  '---',
  "script: #{$0}",
  "ARGV: #{ARGV.inspect}"
]

(terminal=false) ARGV is always an empty array
(terminal=true) ARGV is empty in the menu, but ["foo", "bar"] in the terminal

bitbar version is 1.9.2

Relates #323. This seems to be a common misunderstanding...

ah, my bad. was already expecting something like that.
So the only way would be a tmp file that keeps the output?

Yeah, I believe this is a common solution.


Update providers-data.csv

added a couple more take over's

Added the new ones manually after some formatting


Context partially cleared after first turn of conversation

I noticed that after one turn of a conversation, the Context gets some properties cleared. I assume this is because the Context is being saved to an external store and then when it is unfrozen, it doesn't have things like functions or anything else that was hydrated.
Is there a way around this? One thing that makes it confusing is that the hydrated context is fully available on the first turn.

I assume this is because the Context is being saved to an external store and then when it is unfrozen, it doesn't have things like functions or anything else that was hydrated

This is correct

Is there a way around this?

You can either look into conversation.run and https://grammy.dev/plugins/conversations#working-with-plugins, or you can call these functions directly rather than installing them on the context object.

One thing that makes it confusing is that the hydrated context is fully available on the first turn.

That is correct, this was an early design mistake. It will be fixed for the next major version.


Add full provenance info to metadata.tsv files

Jonathan has requested that the full provenance metadata information be included in the metadata.tsv files that are currently being generated (which only include the actual data metadata that we obtain during dataset ingest).
Both for the endpoint that will generate a metadata.tsv file for a dataset and a publication time change the contents of the metadata.tsv file to include the full json response of the entity-api /entities/<dataset id> endpoint and add to that lists of the full response all associated donors, organs and samples.  See example cypher below to obtain the uuids of these associated entities, then make calls to the entity-api /entities/<entity id> endpoint to get the json for inclusion in the list of each type.
Related to https://github.com/hubmapconsortium/ingest-api/issues/411

@tjmadonna It looks like whenever a dataset gets published a metadata.json file gets created and that is what needs to be updated. See: https://github.com/hubmapconsortium/ingest-api/blob/b591031cdf943749f3bf858dd12f594eb5f6421a/src/app.py#L951-L960
Relevant PR where this update is made:
https://github.com/hubmapconsortium/ingest-api/pull/481/files
A companion issue is https://github.com/sennetconsortium/entity-api/issues/285


Add nightly builds of the app

This is to allow people to test the app without us manually sending APKs.
Ideally, at this early stage, location of these builds should be a bit secluded, so only people that know what are doing can get the build (and we are not swamped by lots of bug reports on unfinished app).

Functional as of 4f74c21


Does nodegit support git-upload-pack?

I'm trying to allow clients to clone git repos from my server. Of course there's some back and forth that needs to be done before the official packs are sent, but for now I'm forking a process that calls
git-(upload-pack or receive-pack)--stateless-rpc <local working directory>
to communicate with the client. Any guidance would be appreciated.

https://libgit2.org/libgit2/#HEAD/group/remote/git_remote_upload
https://www.nodegit.org/api/remote/#upload

@saoudrizwan Do you have an example of you want about it? That would be super helpful


Remove the use of KafkaProducer from the dispatcher and use ReactiveKafkaProducer instead

Fixes #3191 and progress in #2928

Proposed Changes
A clean, without-conflict clone PR of #3192

/test all

/retest

/retest

/retest

/retest-required


DRILL-5859: Fixed queued time calculation

@paul-rogers please review

Thanks for the first-time commit! We like to have the JIRA number in the title; please go ahead and ecit the title so it looks like this:
DRILL-1234: Fixed queued time calculation
Use the actual Drill JIRA ticket number in place of "1234".

@paul-rogers Made the changes as suggested. please review

Update the title.


Implement Service & ServiceObject classes

Splitting up the massive #904 into pieces. The first is the new classes, Service and ServiceObject.

Fixed the things! PTAL

Looks good! Thanks :)


[SPARK-17159] [streaming]: optimise check for new files in FileInputDStream

What changes were proposed in this pull request?
This PR optimises the filesystem metadata reads in FileInputDStream, by moving the filters used in FileSystem.globStatus and FileSystem.listStatus into filtering of the FileStatus instances returned in the results, so avoiding the need to create FileStatus instances within the FileSystem operation.

This doesn't add overhead to the filtering process; that's done as post-processing in theFileSystem glob/list operations anyway.
At worst it may result in larger lists being built up and returned.
For every glob match of a file, the code saves 1 RPC calls to the HDFS NN; 1 GET against S3
For every glob match of a directory, the code the code saves 1 RPC call and 2-3 HTTP calls to S3 for the directory check (including a slow List call whenever the directory has children as it doesn't exist as a blob any more)
for the modtime check of every file, it saves a Hadoop RPC call, against all object stores which don't implement any client-side cache, an HTTP GET.
By entirely eliminating all getFileStatus() calls in the listed files, it should reduce the risk of AWS S3 throttling the HTTP request, as it does when too many requests are made to parts of a single S3 bucket.

How was this patch tested?
Running the spark streaming tests as a regression suite. In the SPARK-7481 cloud code, I could add a test against S3 which prints to stdout the exact number of HTTP requests made to S3 before and after the patch, so as to validate speedup. (the S3A metrics in Hadoop 2.8+ are accessible at the API level, but as they are only accessible in a new API added in 2.8; it'd stop that proposed module building against Hadoop 2.7. Logging and manual assessment is the only cross-version strategy.

Test build #64140 has started for PR 14731 at commit 738c51b.

Test build #64140 has finished for PR 14731 at commit 738c51b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64140/
Test PASSed.

LGTM. Does this sort of change make sense elsewhere where PathFilter is used? I glanced at the others and it looked like a wash in other cases.

I'm going to scan through and tune them elsewhere; really I'm going by uses of the listFiles calls
There's actually no significant use elsewhere that I can see; just a couple of uses which filter on filename —so there is no cost penalty.

SparkHadoopUtil.listLeafStatuses() does implement its own directory recursion to find files; FileSystem.listFiles(path, true) does that, and on S3A will do flat scan that is O(files/5000); no directory overhead at all.
Otherwise, globStatus() can be pretty slow against object stores, but the fix there isn't in the client code; it means someone needs to implement HADOOP-13371, S3A globber to use bulk listObject call over recursive directory scan —more specifically, an implementation scalable to production datasets.

Returning to this patch, should I cut out the caching? I think it is superfluous.

Why is the caching superfluous -- because no file is evaluated more than once here?

to be precise: the caching of file modification times is superfluous. It's there to avoid the cost of executing getFileStatus() on previously scanned files. Once you use the FileStatus returned in a listing, you aren't calling getFileStatus(), hence: no need to cache

Test build #64142 has started for PR 14731 at commit 6e8ace0.

Ah right, you already have the modification time for free. Sounds good, remove the caching.

Test build #64142 has finished for PR 14731 at commit 6e8ace0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64142/
Test PASSed.

Test build #64156 has started for PR 14731 at commit b08e3c9.

Test build #64156 has finished for PR 14731 at commit b08e3c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64156/
Test PASSed.

This is ready to go right @steveloughran ? LGTM

LGTM. I was trying to see if there was a way to create a good test here by triggering the takes-too-long codepath and having a counter, but there's no obvious way to do that deterministically. I am doing a test for this against s3 in the spark-cloud module I'm writing; I can look at the printed counts of getFileStatus before/after the patch to see the difference, but the actual (testable) metrics are only accessible with forthcoming Hadoop 2.8 release.
TL;DR: no easy test, so there's nothing left to do

Actually, I've just noticed that DStream behaviour isn't in sync with the streaming programming guide, which says "files written in nested directories not supported)". That is: SPARK-14796 didn't patch the docs.
it may as well be fixed in this patch. How about, in the bullet points underneath

Wildcards may be used to specify a set of directories to scan for new files, for example hdfs://nn1:8050/users/alice/logs/2016-*/*.gz

-New directories and their contents will be discovered as they arrive
Special points for object stores

Wildcard lookup may be very slow with some object stores.
Directory rename is not atomic; if a directory is renamed into the streaming source, then the files within may only be discovered and process across a multiple streaming windows.

there's another optimisation; use the {{SparkHadoopUtils.isGlobPath()}} predicate to recognise when the dir path isn't a wildcard, in which case just do a simple listFiles()}}. Until that shortcutting is done automatically in the Hadoop FS implementation, spark can do it on its side. As the {{listFiles()}} call was what was used before SPARK-14796, it has to be compatible, else SPARK-14796 has broken things

I've now done the s3a streaming test/example
this uses a pattern of s3a/path/sub* as the directory path; then creates a file in a directory and renames the dir to match the path; verifies that the file was found in the time period allocated
https://gist.github.com/steveloughran/c8b39a7b87a9bd63d7a383bda8687e7e
Notable that the scan of the empty dir took 150ms; once there's data in the tree the time jumps up to 500ms once there are two entries under the tree, one dir and one file
summary stats show 72 getFileStatus calls at the FS API, mapping to 140 HEAD calls and 88 LIST operations.
 S3AFileSystem{uri=s3a://stevel-ireland-new, workingDir=s3a://hwdev-steve-ireland-new/user/stevel, inputPolicy=sequential, partSize=104857600, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=1048576, multiPartThreshold=2147483647, statistics {292 bytes read, 292 bytes written, 101 read ops, 0 large read ops, 11 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=343b706a-c238-4d71-9ed8-8083601ac28a-hwdev-steve-ireland-new} {fsURI=s3a://hwdev-steve-ireland-new} {files_created=1} {files_copied=1} {files_copied_bytes=292} {files_deleted=1} {directories_created=3} {directories_deleted=0} {ignored_errors=2} {op_copy_from_local_file=0} {op_exists=1} {op_get_file_status=72} {op_glob_status=16} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=27} {op_mkdirs=2} {op_rename=1} {object_copy_requests=0} {object_delete_requests=3} {object_list_requests=88} {object_continue_list_requests=0} {object_metadata_requests=140} {object_multipart_aborted=0} {object_put_bytes=292} {object_put_requests=4} {stream_read_fully_operations=0} {stream_bytes_skipped_on_seek=0} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=292} {streamOpened=1} {stream_backward_seek_pperations=0} {stream_read_operations_incomplete=0} {stream_bytes_discarded_in_abort=0} {stream_close_operations=1} {stream_read_operations=1} {stream_aborted=0} {stream_forward_seek_operations=0} {streamClosed=1} {stream_seek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }}

I'm going to do a test run with the modification here and see what it does to listing and status

Test build #64296 has started for PR 14731 at commit 79b57a2.

updated the code to bypass the glob routine when there is no wildcard; this bypasses something fairly inefficient.
reporting FNFE on that base dir differently; skip the stack trace (maybe: log at a lower level?).
Updated the docs with a special list of blobstore best practises.

It's a bit hard to get some of that phrasing of what the wildcard does right; needs careful review.
Tested using my s3 streaming test, which did use a * in the wildcard. All works, but no improvements in speed on what is a fairly unrealistic structure. The time to recursively list object stores remotely is tangibly slow. Maybe that should go in the text too: "it can be take seconds to scan object stores for new data, with the time being proportional to directory depth and the number of files in a directory. Shallow and wide directory trees are faster"

Test build #64296 has finished for PR 14731 at commit 79b57a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64296/
Test PASSed.

The logic has got complex enough it merits unit tests. Pulling into SparkHadoopUtils itself and writing some for the possible: simple, glob matches one , glob matches 1+, glob doesn't match, file not found

Test build #64368 has started for PR 14731 at commit b63abfe.

Having looked at the source code, FileSystem.globStatus() uses the glob patterns, which are not the same as the posix regexp ones. org.apache.hadoop.fs.GlobPattern does the conversion.
For the docs, I'll just use a wildcard * in the example, rather than try anything more sophisticated.

Test build #64368 has finished for PR 14731 at commit b63abfe.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64368/
Test FAILed.

Test build #64486 has started for PR 14731 at commit 9bc0ea9.

Test build #64486 has finished for PR 14731 at commit 9bc0ea9.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64486/
Test FAILed.

Test build #64488 has started for PR 14731 at commit fe40bd2.

Test build #64488 has finished for PR 14731 at commit fe40bd2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64488/
Test PASSed.

Test build #64534 has started for PR 14731 at commit 4134620.

Test build #64534 has finished for PR 14731 at commit 4134620.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64534/
Test FAILed.

Merged build finished. Test FAILed.

Test build #64662 has started for PR 14731 at commit b60f175.

Test build #64662 has finished for PR 14731 at commit b60f175.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64662/
Test PASSed.

The latest patch pulls out the shortcutting of the globStatus call if there's no wildcard chars in the path; closer to the original patch

Test build #65498 has started for PR 14731 at commit 735fc7c.

Test build #65498 has finished for PR 14731 at commit 735fc7c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65498/
Test PASSed.

Test build #65592 has started for PR 14731 at commit 57f697d.

Test build #65592 has finished for PR 14731 at commit 57f697d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65592/
Test PASSed.

Merged build finished. Test PASSed.

Test build #66656 has started for PR 14731 at commit f8ed8a3.

Test build #66656 has finished for PR 14731 at commit f8ed8a3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66656/
Test PASSed.

Merged build finished. Test PASSed.

@srowen have you got any comments on the last patch?

Test build #70819 has started for PR 14731 at commit a9a6f7b.

Test build #70819 has finished for PR 14731 at commit a9a6f7b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70819/
Test PASSed.

Sean, I think I've managed to delete the lines where you were asking about globs

Am I right that the net change here is not an optimization but an expansion of the behavior to support globs rather than single dirs?

There's no changes in this source to change the expansion policy; that went in with SPARK-14976, "make StreamingContext.textFileStream support wildcard". This updates the docs to cover what goes on (the wildcard covering directories, but not the files inside them), and makes the scan much more efficient on object stores. No changes in the semantics of what gets found or when things get found

@steveloughran Are you still working on this?

let me do a quick review & update

Test build #71866 has started for PR 14731 at commit 06b2bee.

@uncleGen I've updated it. Note that HADOOP-13946 tracks the changes in the Hadoop docs, which writes down what HDFS actually does, then note how cloud object stores have no consistent behaviour w.r.t. timestamps.While I personally believe that direct PUT calls is the way to write data, there's still ambiguity then as to when the objects get a timestamp (S3 : when the PUT/multipart put is first initiated, and not updated on the close() if the put was started earlier) —so when they become visible. So: I don't go into the details, just say "look at the docs, then test on your system". That's about as authoritative as you can get

Test build #71866 has finished for PR 14731 at commit 06b2bee.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71866/
Test PASSed.

@uncleGen: reviewed this, tweaked the docs slightly but otherwise, there's nothing left to do that I can see

Test build #73433 has started for PR 14731 at commit 04f4967.

Test build #73434 has started for PR 14731 at commit 724495b.

Test build #73433 has finished for PR 14731 at commit 04f4967.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73433/
Test PASSed.

Merged build finished. Test PASSed.

Test build #73434 has finished for PR 14731 at commit 724495b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73434/
Test PASSed.

@srowen Waiting for your final OK

The Hadoop FS Spec has now been updated to declare exactly what HDFS does w.r.t timestamps, and warn that what other filesystems and object stores do are implementation and installation specific features: filesystem.md
That is the associated documentation update with this one; some of the content there was originally here, but moved over to the hadoop docs for the HDFS team to take the blame for when it changes.

Any more comments?

Test build #74990 has started for PR 14731 at commit a3aaf26.

Test build #74990 has finished for PR 14731 at commit a3aaf26.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74990/
Test PASSed.

Is there anything else I need to do here?

@srowen anything else I need to do here?

Ok. what is the way? Do I write a formal proposal?
Because right now there is no reliable way to get the full dependency graph of Spark + hadoop cloud JARs + direct cloud provider JARs (azure,aws) and their dependencies (jackson) in sync.
Which means that getting Spark to talk to object stores is more miss than hit.
I'm happy to follow the proposal mechanism, including progress reports &c, but I do at least need some kind of hope that my work will actually get in.

Steve I think the main point is you should also respect the time of reviewers. The way most of your pull requests manifest have been suboptimal: they often start with a very early WIP (which is not necessarily a problem), and once in a while (e.g. a month or two) you update it to almost completely change it. The time itself is a problem. It requires a lot of context switching to review your pull requests.

Reynold, I know very much about the time of reviewers, I put 1+h a day on the hadoop codebase reviewing stuff, generally trying to review the work of non-colleagues, so as to pull in the broad set of contributions which are needed..
I have been trying to get some object store related patches into spark alongside the foundational work in fundamentally transforming how we work with object storage, especially S3, in Hadoop. Without the spark side changes, a lot gets lost: here the performance is approx 100-300mS/file when scanning an object store.
here I've split things in two, docs and diff. Both are independent, both are reasonably tractable. If they can be reviewed fast and added, there's no problems of patches ageing, everyone having to resync.
We can get this out the way, and you've have fewer reasons to be unhappy with me.


⚠ Bofrost.es has degraded performance

In 862111a, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 4102 ms

Resolved: Bofrost.es performance has improved in 3870419 after 10 minutes.


hosting websocket/http from same port

I have a working http(s) server solution which has been developed directly with boost asio. One of my collegues also have recently developed a websocket library using beast which we use for hosting some services in the same server application.
I have been planning (and already started development to some extent) to use beast for both http server and websocket server functionalities, merge two of our libraries and maybe host http/ws on the same port etc.
I have stumbled upon the following link
https://www.reddit.com/r/cpp/comments/6in9e9/multi_port_http_https_ws_wss_websocket_openssl/#bottom-comments
and soon after that I've seen the following comment
https://github.com/boostorg/beast/issues/946#issuecomment-353678615
Since it has similarities to what I want to achieve, I would be glad to hear the reason(s) for abandoning this example. And I also would like to get your recommendations/warnings how I should and should not use beast for such a scenario if any.
Thanks for good work that has been put into this library

Thanks for the kind words! Both of the advanced server examples do HTTP and WebSocket on the same port. The "advanced-server-flex" program also supports plain and SSL on the same port:
https://github.com/boostorg/beast/tree/1da229a27c6f0539d422bcedbcf47cfe2517164a/example/advanced

Oh great then. That means its just a change in the way you distribute your examples and beast continues to support this approach.
Thanks & Regards


🐞 [Bug]: UI fix to the delete button color in workers

Is there an existing issue for this?

[X] I have searched the existing issues

which package/s did you face the problem with?
Dashboard
What happened?
Change the color of the delete worker buttons; both remove and cancel text are in red, which is confusing, and the remove color background should be the one in red, not the other way around.
Steps To Reproduce
No response
which network/s did you face the problem on?
Test
version
2.4.0
Twin ID/s
No response
Node ID/s
No response
Farm ID/s
No response
Contract ID/s
No response
Relevant screenshots/screen records

Relevant log output
.

The same thing is true for deleting a domain from a VM.
Cancel shouldn't be in the green color.

the buttons moved to the wrong side.

verified on devnet
version : 0139c63
the buttons are now placed in the right side


Data naming and organization updates to pna_control

Changed data naming and organizational structure. Made new user files that are convenient for common measurement types and allow for easy retrieval of data.

@nmaterise I'm putting all of my local changes in mcrae-patch2


Fix typo on Azure SQL database backup restore page (busniess => business)

@lfittl : Thanks for your contribution! The author(s) have been notified to review your proposed change.

The update looks good; merging.
@lfittl - Thanks!


Would this work for a containerized function deployed with ECR?

As title says:
I have a function that I needed to deploy using ECR due to the fact that it uses some heavy ML libraries that couldn't possibly fit within the lambda limits.
A cold start takes quite a while, and I was wondering if this plugin would work for this specific case, since I haven't seen any specifics as for what regards functions deployed with ECR.
Thanks in advance for any help!

Hi @wtfzambo sorry for the delay. It seem that my previous answer never sent.
This plugin only works for lambda functions deployed using the serverless framework.
Whether the image comes from ECR or not should be totally irrelevant...

@juanjoDiaz hey, don't worry, thanks for the reply!
I am indeed using Serverless Framework, with ECR as the source of my code:
functions:
  myFunc:
    image:
      name: latest

The reason I'm asking this is due to the fact that I saw a post on reddit which said that OP couldn't manage to make this work while having his function hosted on ECR. It sounded weird to me, which is why I asked. At this point I have the suspicion that that guy messed something up in the configuration.
I'll do some test and let you know!

Closing since there seems to be no issue.
Feel free to reopen if you feel that there is anything left to tackle.

@juanjoDiaz yo, sorry for the late update, anyway I just wanted to inform you that the plugin works flawlessly even with lambdas deployed using ECR. That guy on reddit must have messed something up!


Unable to import TabularAutoML

🐛 Bug
Hello,
I am trying to use LightAutoML in my local, Google Colab and Kaggle with the latest version.
when I am importing
from lightautoml.automl.presets.tabular_presets import TabularAutoML
from lightautoml.tasks import Task

I get this error

TypeError                                 Traceback (most recent call last)
[<ipython-input-8-e1576580262b>](https://localhost:8080/#) in <cell line: 17>()
     15
     16 # LightAutoML presets, task and report generation
---> 17 from lightautoml.automl.presets.tabular_presets import TabularAutoML
     18 from lightautoml.tasks import Task

10 frames
[/usr/local/lib/python3.10/dist-packages/log_calls/log_calls.py](https://localhost:8080/#) in _add_class_attrs(self, klass)
   2136         this_deco_class = self.__class__
   2137         this_deco_class_name = this_deco_class.__name__
-> 2138         setattr(
   2139             klass,
   2140             'get_' + this_deco_class_name + '_wrapper',

TypeError: cannot set 'get_record_history_wrapper' attribute of immutable type 'object'

To Reproduce
Steps to reproduce the behavior:

https://colab.research.google.com/github/AILab-MLTools/LightAutoML/blob/master/examples/tutorials/Tutorial_9_neural_networks.ipynb
Install the Library
Try to Import -> from lightautoml.automl.presets.tabular_presets import TabularAutoML

Attached is the screenshot of the error.

@jatinchawda1503 pip install lightautoml==0.3.8b1 will solve your issue as it is suitable for Colab's python 3.10


irmin-pack: Use temp control files upon writing

This PR changes the writing of control files: They were currently updated while making the assumption that the file itself being smaller than a page, the writing would be atomic. However it was deemed dubious that we could use such an assumption.
This is why this PR:

Adds the usage of temporary control file when writing a new payload (path of the cf + .tmp).
Moves the temporary file in stead of the old control file.
Then updates the Io file stored in the Control_file.t with the new
On the RO instances side, it simply re-opens a new Io file, as the old file descriptor might point to the inode of a file replaced by the RW instance.
Updates the errors of the related functions.

This leaves however one question in the case of RO instances:
Because we open a new Io each time we want to reload the payload, storing said Io has currently purpose. Maybe we could do something about that (separating RO & RW control files ? Using an option ? Remove the reload function ?)

Codecov Report

Merging #2206 (80e18e9) into main (eddf517) will increase coverage by 0.02%.
The diff coverage is 100.00%.

:exclamation: Current head 80e18e9 differs from pull request most recent head 4e6fd34. Consider uploading reports for the commit 4e6fd34 to get more accurate results

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##             main    #2206      +/-   ##
==========================================
+ Coverage   68.04%   68.06%   +0.02%
==========================================
  Files         135      135
  Lines       16378    16385       +7
==========================================
+ Hits        11144    11153       +9
+ Misses       5234     5232       -2

Impacted Files
Coverage Δ

src/irmin-pack/unix/control_file_intf.ml
82.14% <ø> (ø)

src/irmin-pack/unix/control_file.ml
88.46% <100.00%> (+0.65%)
:arrow_up:

src/irmin-test/store.ml
94.95% <0.00%> (+0.05%)
:arrow_up:

src/irmin-fs/unix/irmin_fs_unix.ml
68.38% <0.00%> (+0.64%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Thanks for looking into this! current-bench doesn't report anything alarming but it would be nice to also run the replay trace, I'm curious if the posix way has any impact on performances in the long run :)

Finally ran the replay trace thx to the help of @art-w & @adatario, here are the results:
</details>

 |                          | baseline  |       pr
 | --                       | --        | --
 | -- main metrics --       |           |
 | CPU time elapsed         |   103m42s |   103m54s 100%
 | Wall time elapsed        |   103m43s |   103m55s 100%
 | TZ-transactions per sec  |   739.362 |   737.883 100%
 | TZ-operations per sec    |  4829.344 |  4819.686 100%
 | Context.add per sec      | 14179.475 | 14151.116 100%
 | tail latency (1)         |   1.416 s |   1.227 s  87%
 |                          |           |
 | -- resource usage --     |           |
 | disk IO (total)          |           |
 |   IOPS (op/sec)          |   181_302 |   180_946 100%
 |   throughput (bytes/sec) |  18.099 M |  18.063 M 100%
 |   total (bytes)          | 112.608 G | 112.610 G 100%
 | disk IO (read)           |           |
 |   IOPS (op/sec)          |   181_204 |   180_849 100%
 |   throughput (bytes/sec) |  10.648 M |  10.627 M 100%
 |   total (bytes)          |  66.249 G |  66.251 G 100%
 | disk IO (write)          |           |
 |   IOPS (op/sec)          |        97 |        97 100%
 |   throughput (bytes/sec) |   7.451 M |   7.436 M 100%
 |   total (bytes)          |  46.359 G |  46.359 G 100%
 |                          |           |
 | max memory usage (bytes) |   0.374 G |   0.391 G 104%
 | mean CPU usage           |      100% |      100%     ```

Here's the stats.txt for another benchmark run where the correct version (this PR commit) was used:

baseline
pr

hostname
posada

os type
Unix

big endian
false

runtime params (no gc)
b=1,H=0,p=0,t=0,v=0,w=1,W=0

backend type
native

ocaml version
4.14.0

word size
64 bits

Start Time
2023/03/06 10:49:35 (GMT)
2023/03/15 09:30:27 (GMT)

Store Type
pack

irmin-pack GC count
0

gc.minor_heap_size
262144

gc.major_heap_increment
15

gc.space_overhead
120

gc.verbose
0

gc.max_overhead
500

gc.stack_limit
0

gc.allocation_policy
2

gc.window_size
1

gc.custom_major_ratio
44

gc.custom_minor_ratio
100

gc.custom_minor_max_size
8192

baseline
pr

-- main metrics --

CPU time elapsed
106m05s
108m17s 102%

Wall time elapsed
106m05s
108m21s 102%

TZ-transactions per sec
722.768
708.047  98%

TZ-operations per sec
4720.957
4624.802  98%

Context.add per sec
13861.239
13578.917  98%

tail latency (1)
1.368 s
1.153 s  84%

-- resource usage --

disk IO (total)

IOPS (op/sec)
177_241
173_625  98%

throughput (bytes/sec)
17.693 M
17.332 M  98%

total (bytes)
112.611 G
112.609 G 100%

disk IO (read)

IOPS (op/sec)
177_146
173_531  98%

throughput (bytes/sec)
10.409 M
10.197 M  98%

total (bytes)
66.252 G
66.250 G 100%

disk IO (write)

IOPS (op/sec)
95
93  98%

throughput (bytes/sec)
7.284 M
7.135 M  98%

total (bytes)
46.359 G
46.359 G 100%

max memory usage (bytes)
0.386 G
0.388 G 100%

mean CPU usage
100%
100%

Nice, thx a lot, it should be good to merge now

@adatario thanks for running the benchmark! Do you have the full stats.txt? I'd like to see some of the more detailed stats around commit times.

@adatario thanks for running the benchmark! Do you have the full stats.txt? I'd like to see some of the more detailed stats around commit times.

Yes, they are in the irmin-tezos-benchmarking repo: https://github.com/tarides/irmin-tezos-benchmarking/tree/main/benchmarks/2023-03-temp-control-file (currently not public)

Thanks @adatario!
These are the lines I wanted to look at:
 |                                                   |          |        cumu         | share |      min       |       max       |      avg

...

 | commit duration (s)                               | baseline |       36min25s      |   34% |  8.123 ms      |   1.368  s      | 21.855 ms
 |                                                   | pr       |       36min57s 101% |   34% |  8.721 ms 107% |   1.153  s  84% | 22.176 ms 101%

There is a slight overhead on average for commit (we write the control file on every commit), but it is very small. I think this change is good to go! :tada:


Fix Windows polling

Noticed that #88 broke Windows SocketSet functionality, but it wasn't caught by any tests. Added coverage for SocketSet functions and polling. Also added one additional check to the "Connect to an echo server" test (expecting that the read message from the socket should match the write message), and changed allocator usage to std.testing.allocator instead of the previous page_allocator. Reverted pertinent polling-related parts of #88.

@ikskuh My apologies, PR #88 introduced breakage for SocketSet and waitForSocketEvent on Windows. Related changes were reverted, and I've now set up wine-based testing locally and added tests to the suite for CI to avoid such issues in the future.
As an aside, it appears possible to completely remove select and all of WindowsOSLogic in favor of simply using std.posix.poll pending this upstream PR, but I wanted to check in with you and ask if you were aware of any particular reason to specifically favor select on Windows over WSAPoll?


Edit menu stays when last tab is closed

Describe the Bug
Edit menu stays and editor actions are still emitted on shortcuts despite the last tab being closed:

Steps to Reproduce
Steps to reproduce the behavior:

Create BPMN tab
Close BPMN tab
(no tabs are open anymore at this point)
Edit Menu is still active, keyboard shortcuts still trigger

Expected Behavior

[ ] No tabs are open -> Edit menu is not visible, keyboard shortcuts don't trigger

Environment

OS: Any
Camunda Modeler Version: next

Closed via https://github.com/camunda/camunda-modeler/pull/1072


Randomly generated forex and crypto data cannot be used

Expected Behavior

Forex and crypto data generated by the random data generator can be used in backtests.
Actual Behavior

Using forex or crypto data generated by the random data generator in backtests causes errors because of invalid symbols and non-existent symbol properties database entries.
I realize this issue contains two separate bugs, but they are very similar to each other so I decided to create a single issue for them.
Potential Solution

Both errors come from this snippet:
https://github.com/QuantConnect/Lean/blob/4de25b6cd4688c9d8eb1e03802bae537f5d702ef/Common/Securities/SecurityService.cs#L73-L82
I'm not entirely sure how to fix this. Here are my thoughts:

Generated forex and crypto tickers should be currency pairs of length 6, not 3 like they are now. Possible solutions:

Generate a random 6-character string for forex and crypto tickers.
Generate a random 3-character string for forex and crypto tickers and append USD to it. This may be better because the default account currency is also USD?

For crypto symbols we currently require an entry in the symbol properties database because unlike forex symbols not all crypto symbols are of length 6 (e.g. BTC-USDT). Possible solutions:

Make the random data generator add generated crypto symbols to the symbol properties database.
Add a fallback to the second if statement above. If we have a crypto symbol of length 6 not existing in the symbol properties database, we could assume the last 3 characters are the quote currency. If the length is not 6, we can still throw the exception (but randomly generated crypto symbols would always have length 6).

Reproducing the Problem

Run the random data generator using the --start=20201230 --end=20210404 --symbol-count=1 --security-type=<Forex or Crypto> --resolution Daily options and run a backtest which adds the generated symbol.
This error appears for generated forex symbols:
20210406 15:28:13.947 ERROR:: Engine.Run(): During the algorithm initialization, the following exception has occurred: ArgumentException : Cash symbols cannot be null or empty.
at QuantConnect.Securities.Cash..ctor (System.String symbol, System.Decimal amount, System.Decimal conversionRate) [0x00019] in <9e81f04a2ec74d69bcaf40c72e08f334>:0
at QuantConnect.Securities.CashBook.Add (System.String symbol, System.Decimal quantity, System.Decimal conversionRate) [0x00000] in <9e81f04a2ec74d69bcaf40c72e08f334>:0
at QuantConnect.Securities.SecurityService.CreateSecurity (QuantConnect.Symbol symbol, System.Collections.Generic.List`1[T] subscriptionDataConfigList, System.Decimal leverage, System.Boolean
addToSymbolCache) [0x000eb] in <9e81f04a2ec74d69bcaf40c72e08f334>:0
at QuantConnect.Securities.SecurityManager.CreateSecurity (QuantConnect.Symbol symbol, System.Collections.Generic.List`1[T] subscriptionDataConfigList, System.Decimal leverage, System.Boolean
addToSymbolCache) [0x00000] in <9e81f04a2ec74d69bcaf40c72e08f334>:0
at QuantConnect.Algorithm.QCAlgorithm.AddSecurity[T] (QuantConnect.SecurityType securityType, System.String ticker, System.Nullable`1[T] resolution, System.String market, System.Boolean fillDataForward,
System.Decimal leverage, System.Boolean extendedMarketHours) [0x000a9] in <ee90c99d38844a78be20c2ceb3056942>:0
at QuantConnect.Algorithm.QCAlgorithm.AddForex (System.String ticker, System.Nullable`1[T] resolution, System.String market, System.Boolean fillDataForward, System.Decimal leverage) [0x00001] in
<ee90c99d38844a78be20c2ceb3056942>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture)
[0x00032] in <b0e1ad7573a24fd5a9f2af9595e677e7>:0
at Initialize
self.symbol = self.AddForex("VFO" in main.py:line 15

This error appears for generated crypto symbols:
20210406 14:49:05.618 ERROR:: During the algorithm initialization, the following exception has occurred: ArgumentException : Symbol can't be found in the Symbol Properties Database: DPA
at QuantConnect.Securities.SecurityService.CreateSecurity (QuantConnect.Symbol symbol, System.Collections.Generic.List`1[T] subscriptionDataConfigList, System.Decimal leverage, System.Boolean
addToSymbolCache) [0x0009f] in <9e81f04a2ec74d69bcaf40c72e08f334>:0
at QuantConnect.Securities.SecurityManager.CreateSecurity (QuantConnect.Symbol symbol, System.Collections.Generic.List`1[T] subscriptionDataConfigList, System.Decimal leverage, System.Boolean
addToSymbolCache) [0x00000] in <9e81f04a2ec74d69bcaf40c72e08f334>:0
at QuantConnect.Algorithm.QCAlgorithm.AddSecurity[T] (QuantConnect.SecurityType securityType, System.String ticker, System.Nullable`1[T] resolution, System.String market, System.Boolean fillDataForward,
System.Decimal leverage, System.Boolean extendedMarketHours) [0x000a9] in <ee90c99d38844a78be20c2ceb3056942>:0
at QuantConnect.Algorithm.QCAlgorithm.AddCrypto (System.String ticker, System.Nullable`1[T] resolution, System.String market, System.Boolean fillDataForward, System.Decimal leverage) [0x00001] in
<ee90c99d38844a78be20c2ceb3056942>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture)
[0x00032] in <b0e1ad7573a24fd5a9f2af9595e677e7>:0
at Initialize
self.symbol = self.AddCrypto("DPA" in main.py:line 15

System Information

OS: Ubuntu 20.04
Checklist

[x] I have completely filled out this template
[x] I have confirmed that this issue exists on the current master branch
[x] I have confirmed that this is not a duplicate issue by searching issues

[x] I have provided detailed steps to reproduce the issue

Make the random data generator add generated crypto symbols to the symbol properties database.
Perhaps the opposite - only generate tickers in the SPDB?

Good idea, that sounds like a more robust solution than my ideas.
That would fix the crypto issue, would you do the same for forex symbols? If not, do you have any thoughts on this?:

Generated forex and crypto tickers should be currency pairs of length 6, not 3 like they are now. Possible solutions:

Generate a random 6-character string for forex and crypto tickers.
Generate a random 3-character string for forex and crypto tickers and append USD to it. This may be better because the default account currency is also USD?


iceberg external table adapts to desc table_name all interface

Enhancement
mysql> desc ex_iceberg_orc_gzip all;
ERROR 1064 (HY000): Unknown storage engine 'ICEBERG'

Support iceberg external table #1030

supported in #6551


What is BGP peering limit for Azure vHub router

[Enter feedback here]
This page is helpful but it doesn't state the max number of BGP peers. If there is no max, it should state so. For example, Azure Route Server has a max of 8 BGP peers per Route Server. Azure vHub has its own hub router. What is the max BGP peers for each hub router?
https://docs.microsoft.com/en-us/azure/virtual-wan/scenario-bgp-peering-hub
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9205b7a3-51f0-70fb-5125-97fba938f421
Version Independent ID: e9d061eb-38a3-808f-a11c-4246289a64c3
Content: About BGP peering with a virtual hub - Azure Virtual WAN
Content Source: articles/virtual-wan/scenario-bgp-peering-hub.md
Service: virtual-wan
GitHub Login: @cherylmc
Microsoft Alias: cherylmc

Hi @JeffGiroux , thanks for your feedback. We will review this and get back to you shortly.

@JeffGiroux , we have assigned the thread lto author for further review.
@cherylmc, I was not able to find the BGP Peers limit on Azure Networking limits page as well.
Kindly check and update

#reasssign:cfields475

#reassign:cfields475

Thank you for you dedication to our documentation.
Unfortunately, we have been unable to review this issue in a timely manner. We sincerely apologize for the delayed response. We are closing this issue. If you feel that the problem persists, please respond to this issue with additional information.
Please continue to provide feedback about the documentation. We appreciate your contributions to our community.
#please-close


WAFv2 Table Populated AssociatedResources from Wrong Service

Describe the bug
When calling the query:
select
name,
associated_resources
from
aws_wafv2_web_acl
Steampipe attempts to call the "listAssociatedResources" action from the "servicecatalog-appregistry" service. giving the error "table 'aws_wafv2_web_acl' column 'associated_resources' requires hydrate data from listAssociatedResources, which failed with error operation error WAFV2: ListResourcesForWebACL, https response error StatusCode: 400, RequestID: 70b443fc-b463-4a75-9e3c-fffa7630a48f, api error AccessDeniedException: You don't have the permissions that are required to perform this operation"
It should just need to call "ListResourceForWebACL" from the "waf" service. I can call this command fine with just my creds however the readonly/securityaudit role I am using does not have servicecatalog-appregistry permissions.
Why is it required that steampipe call a servicecatalog action for this but the same info is gotten via the cli just using a call the waf service?
Steampipe version (steampipe -v)
v0.20.8
To reproduce
test the above query on a user with just the standard AWS readonly perms
Expected behavior
steampipe returns the associated resources for the WebACL

@hrmcardle0, Welcome to Steampipe!
We're sorry to hear that you are experiencing some issues. Before we dig into this, can you please provide a few informations?

What is the version of the AWS plugin you are using?
Can you please provide the list of permissions assigned to the readonly/securityaudit role?
Assuming that you are running it in your local terminal?

Thanks!

Hey, @hrmcardle0  Do you have any update or feed back regarding the above comment?

Hi @hrmcardle0 , we're closing this issue as we haven't heard back from you yet, but if you still see this issue, please let us know. Thanks!


Module evdi not found in directory /lib/modules/6.10.3-arch1-2

Are you using the latest driver? Y

Are you using the latest EVDI version? Y

If you are using a DisplayLink device, have you checked 'troubleshooting'
on DisplayLink's website? Y

Is this issue related to evdi/kernel?

Linux distribution and its version Arch Linux
Linux kernel version 6.10.3-arch1-2
Xorg version (if used)
Desktop environment in use Gnome

Hi,
I have just hit that issue after upgrading my system this morning (Arch Linux, Kernel 6.10.3)
My diplayport monitors are not detected. The journal shows that the displaylink system is down and cannot find evdi.
I have tried to update evdi using the git repo (AUR evdi-git package) without luck...
Aug 12 09:42:27 njord systemd[1]: Starting DisplayLink Manager Service...
Aug 12 09:42:27 njord modprobe[2976]: modprobe: FATAL: Module evdi not found in directory /lib/modules/6.10.3-arch1-2
Aug 12 09:42:27 njord systemd[1]: displaylink.service: Control process exited, code=exited, status=1/FAILURE
Aug 12 09:42:27 njord systemd[1]: displaylink.service: Failed with result 'exit-code'.
Aug 12 09:42:27 njord systemd[1]: Failed to start DisplayLink Manager Service.
lines 4261-4316/4316 (END)

Thanks for your help!

Perhaps the solution in #481 works for you. You have to rebuild the evdi module with dkms.

Thanks @wvdakker. Yes, that fixed the issue.


add more sites to porn list

I found some porn sites that are not in the list:
thelesbianconnection.com
collegesexlives.com
dm.damcdn.net
viplocal-hookups.com
18girlsex.net
teenageslut.net
3dporntubex.info

Bump, @sinfonietta & @clefspeare13 ^

I'll try my best to help research more obscure porn sites to help contribute to the cause ;)

I've added these domains

This is in the latest release.
Closing.

Thank you all for such a prompt response and fix. ;) :+1:


Publish OpenTimelineIO to maven via sonatype

This is a post-documentation of some of the work to get OpenTimelineIO publishing to maven central.
Here was the sonatype issue we used: https://issues.sonatype.org/browse/OSSRH-64444?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=1016349
From @KarthikRIyer, here were the steps we started with for GPG key generation:

sudo apt install gnupg2
gpg2 --gen-key
gpg2 --keyserver hkp://keys.openpgp.org --send-keys {KEY}
Make sure the key can be found on keys.openpgp.org

The key is then exported to be stored as a github secret using: gpg2 --export-secret-keys 'Key ID' | base64 > private.key
Here is the template settings.xml that is added as a secret to our build (replacing username, password and gpg passphrase):
<?xml version="1.0" encoding="UTF-8"?>
<settings
    xmlns="http://maven.apache.org/SETTINGS/1.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
    <pluginGroups></pluginGroups>
    <proxies></proxies>
    <servers>
        <server>
            <id>ossrh</id>
            <username>username</username>
            <password>password</password>
        </server>
    </servers>
    <mirrors></mirrors>
    <profiles>
        <profile>
            <id>ossrh</id>
            <activation>
                <activeByDefault>true</activeByDefault>
            </activation>
            <properties>
                <gpg.executable>gpg2</gpg.executable>
                <gpg.passphrase></gpg.passphrase>
            </properties>
        </profile>
    </profiles>
</settings>

Fixed by #2
Fixed by #6


Marking cells for thumbnailing

This is a tricky issue that has had a simple solution for holoviews: just use the first (or last) displayed cell.
After a conversation with @jsignell, we decided that some of our projects need an extended heuristic (mainly for topic type notebooks):

If a cell uses panel's servable, thumbnail that.
If a bokeh plot has a title matching the notebook title always use that.
If none of the information above exists, thumbnail the last display.

After talking to @jbednar, we decided that this isn't going to be sufficient in general. We considered ways of explicitly marking cells, either with cell metadata or comments.
We came to the conclusion that the comments approach is easiest to manage and we can strip these special comments out from the website build. We have two types of things we can do 1) force inclusion of a thumbnail (i.e whitelist) 2) force exclusion(i.e blacklist).
Provisionally, these could be marked by #+ and #- (unlikely to ever appear in normal code). There are two policies you could use:

Use the heuristic above to automatically include thumbnails (e.g first/last or both first and last) then you can use #+ to add to this set and #- to exclude the first and last if unsuitable.
Get rid of the heuristics and only use either #+ (add thumbnails) or #- (exclude thumbnail, if everything is thumbnailed by default).

I have no strong opinion but my inclination is to keep things as they are now with the current heuristic(s) and then add support for special comment syntax on top. That way everything existing that builds with nbsite is unaffected.

I'm happy with the combination of heuristic plus comments to override when needed, particularly if we can strip out the comments so they don't appear on the website.


Two tests failing to compile with MSVC++ 2017.

I've been poking at the test programs on Windows (Including replacing the current test-high/Makefile with a cmake file (which I can make a pull request for if there's interest), after getting tired of compiling random ones manually), and found two programs that fail to compile: check-pcg128_oneseq_once_insecure and check-pcg128_once_insecure. They both fail with the same error:
(ClCompile target) ->
  c:\users\xxx\source\pcg-cpp\include\pcg_extras.hpp(534): error C2440: '<function-style-cast>': cannot
convert from 'pcg_extras::uint_x4<uint32_t,uint64_t>' to 'delta_t' [C:\Users\xxx\Source\pcg-cpp\test-hig
h\check-pcg128_oneseq_once_insecure.vcxproj]

I haven't looked any deeper into it yet, other than to see thatdelta_t is an iterator difference type.
The good news is that all the tests that do compile pass, though since the size of various structures are different, there are some issues when diffing their output against the expected ones.

And found what appears to be the issue: MSVC won't use any of the uint_x4 conversion operators to turn one into a ptrdiff_t. Added an operator int64_t() method and it works.


[minimal-blog]: Provide ability to turn off Google Analytics Gatsby Plugin

Summary
Some users (me 😇) would like to disable Google Analytics. I use a different service.
Basic example
New gatsby config option maybe titled enable-google-analytics with the default being true to provide backwards compatibility.
Motivation
I host my own analytics service.

Hi!
Google Analytics is not part of the theme, it's installed like any other plugin in the starter for the theme I provide. So you can remove the plugin from the package.json and gatsby-config.js and are good to go :)

@LekoArts
Do you know why this google tag manager script is showing up then?

Repo: https://github.com/aleccool213/blog-4

@LekoArts
Very weird. When I run the site locally, I dont see it either.
It might be my hosting provider putting it in :( Either Netlify or Cloudflare.

Turns out I had a Cloudflare app installed which had this script being injected at the DNS level :(

Well, at least you were able to figure it out :)


Prompt before deleting component

The user will be prompted before deletion of a component, unless
overriden using --force/-f.
Fixes #200

ping @containscafeine


Added Virtualization Setup for Fedora

Pull Request
Added Virtualization Setup for Fedora
Type of Change

[x] New feature

Added Virtualization setup using the default @virtualization group in fedora
Testing
Works Completely
Checklist

[x] My code adheres to the coding and style guidelines of the project.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] My changes generate no errors/warnings/merge conflicts.

@Angxddeep Mark all of these reviews as solved please.

Looks good at the time of review.

Yeah, we need to make it look good, although he's just gonna look at the code


Optimization of NOC Layer's put_object Operation

During the testing process of the isolate-path-env in #159, it has been found that the performance is poor when a group of objects is saved using noc.put_object within the cache.commit stage, especially when the internal write operations in op-env lead to a switch to hub mode for the object-map. This mode switch unavoidably generates a large number of sub object-maps, resulting in a time-consuming commit process.
Therefore, future optimizations should be considered.

Below are a few possible optimization points currently under consideration:

Add batch operations in the noc interface
This is mainly for batch put_object operations, and the corresponding blob and meta layers also need to be optimized for batch processing.

Optimize the blob storage structure
This optimization is not only for improving write performance but also for read and storage improvements. Currently, objects are stored as individual files in the objects directory, which can lead to excessive disk files and performance issues when there are a massive number of objects. Consider using a pack-based storage approach or other similar methods to address this.


[optimize] only prefetch data when hash table was large

What type of PR is this：

[ ] bug
[ ] feature
[x] enhancement
[ ] others

Which issues of this PR fixes ：
Fixes #4856
Problem Summary(Required) ：
query:
 SELECT cab_type, count(*) FROM trips_mergetree2 GROUP BY cab_type;

parallel
baseline
after

1
3852
2759

4
1118
834

8
776
540

16
583
455

The code looks good. Any explaination on this case? Why prefetch works bad on large hash table?

I see the benchmark.

run starrocks_fe_unittest


Missing temperature unit in the performance tab.

Thanks,
I will change it.

It has been fixed.