Add dperny to maintainers

@dperny has been contributing to SwarmKit for several months. Recently he has been working on https://github.com/docker/docker-e2e, a framework for end-to-end testing of Docker, with a particular focus on testing SwarmKit.
Welcome @dperny!

LGTM !

Current coverage is 55.10% (diff: 100%)

Merging #1816 into master will increase coverage by 0.02%

@@             master      #1816   diff @@
==========================================
  Files           102        102
  Lines         17008      17008
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits           9368       9373     +5
  Misses         6478       6478
+ Partials       1162       1157     -5

Powered by Codecov. Last update 5de7335...befc046

🎉

LGTM

@dperny press the button! 🔘 🎉


Implement type inference for DynamicConvOp

Unassigning the ticket, as discussed in https://github.com/openxla/stablehlo/issues/863#issuecomment-1407562475. Will revisit it later on.


More clgs

abstract

This PR goes towards #172.
This PR goes towards #191.
This PR goes towards #193.
This PR goes towards #194.

checklist

[ ] Make sure you read and understood the CONTRIBUTING.md.
[ ] Make sure you applied sufficient labels, milestone and assignee.
[ ] Make sure all TODOs are addressed.
[ ] Make sure there is sufficient documentation.
[ ] Make sure there are sufficient tests.
[ ] Make sure all commits are squashed before merging.
[ ] Make sure to delete the feature branch after merging.

Current coverage is 95.65% (diff: 100%)

No coverage report found for master at e0e7795.

Powered by Codecov. Last update e0e7795...23dcbe5


Add options.domain

I add options.domain param. Look at #4 for details.

So I'd usually add a CNAME file as part of my build — but I can't see any reason not to have the option here too.
I'll just comment on one thing[...]

Thanks for the cleanup too — all eminently sensible changes that, in the time since I wrote this, I've come around to following!

Should I remove var options? Or you do this?

Go ahead and remove it — I can't push to your branch.
It would be fine too if you wanted to change the way options is passed around — I don't mind.

I've removed var options. Is this better?


The trails and year disappear when user clicks on the slider ball again and again

SEVERITY:  Major
MODULE: Bubble Chart
PLATFORM:

Windows 7 / Firefox 44.0
Windows 7 / Chrome 49.0
Windows 7 / Internet Explorer 11

STEPS TO REPRODUCE:

Navigate to the URL: http://static.gapminderdev.org/vizabi/develop/preview
Click on Aruba check box from country
Drag the slider to year 1900
Now drag the slider to the year 2015
Observe the changing year on the label
Click on the slider ball again and again

NOTE: The trails disappear also when user drag the label on the chart at different positions
ACTUAL RESULTS:
The trails and year disappear when user clicks on the slider ball again and again
EXPECTED RESULTS:
The trails should not disappear when user clicks on the slider ball again and again
SCREENSHOT
http://g.recordit.co/Uio3Rw0k4z.gif

resolved in #1765


Routing audio player output as backing track?

Is it possible to play an audio backing track as a source from Windows 10 to a JACK-based online session for others to jam with? I've been trying for hours to configure VLC under Windows 10 to select my JACK interface as its Audio Device for output, but so far only 'Default' and 'Speakers' are offered by VLC's dropdowns. Numerous forum posts refer to a 'vlc-jack-plugin' for this feature but that seems to be Linux only (or 'not Windows', at least). The Sounds settings pages in Windows only offer the same two source options as VLC.
Working from the JACK interface and looking back upstream towards the source, is it possible to select an audio source (E.g. VLC, YT or [coughs] pikaraoke) as Jamtrip's input feed, rather than the microphone or an external interface device?

Maybe I've misunderstood (and haven't had time to try Jamtrip out yet, but it looks cool) but can't you just use Carla etc to route your Jack audio into Jamtrip's sinks as normal?
Then the challenge is just any Windows media player with Jack support (shameless plug: I added Jack support to Quodlibet recently) and then patch this in directly. Unless you want the backing track to appear as a separate participant, that's harder I guess

Maybe I've misunderstood (and haven't had time to try Jamtrip out yet, but it looks cool) but can't you just use Carla etc to route your Jack audio into Jamtrip's sinks as normal?
Then the challenge is just any Windows media player with Jack support (shameless plug: I added Jack support to Quodlibet recently) and then patch this in directly. Unless you want the backing track to appear as a separate participant, that's harder I guess


refactored

Great changes, appreciate the const-correctness. Hope you don't mind me merging these.


Spells Icon Out Of Screen

This issue is present only when the character have quite a bit of spells available to cast, because level 4 spells icons and above appear over the lower level spells, it become impossible to select some of them, as they are being rendered off screen.
The old version of "community expansion" had an option to move spells icons on the right after the third level, preventing this issue

Admetos Char.zip
Here is the screenshot where the level 4 spells are only half rendered, and 2 spells are rendered out of screen entirely, plus the character file, savegame is irrelevant as this problem appear in any condition regardless, as long as the char can memorize this many spells
Id like to clarify that i am playing by enabling all spells for wizard class, and a personal modification that allow this char to keep memorized 80 spells or so at the same time, 72 here. The char level is 7 so to show the problem i encountered while playing a custom campaign, this issue do not appear with the normal memorization limit, as the game simply do not allow to have enough spells active to fill up the screen

Weird Error With screenshot, trying again as zip
Lev4outofscreen.zip

I cannot support you on custom modifications. That screen was never meant to have 80+ spells. Nothing we can do to help here.


Updated vent tests to confirm if triggered

I forgot this can fire multiple events and doesn't return the value. There might have to be a side effect in the event that confirms it was run


Streaming integration API

There's currently no API in ChiselStrike to integrate with streaming systems such as Red Panda or Apache Kafka.

We started this effort in a389809ee4a197138a2d9f016c990532c656cf61, so closing this in favour of more fine-grained tickets.


Remove IsLocal

We keep getting questions like this: https://github.com/aspnet/Hosting/issues/570#issuecomment-171571555
Environments are the correct way to handle these scenarios most of the time.
We should remove IsLocal, and possibly replace it with an extension method that just returns RemoteIP.Equals(LocalIP).
@davidfowl @muratg @blowdart

Yea, kill it IMO

We should remove IsLocal, and possibly replace it with an extension method that just returns RemoteIP.Equals(LocalIP).

If we add it back as an extension method won't be in the same place? It seems like an OK thing to remove from ConnectionInfo though.

Current recommendation is to remove it without adding an extension.

Yup, no extension method. This was redundant anyway.

Fixed via 1f21540fd53a839dcfe7004788abe1e9a1d820a4

I found this useful for authorising a user based on if it is a Local request or not. I am aware that this information (ip address) could be spoofed, but I am happy to accept that risk given that this functionality is in place only when the web application is first installed, and it treats the local user as a local admin whilst they set up the site, and blocks non local requests.
What's the replacement?

IsLocal is more dangerous than it once was because of the prevalence of local reverse proxies (e.g. Nginx, HttpPlatformHandler, AspNetCoreModule, etc.), most requests now appear to be local.
How would your setup process work for a site deployed to a hoster like Azure Web Sites where there is no local access?
Consider using a Staging Environment mode or real credentials based authentication.

@Tratcher understood. If there is a chance of IsLocal no longer being accurate then I don't wish to use it. I'll have to re-think my setup process. :(

So what I've seen working for some folks is

Check for the presence of a marker file or database entry to see if already configured. If present goto 6.
Check for the presence of a first run password in a settings file, if not present throw error with instructions on how to set it. Never set a default one in any installer.
Login page which asks for the setup password
Run setup
Drop the "I've been configured" marker
Profit.

@blowdart haha thanks - I like that suggestion, Team City uses that concept too, it generates an system admin login token which it writes to a local log file. Cheers


Render only first component in slot if render in browser

Version
2.6.9
Reproduction link
https://codepen.io/PNKBizz/pen/zYxYQbr
Steps to reproduce
Just create component with slot and place inside two another components.
What is expected?
Both components rendered
What is actually happening?
Only first component rendered

Don't use self-closing tags in codepen

Ooooh. Sorry )))


Bot should fallback to normal behaviour when under circle limit size

After watching/playing several sessions trying to rescue getting circling around when in circle mode the bot should resume normal when under 5000 points

Guess we need to reset bot.stage
            } else {
+                 bot.stage = 'grow';
                if (bot.snakeLength > bot.opt.followCircleLength) {
                    bot.stage = 'tocircle';
                }

Ah, I start to understand. You mean, when the bot dies and respawns, it is still in toCircle mode.

I took over manual control and drained its score under 5000 and it still tried to circle

2d60618945


Authentication issue

The problem
open(url, domain, login, password) like described in the examples
open("/basic-auth/hello", "", "scott", scottPassword());
$("#greeting").shouldHave(text("Hello, scott:" + scottPassword()));

doesn't work
only following code seems to do the trick:
open(url);
((HasAuthentication)Selenide.webdriver().object()).register(login, password);

environment

Selenide Version:  6.15.0
Chrome: 114.0.5735.111

@TomaszSzyborski What exactly means "doesn't work"?
There is a test in Selenide project, and it works:
https://github.com/selenide/selenide/blob/main/statics/src/test/java/integration/BasicAuthTest.java#L9

@asolntsev my bad, I should be more specific:
in my case. When using BasicAuth from Selenide I end up with Chrome BasicAuth modal opened after opening the browser.
It may be implementation specific - as the auth is needed for one domain that redirects immediately to another, so the only working way for me.
I tried opening: something.domain.com with BasicAuth (the BasicAuth modal does not appear), then opening other.domain.com - the BasicAuth modal says : other.domain.com (the domain.com portion remains the same), but the url in the browser stays something.domain.com
Entering the data manually works as well.

@TomaszSzyborski I think the problem is with domain. You open domain something.domain.com with BasicAuth, but it opens some resource (iframe or image?) on another domain other.domain.com that actually requires BasicAuth. Current Selenide mechanism just adds username+password to current URL (something.domain.com), so it doesn't help with different domains.
But I will try to add your trick with HasAuthentication to Selenide code.

fixed in https://github.com/selenide/selenide/pull/2358


DPL-348 - As a user (Michelle) I would like to decouple tube creation from printing in samples extraction so I can print the barcodes when I need them and ensure that tubes are still created when printing is broken [C=S, V=4]

User story
As a user (Michelle) I would like to decouple tube creation from printing in samples extraction so I can print the barcodes when I need them and ensure that tubes are still created when printing is broken
Who are the primary contacts for this story
Michelle
Acceptance criteria
To be considered successful the solution must allow:

[ ] When tubes are created the labels will not be automatically printed
[ ] There should be a printing button for tube labels

Dependencies
This story is blocked by the following dependencies:

#<issue_no.>
sanger/#<issue_no.>

References
This story has a non-blocking relationship with:

#<issue_no.>
sanger/#<issue_no.>

Additional context
Add any other context or screenshots about the feature request here.

Current print calls:
flowchart
PrintBarcodes#process-->Asset#print
Step#process-->AssetGroup#print
AssetsContoller#print_search-->AssetGroup#print
AssetsController#print-->Asset#print
AssetGroupsController#print-->AssetGroup#print
Asset#print--save-->PrintMyBarcodeJob
AssetGroup#print--save-->PrintMyBarcodeJob

That's five key routes in, and Step#process is the one where we're currently coupling the process to printing. AssetsController#print_search is the route I used yesterday. So clear needs:

Means of easily triggering printing from an activities page. Might be we can hook into the existing AssetGroups#Controller here, although a bit of fine-grained control over what to print wouldn't go amiss.
Step#process is a little more complicated, as we have more users that just those who requested the change. So stripping it out entirely may not be the best option. However we definitely need to make sure that it can't trigger step failure

Set up a pol to work out how to handle the automatic printing: https://linkto.run/r/L504XSQ4

Poll appears to be everyone either wanting to decouple the steps, or not caring. So that makes this simple.


Add hendrikharlichs.de

This PR is:

[x] Adding a new domain
[ ] Updating existing domain size
[ ] Changing domain name
[ ] Removing existing domain from list
[ ] Website code changes (512kb.club site)
[ ] Other not listed

[x] I used the uncompressed size of the site
[x] I have included a link to the GTMetrix report
[x] The domain is in the correct alphabetical order
[x] This site is not an ultra lightweight site
[x] The following information is filled identical to the data file

I confirm that I have read the FAQ section, particularly the two red items around minimal pages and inappropriate content and I attest that this site is neither of these things.

[x] Check to confirm

- domain: hendrikharlichs.de
  url: https://hendrikharlichs.de/
  size: 203
  last_checked: 2023-09-26

GTMetrix Report: https://gtmetrix.com/reports/hendrikharlichs.de/F9LH5XCT/

This PR is also fixing some minor formatting issues (misplaced whitespace and blank lines).

Hendrik! Didn't expect to see you here. 😄 I hope you're well. Thanks for adding your site!

Hi Garrit! I'm good, thank you! In fact, I stumbled upon this project through you and your website! 🙂


Distribution of user classes is not respected and some user classes are just never spawned

Describe the bug
When using LoadTestShape in distributed mode and with multiple user classes having different weights, the distribution of users does not respect the weights (even within some tolerance). Furthermore, users with the smaller weights are often never picked up. The problem appears especially when the LoadTestShape specifies stages with small increments (e.g. 5 users at a time).
Expected behavior
The distribution of users should take into account the overall distribution of users across all workers.
Actual behavior
Some user classes with lower weights are never picked up and the distribution of users does not respect the weights (even within a certain tolerance).
Steps to reproduce
I think the problem is that each worker is responsible for spawning its own users. Consider the following setup:

5 workers
Load test shape that increases the users by 5 at a rate of 1/s each minute until 100 users
3 user classes with weights [35, 55, 10]
Once the test starts, the master will instruct each worker to spawn 1 user every minute. However, the weight_users function will always return the user with the weight of 55.

Possible solutions
I see two aspects that needs to be implemented:

I think that when running in distributed mode, the master runner should instruct the number of each user class to the workers instead of only the number of users. The worker runner would thus spawn the specified users as-is instead of computing the buckets.
The master runner should keep a state of all the running users and their class so that it can spawn the appropriate classes in order to preserve the distribution as much as possible. This state could also serve to solve  https://github.com/locustio/locust/issues/896.

I'm not super familiar with the codebase, but would that make sense? Is there some technical limitation I'm not aware of?
 Also, this file is rendered from a template, so that is why the classes and tasks have generic names.

import json
import os
import random
import uuid
from pathlib import Path

import locust.stats
from essential_generators import DocumentGenerator
from locust import (
    HttpUser,
    LoadTestShape,
    between,
    task,
)

from load_tests.api.common import (
    create_user,
    delete_user,
    get_password,
)

locust.stats.CSV_STATS_INTERVAL_SEC = 2

current_path = Path(os.path.split(__file__)[0])

host = os.environ['API_HOST']

cached_data_file_path = os.environ['CACHED_DATA_FILE_PATH']

gen = DocumentGenerator()

random.seed()

class Test1(HttpUser):

    wait_time = between(5, 10)
    weight = 35
    host = host

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.user_id = None
        self.username = None
        self.password = None
        with open(cached_data_file_path, "rt") as file:
            self.cached_data = json.load(file)

    def on_start(self):
        self.username = f"test-user-{uuid.uuid4()}"
        self.password = get_password()
        self.user_id, user = create_user(self.client, self.username, self.password)

    def on_stop(self):
        if self.user_id is not None:
            delete_user(self.client, self.user_id)
        self.user_id = None
        self.username = None
        self.password = None

    @task(8)
    def test1(self):
        # omitted
        pass

    @task(8)
    def test2(self):
        # omitted
        pass

    @task(8)
    def test3(self):
        # omitted
        pass

    @task(8)
    def test4(self):
        # omitted
        pass

    @task(8)
    def test5(self):
        # omitted
        pass

    @task(8)
    def test6(self):
        # omitted
        pass

    @task(8)
    def test7(self):
        # omitted
        pass

    @task(8)
    def test8(self):
        # omitted
        pass

    @task(8)
    def test9(self):
        # omitted
        pass

    @task(8)
    def test10(self):
        # omitted
        pass

    @task(8)
    def test11(self):
        # omitted
        pass

    @task(12)
    def test12(self):
        # omitted
        pass

class Test2(HttpUser):

    wait_time = between(0, 0.5)
    weight = 55
    host = host

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.user_id = None
        self.username = None
        self.password = None
        with open(cached_data_file_path, "rt") as file:
            self.cached_data = json.load(file)

    def on_start(self):
        self.username = f"test-user-{uuid.uuid4()}"
        self.password = get_password()
        self.user_id, user = create_user(self.client, self.username, self.password)

    def on_stop(self):
        if self.user_id is not None:
            delete_user(self.client, self.user_id)
        self.user_id = None
        self.username = None
        self.password = None

    @task(50)
    def test1(self):
        # omitted
        pass

    @task(50)
    def test2(self):
        # omitted
        pass

class Test3(HttpUser):

    wait_time = between(5, 10)
    weight = 10
    host = host

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.user_id = None
        self.username = None
        self.password = None
        with open(cached_data_file_path, "rt") as file:
            self.cached_data = json.load(file)

    def on_start(self):
        self.username = f"test-user-{uuid.uuid4()}"
        self.password = get_password()
        self.user_id, user = create_user(self.client, self.username, self.password)

    def on_stop(self):
        if self.user_id is not None:
            delete_user(self.client, self.user_id)
        self.user_id = None
        self.username = None
        self.password = None

    @task(100)
    def test4(self):
        # omitted
        pass

class StagesShape(LoadTestShape):
    """
    A simply load test shape class that has different user and spawn_rate at
    different stages.
    Keyword arguments:
        stages -- A list of dicts, each representing a stage with the following keys:
            duration -- When this many seconds pass the test is advanced to the next stage
            users -- Total user count
            spawn_rate -- Number of users to start/stop per second
            stop -- A boolean that can stop that test at a specific stage
        stop_at_end -- Can be set to stop once all stages have run.
    """
    stages = [
        {"duration": 300, "users": 5, "spawn_rate": 1},
        {"duration": 600, "users": 25, "spawn_rate": 1},
        {"duration": 900, "users": 50, "spawn_rate": 1},
        {"duration": 4500, "users": 100, "spawn_rate": 1},
        {"duration": 5400, "users": 1, "spawn_rate": 1},
        {"duration": 6300, "users": 50, "spawn_rate": 1},
    ]

    for previous_stage, stage in zip(stages[:-1], stages[1:]):
        assert stage["duration"] > previous_stage["duration"]

    def tick(self):
        run_time = self.get_run_time()

        for stage in self.stages:
            if run_time < stage["duration"]:
                tick_data = (stage["users"], stage["spawn_rate"])
                return tick_data

        return None

Thanks for this report! I actually think this is an issue with normal user distribution as well (though it is exacerbated by Shapes, because we are spawning few new users at a time). For example, if you have 10 workers and say you want 10 users, you will only ever get one User type (as all workers will start with the first one)
I dont think I'll have time to fix this any time soon, but my suggestion for solution is for every worker to "shift" which users we start with in weight_users (https://github.com/locustio/locust/blob/master/locust/runners.py#L133)
The math in that function is (overly?) complicated, but I think it can be fixed.
Perhaps it is easiest if we first fix this https://github.com/locustio/locust/issues/1601, so there is a consistent ID for each worker (and not just a guid)

I created a draft PR https://github.com/locustio/locust/pull/1621 with some changes that I think would help alleviate the problem without having to change or re-factor too much stuff.

Unfortunately that approach is not good enough. When using a large number of users it might be, but imagine running two types of users and launching two users and sometimes getting 2xUser1, sometimes 1xUser1 + 1xUser2 and sometimes 2xUser2.

For the determinism, I think that initializing the random seed to a constant is sufficient to give repeatable runs.
I don't think that the users of locust expect absolute determinism regarding the spawned users. Even in the doc it says:

If you wish to simulate more users of a certain type you can set a weight attribute on those classes. Say for example, web users are three times more likely than mobile users:

the keyword being "likely" implying a probability.
However, we should guarantee that each user class will be spawned at least once if the number of users is greater or equal to the number of user classes.
I think the solution is for the weight_users function to compute the desired users based on two constraints:

at least one user of each user class must be running at all time (if the number of users is >= to the number of user classes)
the distance between the desired and actual distribution of each user class must be minimal, i.e. math.sqrt(math.fsum(map(lambda x: x**2, (user_class.actual_percentage - user_class.desired_percentage for user_class in user_classes)))). To compute this optimal distribution, we can use a small brute-force algorithm that perform a given number of trials and keep the users that minimize this distance. Each trial would compute the users using the random.choices function.

I wrote some code to test this idea on my side and it seems to give good results. Of course, it is not completely deterministic (unless the seed is fixed), but at least it does not have the problems of the actual implementation. The most difficult part will be to refactor the master and worker runners to have the master decide which users to spawn on each worker.

A constant random seed could be ok, but you'd need to distribute it to all the worker nodes, determinism is super important.
I didnt have time to think about your math there, but I guess it might be good :) Not entirely sold on a brute force algoritm.
Also, it feels a bit heavy handed to just rewrite everything, I think there has been a fair amount of thought put into the current implementation (I didnt write it though)

The brute-force algorithm might not be the smartest approach, although according to my preliminary tests, it converges to a solution quickly. The alternative is to add a dependency such as SciKit and use a real optimization algorithm and I don't know if we want that.
Right now, the idea would be to have the master compute the optimal users distribution at each iteration, then dispatch the users to each worker. The dispatch could take into account the users already running on each worker so that the disruption on each one is minimized (i.e. prevent excessive stop of users). The dispatch will also try to balance the users across the workers as well as ensure the spawn rate (this will fix https://github.com/locustio/locust/issues/896).
I think it's clear the current implementation is somewhat broken and needs to change. Especially regarding handling the new load test shape feature.
I tested some stuff if you want to run it on your machine: https://gist.github.com/mboutet/5047465a315868ac7e7290a354c24bb4.
It's not clean and far from complete but it shows my idea. You can run it and it will print out on the terminal. You can also run it multiple times to see that the distribution is accurate, fairly repeatable and that it guarantees that each user class is present at least once whether the number of users is small or large.
I'm willing to contribute and make all these changes.

Ok, I'm slightly warming up to your idea :)
But what if we could do something much simpler, like always choosing to start/stop the user type with the biggest diff between current and desired distribution (and picking the first one if there are multiple ones with the same diff), and then looping? Or is that not possible for some reason?

Another desirable trait of the algorithm should be to spawn the same distribution of users on all workers (so one worker doesnt get a radically different user distribution than another, if it can be avoided)

Another desirable trait of the algorithm should be to spawn a similar distribution of users on all workers (so one worker doesnt get a radically different user distribution than another, if it can be avoided)

Yes, I agree. And this is something I actually need in my own use cases where some user classes perform quite intensive computation to generate random texts whereas some others are low-cpu users. This will be implemented where the current dispatch logic is:
https://github.com/locustio/locust/blob/30756ce705e9a6abc294c6f264e5148fe71cf4ac/locust/runners.py#L546-L559
This is also where the logic for handling the distributed spawn rate will take place.

But what if we could do something much simpler, like always choosing to start/stop the user type with the biggest diff between current and desired distribution (and picking the first one if there are multiple ones with the same diff), and then looping? Or is that not possible for some reason?

I'm not sure I understand your idea, but I think this is similar to what I'm proposing. In my small PoC, I always consider the current set of users in the calculation of the new set. So, I'm not computing a completely new set of users each time the weight_users is called.
Having the master runner control the distribution will also open the door to presenting stats on the users distribution. This could also be shown in a new tab of the web UI similar to the one that shows the workers.
I'm aware that all of this is quite a big and complicated refactor, but I'll do my best to keep things simple and to not disrupt too much the existing codebase. I believe that if this can be successfully implemented, it will bring locust much more value.

What I mean is instead of choosing a random user, and then checking to see what the size of diff we get, we try adding User1, calculate the diff, try adding User2, calculate the diff, etc and then just pick the one with the minimal diff. So no random selection of Users the way you are currently doing in get_random_weighted_users()
I'm not exactly sure how to scale this for spawning lots of users (we probably dont want to do the above procedure once for each user to spawn), but there should be a way without resorting to random.

@cyberw, I went with your approach, so everything is deterministic. I still have some work to do on my PR, but once it is ready for review, I will remove the "Draft" status.

Cool! I left a few comments on the current version and will be back for more later, once you are finished!

I don't know if it is important but today I have met the same issue but only for FastHttpUsers. HttpUser are working correctly.
locust version: 1.4.1

From my understanding of the codebase, FastHttpUser and HttpUser are not treated differently in regard to the distribution/spawning logic.

I was a bit surprised about this but it is reproducible. If I use HttpUser then it is working correctly but with FastHttpUser only users with first user class are spawned.

/remove-lifecycle stale

/remove-lifecycle stale


How to get Terminal width?

In Ruby, I can use the https://github.com/akr/ruby-terminfo gem to get the character width of my terminal. How can I do this in Crystal?

I think that should be provided by an external library. I don't know if there is one, try searching here.

Sadly, there doesn't appear to be one. I guess I'll have a crack at implementing it myself.

@stugol I don't think you need a library for this. You already can do it, you can access this information through environment variables:
# print width of the current terminal window
puts ENV["COLUMNS"]  # => 142

#  the same, but for hight
puts ENV["LINES"]    # => 38

I guess, the issue can be closed)

No, that doesn't always work. Sometimes the environment variables are not set, and you have to get it some other way.

Ruby is battle tested and there are gems for everything. Crystal is young, and there aren't that many shards, yet. You'll have to roll up your sleeves!

Please open this issue again. The problem cannot be solved by writing Crystal code, as the "magic numbers" needed for the ioctl() call are platform-specific. It is impossible to write a third-party Crystal library to implement this.

@stugol I wonder, why is it impossible? If I understand you correctly, here are plenty of examples of how to write platform-dependent code: https://github.com/manastech/crystal/search?utf8=✓&q=ifdef - good old ifdef.

For example, the code here works perfectly on my Debian system:
module I::Terminal
	lib C
		struct Winsize
			ws_row : UInt16			# rows, in characters */
			ws_col : UInt16			# columns, in characters */
			ws_xpixel : UInt16	# horizontal size, pixels
			ws_ypixel : UInt16	# vertical size, pixels
		end
		fun ioctl(fd : Int32, request : UInt32, winsize : C::Winsize*) : Int32
	end

	def self.get_terminal_size()
		C.ioctl(0, 21523, out screen_size)
		screen_size
	end
end
puts I::Terminal.get_terminal_size.ws_col

But on your system, it may give nonsense values. Try it here for instance. See? Utter madness!

@waterlink: It is impossible because it can be different on different systems, even if they're all 64-bit or all running Linux!
According to stackoverflow: "[TIOCGWINSZ] is a magic constant determined by the system you are running on [...] by the terminal driver."
So it can't be relied upon to remain constant between systems! You need the platform headers specific to that system.

@stugol Do I understand correctly, that values will change depending on the terminal you are running?

It seems like one can achieve that by either:

write small C library (with sane function, that does not accept any magic constants) and call it from crystal + provide a way to build this beast properly (crystal + C code); or
make *.h => *.cr auto-conversion possible and do it during build process.

For instance, Ruby itself, doesn't allow you to do it with standard library, you will have to bring in some dependency (like highline, for example). And all of these libraries, you can bring in to get terminal width, use C extension to do that.

Aforementioned library ruby-terminfo does exactly that: https://github.com/akr/ruby-terminfo/blob/master/terminfo.c
So I suppose, the answer to the original question: do the same thing ruby-terminfo does, write some C code and bind to it.

We have plans to integrate crystal_lib into the language, so that this can be done: you tell the compiler you want to use a constant from a give header, and this is determined at compile time. That way there's no need to use ifdef and the correct value is picked up according to the header for the current OS. The same goes for structs and functions.
But, like everything else, it will take time, and is not the top-priority at the moment.

@waterlink: It is normal for a gem to compile itself when you install it. The gem command exists for this purpose. If Crystal has such a system, I am unaware of it.

A quick search would have revealed:
https://github.com/ysbaddaden/shards/blob/master/SPEC.md#scripts
And a full blown example:
https://github.com/ysbaddaden/shards/issues/73#issuecomment-159206265

Oh I was unaware that shards can do that! That is very good! Thanks!
Von meinem iPhone gesendet

Am 25.01.2016 um 8:17 AM schrieb Julien Portalier<EMAIL_ADDRESS>A quick search would have revealed:
https://github.com/ysbaddaden/shards/blob/master/SPEC.md#scripts
And a full blown example:
ysbaddaden/shards#73 (comment)
—
Reply to this email directly or view it on GitHub.

I was finally able to these values with termbox-crystal which wraps the Termbox C library. You can use the whole library, or just grab the width and height and continue on your own, as I did.

Note of caution: including / using Termbox seems to change terminal modes, such that arrow keys are now received as different bytes. I'm guessing this has to do with all the different possible flags for the terminal mode, but it was extremely confusing to me when my main program and my tiny keyboard input demo program were receiving different bytes / chars from pressing the arrow keys.

print "\x1b[999C\x1b[999B"
print "\x1b[6n"

puts ""
size = String::Builder.new

STDIN.raw do |io|
  io.each_char do |c|
    unless c.nil?
      unless c.ascii_control?
        size << c
        break if c == 'R'
      end
    end
  end

  matches = size.to_s.match(/(\d+);(\d+)/)

  if matches
    _all, rows_s, cols_s = matches
    puts "\n\ncolumns: #{cols_s}, rows: #{rows_s}"
  else
    puts "\n\nCould not determine screen size, exiting"
  end
end

@Blacksmoke16 what don't you like about the process above? (just out of interest)

@stakach This just feels like a hacky way to handle it. Calling top level print and using STDIN.raw. Like what would this do to your logging if you're also using Log that's writing to STDOUT? Esp when there are better ways to go about it.
E.g. https://gist.github.com/Blacksmoke16/fc04a1b3ffe00d89b4237c66c8d94c35 is what I do in Athena Console. Uses LibC on both windows and linux as the primary way, falling back on some common CLI utilities as fallback.

Thanks for the link to your code, definitely agree your method is better.
Personally I think it should be part of the standard lib as it is in Node.js:

and python:
https://www.w3schools.com/python/ref_os_get_terminal_size.asp

@straight-shoota @ysbaddaden worth considering re-opening this issue?

I think this could be a valuable feature to have in the standard library. Conceptually, it's similar to the Color library, so it would fit in next to that.
Not saying this is definitely getting accepted, but we can keep discussing the possibility and figure out the details.


KeryError from numentries when branch is missing

when ise on a root file w/o the requested branch, a KeyError is rised.
Here's an example with stack trace.
In [21]: nentries = uproot.numentries(dsLFNs, "Events", total=False )
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-21-e1e5fb76980a> in <module>()
----> 1 nentries = uproot.numentries(dsLFNs, "Events", total=False )

/uscms/home/belforte/.local/lib/python2.7/site-packages/uproot/tree.pyc in numentries(path, treepath, total, localsource, xrootdsource, httpsource, executor, blocking)
   1592     else:
   1593         paths = [y for x in path for y in _filename_explode(x)]
-> 1594     return _numentries(paths, treepath, total, localsource, xrootdsource, httpsource, executor, blocking, [None] * len(paths))
   1595
   1596 def _numentries(paths, treepath, total, localsource, xrootdsource, httpsource, executor, blocking, uuids):

/uscms/home/belforte/.local/lib/python2.7/site-packages/uproot/tree.pyc in _numentries(paths, treepath, total, localsource, xrootdsource, httpsource, executor, blocking, uuids)
   1629     if executor is None:
   1630         for i in range(len(paths)):
-> 1631             _delayedraise(fill(i))
   1632         excinfos = ()
   1633     else:

/uscms/home/belforte/.local/lib/python2.7/site-packages/uproot/tree.pyc in _delayedraise(excinfo)
     94         cls, err, trc = excinfo
     95         if sys.version_info[0] <= 2:
---> 96             exec("raise cls, err, trc")
     97         else:
     98             raise err.with_traceback(trc)

/uscms/home/belforte/.local/lib/python2.7/site-packages/uproot/tree.pyc in fill(i)
   1618                 source = file._context.source
   1619                 file._context.classes["TTree"] = _TTreeForNumEntries
-> 1620                 out[i] = file[treepath]._fEntries
   1621                 uuids[i] = file._context.uuid
   1622             except:

/uscms/home/belforte/.local/lib/python2.7/site-packages/uproot/rootio.pyc in __getitem__(self, name)
    234
    235     def __getitem__(self, name):
--> 236         return self.get(name)
    237
    238     def __len__(self):

/uscms/home/belforte/.local/lib/python2.7/site-packages/uproot/rootio.pyc in get(self, name, cycle)
    354                 return last.get()
    355             elif cycle is None:
--> 356                 raise KeyError("not found: {0}".format(repr(name)))
    357             else:
    358                 raise KeyError("not found: {0} with cycle {1}".format(repr(name), cycle))

KeyError: "not found: 'Events'"

I'm looking at the file and I see logic for dealing with the problem that you have. I think this was fixed in #199. Do you have uproot 3.3.1 or later?
If not, could you send me the file that breaks it? This is not an unimplemented feature, but it might be a broken one.

Indeed I pip-installed a few days ago and have this:
In [171]: uproot.file
Out[171]: '/uscms/home/belforte/.local/lib/python2.7/site-packages/uproot/init.pyc'
In [172]: uproot.version
Out[172]: '3.3.0'
will look into upgrading to 3.3.1+ and let you know

That should be it, then. Issue #199 was asking for exactly this feature—skipping files with missing TTrees—and at the end of that conversation, I said that I released it as 3.3.1. So it looks like you're exactly one version too early for this feature.

I should still document it, though. That's on my to-do list!

I have moved to uproot 3.4.8 and confirm that it works, returning 0 entries for the missing tree:
In [9]: ne=uproot.numentries(f,'Events',total=False)

In [10]: ne
Out[10]:
OrderedDict([('/eos/uscms/store/group/lpcbacon/15/WplusH_HToCC_WToQQ_M125_13TeV_powheg_pythia8/WplusH_HToCC_WToQQ_M125_13TeV_powheg_pythia8_48.root',
              0)])

In [11]:

Great!

I should still document it, though. That's on my to-do list!

It is done: https://uproot.readthedocs.io/en/latest/opening-files.html#uproot-tree-numentries


don't ignore README.md on .gitattributes

while it is okay to ignore the docs folder with all the detailed information, I think a README file is a bit too essential to exclude.
see https://github.com/aidantwoods/SecureHeaders/commit/7c64bd4fe4c3a4a3cfbf3cae712489f2cce5de8d#r32537766

Seems sensible :)
Tests are failing, but that's not to do with the change (obviously :) ). I think this error isn't reconcilable with PHP: https://travis-ci.org/aidantwoods/SecureHeaders/jobs/500603956#L509 using & in a by reference arg will cause an error (unfortunately) in PHP, but HHVM requires it. Guess I'll have to remove HHVM from the tests.


Use GNUInstallDirs to allow changing target directories

Multi-lib distributions need to change the libdir
which is only portably possible with GNUInstallDirs.
Multi-arch distributions such as Debian and Exherbo
need to be able to change the bindir.

I have no opinion either way, but wouldn't cmake 2.8.5 be sufficient rather than bumping the minimum requirement to 3.0 (granted both are not brand new) ?

@martin-frbg GNUInstallDirs was only added with 3.0. Some packages tend to bundle the upstream cmake file, but imo this is less than optimal, as it will lead to bitrot. Debian includes 3.0 and thus can deal with this build system. I know that this is painful for some users, but not having to hack the build system to finally support changing all paths is a bigger win in my book.

https://cmake.org/Wiki/CMake_Version_Compatibility_Matrix/StandardCMakeModulesFindL
has it as a feature of 2.8.5 though it could be that early implementations were flawed ?
I do agree that making it easier to change the default locations would be an advantage, at least as long as the defaults are at least as "sane" as the current ones. (Found reference to one project that dropped it claiming that it changed their "lib" default to something like lib/x86_64-linux-gnu on at least ubuntu 14)

@martin-frbg changed it to 2.8.5 after checking that the module was indeed only integrated with 2.8.5. I have pushed GNUInstallDir into a number of projects and haven't seen any issues yet.

It is very dangerous to install local build outside /usr/local (see FHS)

@brada4 it is not the job of the build system to prevent users from doing stupid things. The defaults will install in a FHS compatible manner to /usr/local.


RFP30N06LE - discontinued, here's what I was suggested

Hello everyone,
Looking for a RFP30N06LE, I found out by digikey that it's discontinued
https://www.digikey.ca/en/products/detail/onsemi/RFP30N06LE/458769
Digikey offer me these as a substitues
HUF76423P3
STP55NF06
STP36NF06L (back order)
Is any of these a good substitute?
Thank you

Hello everyone,
Looking for a RFP30N06LE, I found out by digikey that it's discontinued https://www.digikey.ca/en/products/detail/onsemi/RFP30N06LE/458769
Digikey offer me these as a substitues
HUF76423P3 STP55NF06 STP36NF06L (back order)
Is any of these a good substitute?

You're honestly better off trying a 2n7000 in TO-92 package and seeing how you go.  It's what I'm about to recommend.

ok, I was reading the doc as last resort scenario to use that

ok, I was reading the doc as last resort scenario to use that

Yeah we're about to wrap this up and change all the recommendations on the main page.  Apologies, it's a really active project right now.

oh, so since I'm starting building very soon, can you "spoil" me the change so I get to the new version?

any of the BS170 in an equivalent package should work.  Its a better choice than RFP30N06LE to begin with for this application.  Much lower CISS, similar Vgs(th) to the 2n7000, and even cheaper.

oh, so since I'm starting building very soon, can you "spoil" me the change so I get to the new version?

Main page updated.  I'm just recommending 2x 2n7000 for through-hole designs.  SMD obviouly will stick with Paul Wieland's chosen components (2n7002 for RX and AO3400A for TX).


USD->EUR not found on any pair converter plugin

Hello! Great tool! I'm using binance plugin and it's working fine but i've always this errror, I think it's clear but i don't know howto solve (I live in spain and we use EUR)
Regards

Are you using the CCXT pricing plugin?

Yes, i've check with binance lock, general conf and without any converter (default converter). I use -s option also.
My conf.
Data loaders
[dali.plugin.input.rest.binance_com sergi]
account_holder = sergi
api_key = XXXXXXXXX
api_secret = XXXXXXXXXXX
native_fiat=EUR
[dali.plugin.pair_converter.ccxt]
historical_price_type = nearest
default_exchange = Binance.com
google_api_key = XXXXXXXXX

The complet error is
ERROR: Fatal exception occurred:
Traceback (most recent call last):
File "/Users/sergi/crypto/dali-rp2/src/dali/dali_main.py", line 193, in _dali_main_internal
resolved_transactions: List[AbstractTransaction] = resolve_transactions(transactions, dali_configuration, args.read_spot_price_from_web)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sergi/crypto/dali-rp2/src/dali/transaction_resolver.py", line 259, in resolve_transactions
transaction = _convert_fiat_fields_to_native_fiat(transaction, global_configuration)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sergi/crypto/dali-rp2/src/dali/transaction_resolver.py", line 187, in _convert_fiat_fields_to_native_fiat
raise RP2RuntimeError(f"Conversion rate for {transaction.timestamp_value}:{from_fiat}->{to_fiat} not found on any pair converter plugin")
rp2.rp2_error.RP2RuntimeError: Conversion rate for 2023-08-29 23:05:58+00:00:USD->EUR not found on any pair converter plugin
INFO: Log file: ./log/rp2_2023_10_03_12_31_35_596868.log
INFO: Generated output directory: output
INFO: Done

If i change
native_fiat=EUR => native_fiat=USD
I have the same error

@lockevod this should have been resolved with #211 . Can you give it a try and let us know if you are getting correct pricing?

I'm closing this as resolved. If the problem crops again, let use know.


when Train on DanceTrack train set ,I  can't find the exps/example/dancetrack/yolox_x.py？

you should create one and got yolox_x.py here[https://github.com/Megvii-BaseDetection/YOLOX/tree/0.1.0/yolox]


Remove dotfiles from build files in CMakeLists.txt

Avoids including unwanted files that could appear in a developer
workspace (e.g. auto-generated files in "dot" folders).

Concrete example: I noticed that when configuring the cquery language server for Cartographer. The corresponding vscode extension generates in a .vscode folder with header files that mess up the CMake build if the current brute-force glob is used.
Since most IDEs generate some kind of metadata folder in the root dir, this could affect also other developers at some point. Therefore: glob a little bit more specific.

Any idea why the build only fails on "Jessie"? https://travis-ci.org/googlecartographer/cartographer/builds/340585647

The build fails iff BUILD_GRPC is enabled (which is currently tested only on one build server).
../cartographer_grpc/framework/rpc_handler_interface.h:23:27: fatal error: grpc++/grpc++.h: No such file or directory #include "grpc++/grpc++.h"

Ah ok, didn't know it's only on one server. I'll try to reproduce on my machine.

I just ran this and it worked fine:
catkin build cartographer --force-cmake -DBUILD_GRPC=true -v --pre-clean > build.txt
grep grpc build.txt > build_grpc_grep.txt

The logs show that grpc was built successfully, so at least with this build command it doesn't look like CMake is the issue? You probably use another build config, or there's sth. wrong with the Jessie environment? Hard to say.
build_grpc_grep.txt
build.txt

Drive by comment: I don't think the Jessie build bot has gRPC installed.
On Tue, 13 Feb 2018, 12:14 Michael Grupp<EMAIL_ADDRESS>wrote:

I just ran this and it worked fine:
catkin build cartographer --force-cmake -DBUILD_GRPC=true -v --pre-clean > build.txt
grep grpc build.txt > build_grpc_grep.txt
The logs show that grpc was built successfully, so at least with this
build command it doesn't look like CMake is the issue? You probably use
another build config, or there's sth. wrong with the Jessie environment?
Hard to say.
build_grpc_grep.txt
https://github.com/googlecartographer/cartographer/files/1719652/build_grpc_grep.txt
build.txt
https://github.com/googlecartographer/cartographer/files/1719653/build.txt
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/googlecartographer/cartographer/pull/897#issuecomment-365235752,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AX4C8c1U6iADklzMtsBGUcPl4eYNTEmJks5tUW6pgaJpZM4SChM4
.

I correct myself, our Jessie build server should not compile or require the grpc parts. It builds the default config.
Just an idea, does cartographer*/*.cc recurse multiple directories?

Just an idea, does cartographer*/*.cc recurse multiple directories?

It has to, otherwise subdirs like sensor, mapping etc. wouldn't be built. Also my own build would have failed.

I correct myself, our Jessie build server should not compile or require the grpc parts. It builds the default config.

Note (to myself): If -DBUILD_GRPC is false, the gRPC dirs should be removed from the list by this logic. Check.
If all build environments don't build gRPC, then this is still strange that it (unintentionally) builds gRPC support only on Jessie.

Just tried to hide my gRPC files in order to reproduce that:
sudo mv /usr/local/include/grpc /usr/local/include/no-grpc

# this fails (as expected)
catkin build cartographer --force-cmake -DBUILD_GRPC=true -v --pre-clean

# this works (as expected)
catkin build cartographer --force-cmake -DBUILD_GRPC=false -v --pre-clean

Another possibility might be that the glob command behaves differently across the CMake versions:

Jessie: 3.0 (https://packages.debian.org/jessie/cmake)
Stretch: 3.7
Xenial (like my PC): 3.5.1
Trusty: 2.8.12

Other than that, I have no idea what could be wrong.

Some CMake debugging tips:

you can always do traditional printf-style debugging with message(STATUS ...)
make a private github fork, go to www.travis-ci.com (not .org!) and start a debug build (this feature is not available for public repos). It provides you with an ssh shell to the debug build VM instance, where you can manually start a travis build and snoop around when it breaks.

I finally took the time today to continue working on this and came with a better approach (in my opinion). Now, all dotfiles are removed from the glob'd source files in a separate step and it's clearer what's going on. More importantly, it builds now on every CI pipeline :tada:
@gaschler PTAL

@wally-the-cartographer merge

Merge requested by authorized user gaschler. Merge queue now has a length of 1.


site: Adds Rust notes

This consolidates rust notes, at least a start of them. cc @clarkmcc

https://deploy-preview-764--wazero.netlify.app/
I'll plan to merge this tomorrow around lunch asia time after any feedback.

ok wove in TinyGo feedback from @aykevl @dgryski @dkegel-fastly @mathetake and jimmysl lee

ok I think I got those comments in, but yell otherwise.

thanks folks!


python312Packages.pymatgen: 2024.7.18 → 2<IP_ADDRESS>

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

I updated pymatgen to the most recent version to fix the broken package.
Upstream now builds with numpy2, but I added a fix to build with numpy1. All tests are passed.
I am not sure, what the current approach to numpy2 is in nixpkgs. Input would be appreciated.

Add a :+1: reaction to pull requests you find important.

also there is similar update on https://github.com/NixOS/nixpkgs/pull/345326
maybe target python-update branch ?

@mweinelt can you update 611a81b5f916340d70f949b976876750524b3ad9 commit message

@mweinelt ah never mind, i see you already did that


Made headers optional

Proposed changes
Made headers optional for requests that don't need authentication headers to work properly.
Types of changes

[ ] Bugfix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Update (if none of the other choices apply)

Checklist

[x] Lint and unit tests pass locally with my changes
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added necessary documentation (if appropriate)
[ ] Any dependent changes have been merged and published in downstream modules

Please revise because test in convocatory register failed


Initial work for replacing P5.js

This replaces all calls to P5.js-library. I tried my best to leave everything backwards compatible, but something might have slipped past me.
Changes:

Drawing now uses Canvas API
Mouse and keyboard input use addEventListener
"Game loop" is implemented using requestAnimationFrame

I tried to re-use already existing code as much as I could, notable exception being handling mouse input.

Thanks for opening this - I've removed P5 in #28


Fix UEFI build on FreeBSD by not including system headers

CLA: trivial

Why is FreeBSD defined when the target is UEFI?

[bjk@kduck ~]$ uname
FreeBSD
[bjk@kduck ~]$ clang -x c /dev/null -dM -E|grep FreeBSD
#define __FreeBSD__ 13
#define __FreeBSD_cc_version 1300000
#define __VERSION__ "4.2.1 Compatible FreeBSD Clang 6.0.1 (tags/RELEASE_601/final 335540)"

Can you build for UEFI on other operating systems? Will they hit the same issue? I imagine that's what @kroeckx is trying to ask.

Can you build for UEFI on other operating systems? Will they hit the same issue? I imagine that's what @kroeckx is trying to ask.

Oh. Yes, the TianoCore EDK2 project builds on macOS, Windows and Linux and have successfully built with the latest OpenSSL 1.1.1 update. So it appears that other operating systems build fine, without hitting this issue.

[master 1fa90bb3a9]
[OpenSSL_1_1_1-stable 444ec8d5e7]


[2.6.0] Adds a breaking change when using single quotted vars

[2.5] foo='{"bar":1}' => parses as foo = {"bar":1}
[2.6] foo='{"bar":1}' => parses as foo = '{"bar":1}'
When using json decode the firsts works fine but when using 2.6 fails with errorno 4 "Malformed JSON"
this added a lot of troubles on our pipeline

What about 2.6.1 please?

I'm reasonably sure the latest 2.x and 3.x releases don't suffer this bug. Please let me know if upgrading to 2.6.1 from 2.6.0 doesn't fix your issue.


fix (update): dynamically get pivot accessor when saving n-n relations

WHY
Minor bug fix
BEFORE - What was wrong? What was happening before this PR?
Here is the bug report
AFTER - What is happening after this PR?
Bug seems to go away :)
HOW
How did you achieve that, in technical terms?
Tested on local machine, followed error traces.
Is it a breaking change?
Shouldn't be, it's quite a small change
How can we test the before & after?
??

Excellent collaboration here guys. Thanks @singeryo for the PR, and @pxpm for the review. I've merged it, it'll be available later today after I merge a few more PRs and tag a patch release.
Keep 'em coming. Cheers!


1.7.0 - Setup 2FA - Invalid Barcode - Google Authenticator

Is it a duplicate question?
No
Describe the bug
Invalid barcode error when setting up 2FA using the Google Authenticator app on IOS.
Error message: The barcode 'REDACTED' is not a valid authentication token barcode.
To Reproduce
Steps to reproduce the behavior:
Go to Settings > 2FA Settings > Open Google Authenticator App (IOS) > + Sign > Scan a QR code > Scan QR code from Uptime-Kuma > Error message is shown.
Expected behavior
The barcode is accepted and a time-based token is generated.
Info
Uptime Kuma Version: 1.7.0-debian
Using Docker?: Yes
Docker Version: 20.10.8
Node.js Version (Without Docker only): N/A
OS: Raspbian GNU/Linux 10 (buster)
Browser: Firefox 92.0.1
Screenshots
N/A
Error Log
N/A
Docker: docker logs <container id>
PM2: ~/.pm2/logs/ (e.g. /home/ubuntu/.pm2/logs)
N/A

Found related issue, it is a iOS' Google Authenticator only problem.
https://stackoverflow.com/questions/29173288/google-authenticator-on-apple-devices-certain-secrets-are-not-valid
Sometimes I am frustrated, why the Android's Google Authenticator is working and the iOS one is not working. It is so inconsistent.

Should be fixed, but I cannot verify it as I don't have a iOS device.
You can try it in the next version or try nightly version now

I will test and get back to you 🙂


feat: try to inline brillig calls with all constant arguments

Description
Problem
Resolves #6453
Summary
Alternative to #6466
For now this PR just finds the brillig calls with all constant values, creates a VM for each, executes them, and if we can get some output values, the call results are replaced with those values.
Pending:

[ ] If we managed to replace some calls, do all SSA optimization passes again as the new values could help further optimize SSA, then check if we can find calls with constant values, and repeat until we can't optimize anymore.

Additional Context
Documentation
Check one:

[x] No documentation needed.
[ ] Documentation included in this PR.
[ ] [For Experimental Features] Documentation to be submitted in a separate PR.

PR Checklist

[x] I have tested the changes locally.
[x] I have formatted the changes with Prettier and/or cargo fmt on default settings.

I think putting it in instruction simplification may be a bit OTT for now, maybe within the constant folding pass would be a good middle step.
The reason why I don't think it should be in instruction simplification is that we want to only do codegen once. With this we'd need to pass in this big structure of all of the functions whenever we push in any instruction even if there isn't a function call.

As discussed with Tom on our 1 on 1, I moved the logic to constant folding, then we do another pass of DIE. I still need to continue refactoring a bit because I need to pass a mutable reference in a loop and I'm having some trouble doing that.

Sorry, I removed from the queue because contracts in Aztec-Packages break. I'll soon push a fix for this, though I'd like to refactor a bit more to avoid always having to pass &mut brillig_functions_we_could_not_inline (ideally I'd pass that as a property of BrilligInfo but I'm having a small fight with the borrow checker (understandably because I'm still not super familiar with it)).

I'm not sure how to refactor the code to avoid passing &mut brillig_functions_we_could_not_inline everywhere.
Alternatively, we could have a pass after this one that removes unused brillig functions. It could do that by starting with a set of all brillig functions, and removing from that set whenever we find a call to one of those functions. It would be slightly slower than doing it in one go, but will surely lead to simpler code. What do you think?
I'm also not sure about doing constant folding plus this new thing in one go, it makes the code a bit more complex... though looking at the optimization results it might be worth it.
That said, I'm running time nargo compile --silence-warnings --inliner-aggressiveness 0 in noir-contracts and it's been running since a few minutes ago... it might be that running all of these functions through the VM slows down things a lot.

Interesting... Seems like putting this in the constant folding pass allows us to benefit in brillig functions as well. This makes sense as when we're linking brillig functions together we don't do any constant folding between them so this ends up acting as a simple case of that.

I pushed another commit and this is now ready for review.
By the way, the nargo compile above took 29 minutes on my machine, on master. I'll leave it running on this branch to see if it takes much longer... I think previously nargo compile took around 1 minute, not sure if something on our SSA optimizations changed, or it's just that now we are passing --inliner-aggressiveness 0 in bootstrap.

Yeah, nargo compilation in aztec-packages is pretty dire atm. A lot of that time is spent inside of the DIE pass, my aim is to get these two codebases back into one and then we can look into it more deeply.

Alternatively, we could have a pass after this one that removes unused brillig functions. It could do that by starting with a set of all brillig functions, and removing from that set whenever we find a call to one of those functions. It would be slightly slower than doing it in one go, but will surely lead to simpler code. What do you think?
I'm also not sure about doing constant folding plus this new thing in one go, it makes the code a bit more complex... though looking at the optimization results it might be worth it.

I'd say lets do this in iterations. Best to create some new issues for followups than do mega-PR.

It seems the slowness comes from MergeExpressionsOptimizer::eliminate_intermediate_variable: a lot of BTreeMap clones are happening, or at least that's what I get when I sample the process.

It seems the merge with master broke this PR. Specifically this commit it seems: https://github.com/noir-lang/noir/commit/7dd71c15cbcbf025ba049b506c94924903b32754#diff-7c64342380b96ff80bc78d976a826185fbf2f08c0a10d77318dde72f697cb3b7
Something in the blackbox solver changed that's making some tests crash:
thread 'run_stdlib_tests::true_i64_min_expects' panicked at /home/runner/work/noir/noir/acvm-repo/brillig_vm/src/black_box.rs:160:62:
called `Result::unwrap()` on an `Err` value: MismatchedBitSize { value_bit_size: 254, expected_bit_size: 1 }

It seems the merge with master broke this PR.

This is potentially from my PR here https://github.com/noir-lang/noir/pull/6547/files. We're shoving an EmbeddedCurvePoint into an array type so that last bool must be causing a problem.

It seems the slowness comes from MergeExpressionsOptimizer::eliminate_intermediate_variable: a lot of BTreeMap clones are happening, or at least that's what I get when I sample the process.

ooooof, yeah that function is chewing through clones like candy. I've made a PR here to remove some of those.

Nice! With that PR it now takes 2 minutes instead of 29! 🎉

This is potentially from my PR here https://github.com/noir-lang/noir/pull/6547/files. We're shoving an EmbeddedCurvePoint into an array type so that last bool must be causing a problem.

It seems the error happens even without that change. What I'm seeing is that the third argument we get is sometimes a U1, sometimes a Field so it explodes when it's a Field. It gets fixed if we handle both cases, but I'll see if I can find why we are getting different types (maybe you know).

It's strange because the signature of the blackbox function (if I'm understanding things correctly) is returning [Field; 3], so I think getting a field as the third argument is good and the U1 is bad, but not sure:
https://github.com/noir-lang/noir/blob/68c32b4ffd9b069fe4b119327dbf4018c17ab9d4/noir_stdlib/src/embedded_curve_ops.nr#L114-L118

It seems after the last merge a call to multi_scalar_mul now has all constants, when before that merge it didn't, so now it's trying to run that and failing. So maybe the bug was always there, or maybe it's not a bug and it's how I'm setting up the VM and calldata.

It seems it might be a bug with inlining. I'm trying this code:
use std::{embedded_curve_ops::{EmbeddedCurveScalar, multi_scalar_mul}, hash::derive_generators};

fn main(x: Field) {
    let mut points = [EmbeddedCurveScalar { lo: 0, hi: 0 }; 1];
    let generators = derive_generators("DEFAULT_DOMAIN_SEPARATOR".as_bytes(), 0);
    let _ = multi_scalar_mul(generators, points);
}

Here's what happens to the SSA at some point:
After Resolving IsUnconstrained:
brillig(inline) fn main f0 {
  b0(v0: Field):
    v2 = make_array [Field 0, Field 0] : [(Field, Field); 1]
    inc_rc v2
    v3 = allocate -> &mut [(Field, Field); 1]
    store v2 at v3
    v19 = make_array [u8 68, u8 69, u8 70, u8 65, u8 85, u8 76, u8 84, u8 95, u8 68, u8 79, u8 77, u8 65, u8 73, u8 78, u8 95, u8 83, u8 69, u8 80, u8 65, u8 82, u8 65, u8 84, u8 79, u8 82] : [u8; 24]
    v22 = call f1(v19, u32 0) -> [(Field, Field, u1); 1]
    inc_rc v22
    v23 = load v3 -> [(Field, Field); 1]
    v25, v26, v27 = call f2(v22, v23) -> (Field, Field, u1)
    return
}
brillig(inline_always) fn derive_generators f1 {
  b0(v0: [u8; 24], v1: u32):
    call assert_constant(v0)
    v4 = call derive_pedersen_generators(v0, v1) -> [(Field, Field, u1); 1]
    return v4
}
brillig(inline) fn multi_scalar_mul f2 {
  b0(v0: [(Field, Field, u1); 1], v1: [(Field, Field); 1]):
    v3 = call multi_scalar_mul(v0, v1) -> [Field; 3]
    inc_rc v3
    v5 = array_get v3, index u32 0 -> Field
    v7 = array_get v3, index u32 1 -> Field
    v9 = array_get v3, index u32 2 -> Field
    v10 = truncate v9 to 1 bits, max_bit_size: 254
    v11 = cast v10 as u1
    return v5, v7, v11
}

After Inlining (1st):
brillig(inline) fn main f0 {
  b0(v0: Field):
    v2 = make_array [Field 0, Field 0] : [(Field, Field); 1]
    inc_rc v2
    v3 = allocate -> &mut [(Field, Field); 1]
    store v2 at v3
    v19 = make_array [u8 68, u8 69, u8 70, u8 65, u8 85, u8 76, u8 84, u8 95, u8 68, u8 79, u8 77, u8 65, u8 73, u8 78, u8 95, u8 83, u8 69, u8 80, u8 65, u8 82, u8 65, u8 84, u8 79, u8 82] : [u8; 24]
    v23 = make_array [Field<PHONE_NUMBER>078719075161482178784387565366481897740339799480980287259621149274, Field -9903063709032878667290627648209915537972247634463802596148419711785767431332, u1 0] : [Field; 3]
    inc_rc v23
    v24 = load v3 -> [(Field, Field); 1]
    v26, v27, v28 = call f1(v23, v24) -> (Field, Field, u1)
    return
}
brillig(inline) fn multi_scalar_mul f1 {
  b0(v0: [(Field, Field, u1); 1], v1: [(Field, Field); 1]):
    v3 = call multi_scalar_mul(v0, v1) -> [Field; 3]
    inc_rc v3
    v5 = array_get v3, index u32 0 -> Field
    v7 = array_get v3, index u32 1 -> Field
    v9 = array_get v3, index u32 2 -> Field
    v10 = truncate v9 to 1 bits, max_bit_size: 254
    v11 = cast v10 as u1
    return v5, v7, v11
}

In particular, this instruction after inlining:
    v23 = make_array [Field<PHONE_NUMBER>078719075161482178784387565366481897740339799480980287259621149274, Field -9903063709032878667290627648209915537972247634463802596148419711785767431332, u1 0] : [Field; 3]

it holds [Field, Field, u1] but the type says [Field; 3], and that's where it ends up breaking (it should be [(Field, Field, u1); 1].
Now I'll check the inlining pass to see if I can find something there...

Found it!
https://github.com/noir-lang/noir/blame/7658c172a2a9725ff90e12b9a525e3b488c039f6/compiler/noirc_evaluator/src/ssa/ir/instruction/call.rs#L809
I'll send a separate PR to fix this.

(ah, the link is to a git blame, but the code was already wrong before that commit... I didn't intend to blame anyone :-))

This is again ready to review.

LGTM


chore: upgrade to ReScript v12 and Core 1.6.1

This upgrades to the latest pre-release for ReScript v12, which is pretty much stable at this point. Not sure if you want to wait until v12 is stable, or release a next tag of this package.
I also updated to the latest version of Core and removed references to Belt and Js.

Thank you for the PR! 🎉
The upgrade to the latest ReScript v12 pre-release looks great, especially if it's already quite stable.


[SPARK-44838][SQL] raise_error improvement

What changes were proposed in this pull request?

Extend the raise_error() function to a two-argument version:
raise_error(errorClassStr, errorParamMap)
This new form will accept any error class defined in error-classes.json and require Map<String, String> to provide values for the parameters in the error classes template.
Externally an error raised via raise_error() is indistinguishable from an error raised from within the Spark engine.
The single-parameter raise_error(str) will raise USER_RAISED_EXCEPTION (SQLSTATE P0001 - borrowed from PostgreSQL).
USER_RAISED_EXCEPTION text is: "" which will be filled in with the str - value.
We will also provide spark.sql.legacy.raiseErrorWithoutErrorClass (default: false) to revert to the old behavior for the single-parameter version.
Naturally assert_true() will also return USER_RAISED_EXCEPTION.
Examples
SELECT raise_error('VIEW_NOT_FOUND', map('relationName', '`v1`');
  [VIEW_NOT_FOUND] The view `v1` cannot be found. Verify the spelling ...

SELECT raise_error('Error!');
  [USER_RAISED_EXCEPTION] Error!

SELECT assert_true(1 < 0);
 [USER_RAISED_EXCEPTION] '(1 < 0)' is not true!

SELECT assert_true(1 < 0, 'bad!')
  [USER_RAISED_EXCEPTION] bad!

Why are the changes needed?

This change moves raise_error() and assert_true() to the new error frame work.
It greatly expands the ability of users to raise error messages which can be intercepted via SQLSTATE and/or error class.
Does this PR introduce any user-facing change?

Yes, the result of assert_true() changes and raise_error() gains a new signature.
How was this patch tested?

Run existing QA and add new tests for assert_true and raise_error
Was this patch authored or co-authored using generative AI tooling?

No

@cloud-fan I've added QA and addressed your initial comments. This is ready for final review.

@cloud-fan It all green! ... ship-it.... :-)

@cloud-fan All comments addressed

unfortunately, there are conflicts now...

Thanks, merging to master


Unset pbs_snapshot_vm_count_list

The variable pbs_snapshot_vm_count_list would not be reset during the loop and therefore the duplicate metrics would be added.
Fixed #2

great catch @ntimo thanks for fixing it!


Google Drive Mount oauth2 token expired

What is your rclone version (eg output from rclone -V)
rclone v1.36-DEV
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Linux CentOS-7 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone -v mount --allow-non-empty --allow-other --max-read-ahead 200M --drive-chunk-size 128M --checkers 40 --dir-cache-time 30m --transfers=32 gdrive:/Media /home/plex/media
A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)
2017/04/23 17:49:51 DEBUG : rclone: Version "v1.36-DEV" starting with parameters ["rclone" "-vv" "mount" "--allow-non-empty" "--allow-other" "--max-read-ahead" "200M" "--drive-chunk-size" "128M" "--checkers" "40" "--dir-cache-time" "30m" "--transfers=32" "gdrive:/Media" "/home/plex/media"]
2017/04/23 17:49:51 Failed to create file system for "gdrive:/Media": couldn't read info about Drive: Get https://www.googleapis.com/drive/v2/about?alt=json: oauth2: token expired and refresh token is not set

I've authorized everything correctly, but every time I mount the google drive, it gives me that error. The same setup worked on a previous Ubuntu machine running rclone 1.35. Please advise. Thank you.

Try rclone config choose edit,  select the remote, skip until you get to token refresh, then choose to refresh token, save and it should work.

This was solved with the linked issue.


Move bad images to new dir. Remove exception handling from generator

Leaving this as a PR in case you want to take a look at it, though there isn't much here.
Merge if it looks good to you.


Fix meta-actions not being applied if multiMatch is enabled in the chain starter rule

Fixes #2867.
(This PR will be marked as ready after #2866 is merged and the new regression tests are updated to also check for the tags in the audit log.)

@martinhsv Fixed.


Error importing Parquet from HDFS using PyArrow

Hi,
I get the following error when attempting to read a parquet file stored on hdfs:
df = dataframe.read_parquet('hdfs://hdfsnn:8020/user/hdfs/ndsparq/Test.parquet', engine='arrow')
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-3-22d8573a1d6f> in <module>()
----> 1 df = dataframe.read_parquet('hdfs://hdfsnn:8020/user/hdfs/ndsparq/Test.parquet', engine='arrow')

/opt/anaconda3/envs/env0/lib/python3.6/site-packages/dask/dataframe/io/parquet.py in read_parquet(path, columns, filters, categories, index, storage_options, engine)
    293         return _read_pyarrow(fs, paths, file_opener, columns=columns,
    294                              filters=filters,
--> 295                              categories=categories, index=index)
    296
    297

/opt/anaconda3/envs/env0/lib/python3.6/site-packages/dask/dataframe/io/parquet.py in _read_pyarrow(fs, paths, file_opener, columns, filters, categories, index)
    164         columns = list(columns)
    165
--> 166     dataset = api.ParquetDataset(paths)
    167     schema = dataset.schema.to_arrow_schema()
    168     task_name = 'read-parquet-' + tokenize(dataset, columns)

/opt/anaconda3/envs/env0/lib/python3.6/site-packages/pyarrow/parquet.py in __init__(self, path_or_paths, filesystem, schema, metadata, split_row_groups, validate_schema)
    619
    620         (self.pieces, self.partitions,
--> 621          self.metadata_path) = _make_manifest(path_or_paths, self.fs)
    622
    623         if self.metadata_path is not None:

/opt/anaconda3/envs/env0/lib/python3.6/site-packages/pyarrow/parquet.py in _make_manifest(path_or_paths, fs, pathsep)
    779             if not fs.isfile(path):
    780                 raise IOError('Passed non-file path: {0}'
--> 781                               .format(path))
    782             piece = ParquetDatasetPiece(path)
    783             pieces.append(piece)

OSError: Passed non-file path: hdfs://hdfsnn:8020/user/hdfs/ndsparq/Test.parquet

The same parquet file can be read using PyArrow without issue:
with hdfs.open('/user/hdfs/ndsparq/Test.parquet', 'rb') as f:
    table = pq.read_table(f)
df = table.to_pandas()

Are you able to help? It looks like the filesystem is not passed to api.ParquetDataset in _read_pyarrow().
Thanks very much!

@stuckeyr : where you have with hdfs.open, does this refer to arrow's hdfs interface, or is it a HDFileSystem from hdfs3?

Yes, Arrow's interface:
import pyarrow as pa
hdfs = pa.hdfs.connect('hdfsnn', 8020, user='hdfs')

@martindurant I would have expected this to import distributed.hdfs and then either fail with an informative error message or else work.  Do you have any thought on why @stuckeyr might be passed a filename instead?

@mrocklin , this is not using distributed.hdfs at all, but arrow's built-in hdfs interface.
My guess would be, that parquet expects to be able to open directories, but they fail the isFile check.

I think the problem here is that pyarrow isn't fed the correct Filesystem instance but always a local one. This should be fixed with https://github.com/dask/dask/pull/2527 ( @fjetter As my index fix is now in master, can you rebase?)

@stuckeyr You could test this assumption by changing this line:
diff --git a/dask/dataframe/io/parquet.py b/dask/dataframe/io/parquet.py
index f4b37346..0dd743af 100644
--- a/dask/dataframe/io/parquet.py
+++ b/dask/dataframe/io/parquet.py
@@ -173,7 +173,7 @@ def _read_pyarrow(fs, paths, file_opener, columns=None, filters=None,
     if isinstance(columns, tuple):
         columns = list(columns)

-    dataset = api.ParquetDataset(paths)
+    dataset = api.ParquetDataset(paths, filesystem=fs)
     schema = dataset.schema.to_arrow_schema()
     has_pandas_metadata = schema.metadata is not None and b'pandas' in schema.metadata
     task_name = 'read-parquet-' + tokenize(dataset, columns)

@xhochy For #2527 to actually work pyarrow needed to introduce the S3FSWrapper. I'd assume the same needs to be done for HDFS, see here
We should maybe think of a more elaborate mechanism to ensure consistent interfaces

Guys, I'm moving house today. Will get back onto it next week. Thanks for all your efforts!

Good luck!
On Wed, Nov 1, 2017 at 5:54 PM, Roger Stuckey<EMAIL_ADDRESS>wrote:

Guys, I'm moving house today. Will get back onto it next week. Thanks for
all your efforts!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/2801#issuecomment-341254703, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszLT-qt8ipBi7AU9M5Xxi8tLuLbGRks5syOh5gaJpZM4QAOe6
.

#2527 is now merged, I believe this to be fixed. @stuckeyr could you try out the master branch and see if it resolves your issue?

Sorry, still not working. It seems the Arrow's ParquetDataset._ensure_filesystem() does not handle HDFS, and create a connection. @fjetter, do you want me to submit an issue to that project?
df = dataframe.read_parquet('hdfs://hdfsnn:8020/user/hdfs/ndsparq/Test.parquet', engine='arrow')
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-2-cf8b5e1e9563> in <module>()
----> 1 df = dd.read_parquet('hdfs://hdfsnn:8020/user/hdfs/ndsparq/Test.parquet', engine='arrow')

/opt/anaconda3/envs/env1/lib/python3.6/site-packages/dask/dataframe/io/parquet.py in read_parquet(path, columns, filters, categories, index, storage_options, engine)
    331         return _read_pyarrow(fs, paths, file_opener, columns=columns,
    332                              filters=filters,
--> 333                              categories=categories, index=index)
    334
    335

/opt/anaconda3/envs/env1/lib/python3.6/site-packages/dask/dataframe/io/parquet.py in _read_pyarrow(fs, paths, file_opener, columns, filters, categories, index)
    175     dataset = api.ParquetDataset(
    176         path_or_paths=paths,
--> 177         filesystem=fs
    178     )
    179     schema = dataset.schema.to_arrow_schema()

/opt/anaconda3/envs/env1/lib/python3.6/site-packages/pyarrow/parquet.py in __init__(self, path_or_paths, filesystem, schema, metadata, split_row_groups, validate_schema)
    614             self.fs = LocalFileSystem.get_instance()
    615         else:
--> 616             self.fs = _ensure_filesystem(filesystem)
    617
    618         self.paths = path_or_paths

/opt/anaconda3/envs/env1/lib/python3.6/site-packages/pyarrow/parquet.py in _ensure_filesystem(fs)
    748                 return LocalFileSystem.get_instance()
    749
--> 750         raise IOError('Unrecognized filesystem: {0}'.format(fs_type))
    751     else:
    752         return fs

OSError: Unrecognized filesystem: <class 'distributed.hdfs.DaskHDFileSystem'>

@stuckeyr yes, please open an issue with Arrow at https://issues.apache.org/jira/projects/ARROW about this.

Looks like the issue is with this logic here: https://github.com/apache/arrow/blob/master/python/pyarrow/parquet.py#L786-L792. This seems like a really fragile way to handle this. Perhaps a better way would be to remove this logic, and have dask pass the wrapped filesystems directly. Whether these live in pyarrow or dask is debatable.

@jcrist Yes, this is a fragile way to handle this but it avoids having pyarrow explicitly depend on S3FS. It is already possible for external libraries to define their own wrapper which inherits from pyarrow.filesystem.FileSystem.
I think pyarrow should have an interest in supporting common filesystem types which I think it would make more sense to implement this wrapper there.

Sorry, I didn't mean that pyarrow should depend on s3fs. Rather I meant that instead of dask passing in its filesystem objects and pyarrow inferring and converting, we should just pass in a valid pyarrow.filesystem.FileSystem originally (and skip the classname string inference hack).
Whether the S3FSWrapper and needed HDFSWrapper live in pyarrow or dask is debatable. I mildly think they should live in dask, since we'd be a consumer of the pyarrow library and would be responsible for passing in a valid pyarrow.filesystemFileSystem object. Ping @xhochy, @wesm for thoughts.

I wonder if it is feasible to agree on a single FileSystem interface between these projects.

Having such a standard here would also make it easier to convince other groups (like Azure) to adhere to that standard.

I wonder if it is feasible to agree on a single FileSystem interface between these projects.

Filed a separate issue to discuss this: #2880.

We could create a pure Python library to define a abstract public API for these sorts of things. Then we aren't relying on duck typing

This should be fixed in #2881, although I'll probably move things around a bit to make testing possible.

HDFS integration has been moved out of distributed and into hdfs3. A pyarrow compatibility wrapper has been added there, as well as a test for reading parquet from hdfs using pyarrow (https://github.com/dask/hdfs3/blob/master/hdfs3/tests/test_dask.py#L442-L461).
See #2894, https://github.com/dask/distributed/pull/1561, and https://github.com/dask/hdfs3/pull/144 for reference.
At this point parquet using pyarrow from hdfs works if you install dask, distributed, and hdfs3 from their respective master branches. We should have a release out of all of these in the next week or so, at which point you can just install the latest version from conda-forge.
Closing.

You should check libraries version. I had the same problem and resetting versions to:

pyarrow = 0.11.1
dask =1.0.0
With this configuration it works

I am getting this error on databricks environment?
import pyarrow.parquet as pq
print(pyarrow.version)
data1 = pq.read_table("output/rfm_data.parquet")
I am getting this error

OSError                                   Traceback (most recent call last)
 in ()
2 print(pyarrow.version)
3 data = sqlContext.read.parquet("output/rfm_data.parquet")
----> 4 data1 = pq.read_table("output/rfm_data.parquet")
/local_disk0/pythonVirtualEnvDirs/virtualEnv-dda14358-e6b1-4176-82cf-11f68d2d5859/lib/python3.5/site-packages/pyarrow/parquet.py in read_table(source, columns, use_threads, metadata, use_pandas_metadata, memory_map, filesystem)
1150         return fs.read_parquet(path, columns=columns,
1151                                use_threads=use_threads, metadata=metadata,
-> 1152                                use_pandas_metadata=use_pandas_metadata)
1153
1154     pf = ParquetFile(source, metadata=metadata)
/local_disk0/pythonVirtualEnvDirs/virtualEnv-dda14358-e6b1-4176-82cf-11f68d2d5859/lib/python3.5/site-packages/pyarrow/filesystem.py in read_parquet(self, path, columns, metadata, schema, use_threads, use_pandas_metadata)
177         from pyarrow.parquet import ParquetDataset
178         dataset = ParquetDataset(path, schema=schema, metadata=metadata,
--> 179                                  filesystem=self)
180         return dataset.read(columns=columns, use_threads=use_threads,
181                             use_pandas_metadata=use_pandas_metadata)
/local_disk0/pythonVirtualEnvDirs/virtualEnv-dda14358-e6b1-4176-82cf-11f68d2d5859/lib/python3.5/site-packages/pyarrow/parquet.py in init(self, path_or_paths, filesystem, schema, metadata, split_row_groups, validate_schema, filters, metadata_nthreads, memory_map)
933          self.metadata_path) = _make_manifest(
934              path_or_paths, self.fs, metadata_nthreads=metadata_nthreads,
--> 935              open_file_func=self._open_file_func)
936
937         if self.common_metadata_path is not None:
/local_disk0/pythonVirtualEnvDirs/virtualEnv-dda14358-e6b1-4176-82cf-11f68d2d5859/lib/python3.5/site-packages/pyarrow/parquet.py in _make_manifest(path_or_paths, fs, pathsep, metadata_nthreads, open_file_func)
1108             if not fs.isfile(path):
1109                 raise IOError('Passed non-file path: {0}'
-> 1110                               .format(path))
1111             piece = ParquetDatasetPiece(path, open_file_func=open_file_func)
1112             pieces.append(piece)
OSError: Passed non-file path: output/rfm_data.parquet

@sushmit86 , this appears to have nothing to do with Dask, maybe post on the pyarrow tracker.

Somehow have the same errors. We are working with pyarrow 0.15.1
table2 = pq.read_table('C:/Data/test-parquet')

works fine in contrast to
parquet_file = pq.ParquetFile('C:/Data/test-parquet')
table2.parquet_file.read_row_group(0)

which raises
ArrowIOError: Failed to open local file 'C:/Data/test-parquet', error: Access is denied.

@RoelantStegmann , this again seems to be specific to pyarrow, no dask or even fsspec code is being used here. I can't even speculate what the Access Denied error might mean.

Thanks, will move it to their issue tracker. :)


Improve error reporting

JavaScript exception in file './js/common/bootstrap/modules.js' at 524,13: [ArangoError 3103: failed to invoke module
File: ./js/node/wut.js]
!      throw err;
!            ^
stacktrace: [ArangoError 3103: failed to invoke module
File: ./js/node/wut.js]
at [object Object].Module.run (./js/common/bootstrap/modules.js:1429:20)
at createModule (./js/common/bootstrap/modules.js:517:19)
at requireModuleFrom (./js/common/bootstrap/modules.js:619:21)
at requirePackage (./js/common/bootstrap/modules.js:777:21)
at [object Object].Module.require (./js/common/bootstrap/modules.js:1214:21)
at require (./js/common/bootstrap/modules.js:50:24)
at :1:1
The result is ok, but the the error is not inside of wut.js, but in hass.js, because of the undefined function func() is invoked.
The exception doesn't tell that I'm calling an undefined function. So the user has no chance to find out what in the code (foxx app) is broken.
One should get to know the primary failure reason like here:
JavaScript exception in file './js/common/bootstrap/modules.js' at 524,13: [ArangoError 3103: failed to invoke module
File: ./js/node/hass.js, Root cause: ReferenceError: func is not defined]
!      throw err;
!            ^
stacktrace: [ArangoError 3103: failed to invoke module
File: ./js/node/hass.js, Root cause: ReferenceError: func is not defined]
at [object Object].Module.run (./js/common/bootstrap/modules.js:1437:20)
at createModule (./js/common/bootstrap/modules.js:517:19)
at requireModuleFrom (./js/common/bootstrap/modules.js:619:21)
at requirePackage (./js/common/bootstrap/modules.js:777:21)
at [object Object].Module.require (./js/common/bootstrap/modules.js:1214:21)
at require (./js/common/bootstrap/modules.js:50:24)
at :1:1
Sample patch to solve the problem, but has to be finalized: https://gist.github.com/d65406076f4b7fea26da
(on behalf of steemann)

@pluma pls finalize and incooperate this patch.


Nothing displayed when clicking the ? symbol in the graph view

When I click the ? symbol at the top right nothing is displayed

fixed apparently https://github.com/PathwayCommons/pathways-search/commit/96f7c8fb1e8e66f04478242e3880430dd7143a8b


Function.grantInvoke(IGrantable) fails with imported function

Goal
My goal is to have two stacks, one representing the application and one representing an integration test. Each stack has a lambda, and I want the integration lambda to be able to call the application lambda.
Describe the bug
When attempting to implement this using the "Get a value from another stack" functionality described in the docs I get the error Policy statement must contain resources:
3/5 | 4:09:34 PM | CREATE_FAILED        | AWS::IAM::Policy      | TestFunction/ServiceRole/DefaultPolicy (TestFunctionServiceRoleDefaultPolicy810BAB0B) Policy statement must contain resources. (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: 83c89c17-6df8-11e9-ae97-4d6d78c6e2d1)
	new Policy (/tmp/jsii-kernel-AFZIiZ/node_modules/@aws-cdk/aws-iam/lib/policy.js:23:26)
	\_ Role.addToPolicy (/tmp/jsii-kernel-AFZIiZ/node_modules/@aws-cdk/aws-iam/lib/role.js:67:34)

To Reproduce
Run cdk deploy on a two-stack app such as this:
import javax.annotation.Nullable;

import software.amazon.awscdk.App;
import software.amazon.awscdk.Construct;
import software.amazon.awscdk.Stack;
import software.amazon.awscdk.services.lambda.Code;
import software.amazon.awscdk.services.lambda.Function;
import software.amazon.awscdk.services.lambda.FunctionImportProps;
import software.amazon.awscdk.services.lambda.FunctionProps;
import software.amazon.awscdk.services.lambda.IFunction;
import software.amazon.awscdk.services.lambda.Runtime;

public class TwoFunctionApp {

    static class ApplicationStack extends Stack {

        private final FunctionImportProps publicFunction;

        public ApplicationStack(@Nullable Construct scope, @Nullable String name) {
            super(scope, name);

            Function publicFunction = new Function(this, "PublicFunction", FunctionProps.builder()
                    .withCode(Code.asset("build/lib/lambda.zip"))
                    .withRuntime(Runtime.JAVA8)
                    .withMemorySize(256)
                    .withHandler("com.amazon.PublicFunction")
                    .build());

            this.publicFunction = publicFunction.export();
        }
    }

    static class IntegrationTestStack extends Stack {
        public IntegrationTestStack(@Nullable Construct scope, @Nullable String name, FunctionImportProps functionImportProps) {
            super(scope, name);

            IFunction appFunctionToCall = Function.import_(this, "ThePublicFunction", functionImportProps);

            Function testFunction = new Function(this, "TestFunction", FunctionProps.builder()
                    .withCode(Code.asset("build/lib/lambda.zip"))
                    .withRuntime(Runtime.JAVA8)
                    .withMemorySize(256)
                    .withHandler("com.amazon.TestFunction")
                    .build());

            appFunctionToCall.grantInvoke(testFunction);
        }
    }

    public static void main(String... args) {
        App app = new App();
        ApplicationStack application = new ApplicationStack(app, "application");
        new IntegrationTestStack(app, "integ", application.publicFunction);
        app.run();
    }
}

Expected behavior
If the two functions are defined in the same stack, the grantInvoke permissions are created as expected. So, I'm hoping for this to work across stacks.
Version:

OS: Ubuntu 18.04
Programming Language: Java 8
CDK Version: 0.29.0

We are working on updating our docs, basically, if these two stacks are within the same app, there is no need to "export" and "import" explicitly. The framework will automatically do that for you.
This means that for your use case the following should work:
public class TwoFunctionApp {

    static class ApplicationStack extends Stack {

        private final Function publicFunction;

        public ApplicationStack(@Nullable Construct scope, @Nullable String name) {
            super(scope, name);

            Function publicFunction = new Function(this, "PublicFunction", FunctionProps.builder()
                    .withCode(Code.asset("lambda"))
                    .withRuntime(Runtime.JAVA8)
                    .withMemorySize(256)
                    .withHandler("com.amazon.PublicFunction")
                    .build());

            this.publicFunction = publicFunction;
        }
    }

    static class IntegrationTestStack extends Stack {
        public IntegrationTestStack(@Nullable Construct scope, @Nullable String name, Function functionFromOtherStack) {
            super(scope, name);

            Function testFunction = new Function(this, "TestFunction", FunctionProps.builder()
                    .withCode(Code.asset("lambda"))
                    .withRuntime(Runtime.JAVA8)
                    .withMemorySize(256)
                    .withHandler("com.amazon.TestFunction")
                    .build());

            functionFromOtherStack.grantInvoke(testFunction);
        }
    }

    public static void main(String... args) {
        App app = new App();
        ApplicationStack application = new ApplicationStack(app, "application");
        new IntegrationTestStack(app, "integ", application.publicFunction);
        app.run();
    }
}

Please let us know if this works for you, and reopen if there are any further issues/questions

https://github.com/awslabs/aws-cdk/issues/2475

Thanks for the response! I like the design allowing users to reference entities directly.
Your example works!
The next hiccup occurs when I attempt to add read permissions to a bucket that was created in the integ stack.
Error: 'integ' depends on 'application' (integ/TestFunction/ServiceRole/DefaultPolicy/Resource -> application/PublicFunction/Resource.Arn). Adding this dependency (application/PublicFunction/ServiceRole/DefaultPolicy/Resource -> integ/TestResources/Resource.Arn) would create a cyclic reference.

Interestingly, I can apply either grantInvoke or grantRead. If I apply both, however, I get the circular dependency error.
 IntegrationTestStack extends Stack {
        public IntegrationTestStack(@Nullable Construct scope, @Nullable String name, Function functionFromOtherStack) {
            super(scope, name);

            Bucket testResources = new Bucket(this, "TestResources");

            Function testFunction = new Function(this, "TestFunction", FunctionProps.builder()
                    .withCode(Code.asset("lambda"))
                    .withRuntime(Runtime.JAVA8)
                    .withMemorySize(256)
                    .withHandler("com.amazon.TestFunction")
                    .build());
            functionFromOtherStack.grantInvoke(testFunction);
            testResources.grantRead(functionFromOtherStack);
        }
    }
}

It kind of makes sense that you get a circular dependency. grantInvoke defines an AWS::Lambda::Permission resource to your "integ" stack that references the Lambda function from your "application" stack, while grantRead adds a statement to the Lambda IAM execution role from your "application" stack that references the bucket from your "integ" stack.
Conceptually, I am wondering what's the use case for the application lambda function to read objects from a bucket defined in your integration stack. That actually sounds like a dependency you wouldn't want in the first place.
@rix0rrr what do you think?

For me, the overall issue is that I don't get circular reference errors when the resources are in a single stack. Once I refactor them into multiple stacks, I start getting circular references. Quite possibly it's a misunderstanding on the user's part, in which case documentation could help here (maybe it exists already?).
Regarding the usecase, the API operates on a large payload out-of-band -- the caller sets up the payload, invokes the lambda, and then waits for the results to be written. It's the caller's responsibility to ensure the callee's IAM role has appropriate read/write access to the S3 bucket.

Totally understand that this can cause confusion and we will make sure to include some information about this when we describe cross-stack references (we are currently working on general doc improvements).
As for your use case, I would still argue this is not the desirable architecture. What happens if you want to add another testing stack (e.g. for development)? Do you want to need to go and update the production stack with IAM permissions for this new stack? In general, I would try to isolate the testing stacks from production and the dependency direction should be test => prod and not the opposite.
I wonder if a 3rd stack that only includes the bucket can help (e.g. "drop box"). This way, your production stack can only depend on this bucket stack and then the testing stacks can also depend on those to write data.

To clarify a bit, the application stack and test stack would get duplicated on every instantiation of the CDK app. So, beta would have both stacks, and prod would have both stacks, but the beta resources would have zero permissions to call anything in prod.
Thanks for all of the clarification. I'll go back to the documentation and see if there are some other CDK primitives (maybe constructs?) that are more aligned with the factoring that we're looking for.