[CS2103-T14-3] WanderLust

WanderLust helps tech-savvy travellers to plan their trips in a structured and efficient manner with by providing them with a holistic travel planner. It is optimized for CLI users so that destinations and details and be added faster by typing in commands.

Codecov Report

Merging #6 into master will decrease coverage by 21.45%.
The diff coverage is 0.00%.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6389d85...5e7a314. Read the comment docs.

Codecov Report

Merging #6 into master will decrease coverage by 32.29%.
The diff coverage is 0.00%.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6389d85...a5e8add. Read the comment docs.


State Error message.

I was just wondering why I was getting this error message
Cannot assign a value of type NumbericLineEdit.State to variable "state" with specified type State.
Line 77:Invalid operands "State" and "NumbericLineEdit.State" for "==" operator.
Line 79:Value of type "NumbericLineEdit.State" cannot be assigned to a variable of type "State".
Line 81:Invalid operands "State" and "NumbericLineEdit.State" for "==" operator.
Line 84:Value of type "NumbericLineEdit.State" cannot be assigned to a variable of type "State".
Line 85:Invalid operands "State" and "NumbericLineEdit.State" for "==" operator.
Line 86:Value of type "NumbericLineEdit.State" cannot be assigned to a variable of type "State".
Line 93:Invalid operands "State" and "NumbericLineEdit.State" for "==" operator.
Line 95:Value of type "NumbericLineEdit.State" cannot be assigned to a variable of type "State".
Line 98:Invalid operands "State" and "NumbericLineEdit.State" for "==" operator.
Line 121:Value of type "NumbericLineEdit.State" cannot be assigned to a variable of type "State".
I was trying to make a state machine for my project and this keeps flagging.

I'm guessing you have a class names State somewhere in your project which is causing a conflict.

I think that is it. I was trying to make a state machine which I think
caused the conflict
On Wed, Dec 13, 2023 at 4:42 PM blackears @.***> wrote:

I'm guessing you have a class names State somewhere in your project which
is causing a conflict.
—
Reply to this email directly, view it on GitHub
https://github.com/blackears/cyclopsLevelBuilder/issues/109#issuecomment-1854302736,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A2OYOUB6IQLPCHERUSPZSQDYJHLHVAVCNFSM6AAAAABAS5QKWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJUGMYDENZTGY
.
You are receiving this because you authored the thread.Message ID:
@.***>

I thought I had solved it but now I am getting spammed with Failed loading resource: res://addons/cyclops_level_builder/docks/uv_editor/uv_editor_preview.gd. Make sure resources have been imported by opening the project in the editor at least once.

Are you using version 1.0.1 or the 1.0.2 dev version?

I grabbed it from the master branch. I did upgrade from 1.0.1 I think

I have the same issue and for me , i thinkm it's in conflict with the "StateChart"-Addon (https://github.com/derkork/godot-statecharts ) that I'm using. If I go with "Lookup Symbol" on the State enum,  it redirects to the state chart addon which has a class named "State"

This was changed.  If you checkout the developer branch, this enum has been renamed,

This was changed. If you checkout the developer branch, this enum has been renamed,

Good to know, thanks. Just downloaded the current release and came across this issue .

@blackears Can you also rename the settings class? Something like CyclopsSettings? I had to re-install because I had a settings class in my project beforehand.

I've changed the class name in the dev branch.


Implement start penalty functionality

How to use:

Set def_start_penalty to a list of floats. For each deferrable load with a penalty P, each time the deferrable load turns on will incur an additional cost of P • P_deferrable_nom • cost of electricity at that time. Intuitively, how many extra hours would the load have to run to avoid starting up again.
Pass def_current_state in runtime_params to avoid incorrectly penalizing a start if a forecast is run when the load is already running.

I also made a couple of other supporting changes:

Added unit tests for the relevant scenarios in test_optimization.py, plus one for the set_def_constant function. In doing so, I found a missing constraint for set_def_constant and added it to the optimizer (P_def_bin2 was not constrained to be zero when P_deferrable was zero).
Added comments for all the start-related constraints to help myself understand it.

Solves #261

Hi @werdnum, is this ready to merge?
Probably some documentation needed somewhere.
From what I see this is only usable from the runtime parameters dictionary, but not from the configuration file?

I think it works in the configuration file. I’ll have to check to make sure.
Will take a look at documentation.
— Andrew Garrett
On Tue, 28 May 2024 at 23:11, David @.***> wrote:

Hi @werdnum https://github.com/werdnum, is this ready to merge?
Probably some documentation needed somewhere.
From what I see this is only usable from the runtime parameters
dictionary, but not from the configuration file?
—
Reply to this email directly, view it on GitHub
https://github.com/davidusb-geek/emhass/pull/291#issuecomment-2135183717,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACCFXV7AF5J4772D6EEAJDZER7BFAVCNFSM6AAAAABIEWH3ROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZVGE4DGNZRG4
.
You are receiving this because you were mentioned.Message ID:
@.***>

Ok, I'm merging. But from what I see this will only works when passing def_start_penalty and def_current_state at runtime. For def_current_state it makes completely sense, but for def_start_penalty this could be set directly in the configuration file.
A small entry on the documentation would be needed don't you think.
I will merge and then try to update the documentation later on another PR

Hi, this was merged but not correctly reviewed I guess.
I was trying to update the documentation an stumbled on this.
There are some  important things that I don't fully understand:
Let's start with this code for def_start_penalty:
        # Add term penalizing each startup where configured
        if ("def_start_penalty" in self.optim_conf and self.optim_conf["def_start_penalty"]):
            for k in range(self.optim_conf["num_def_loads"]):
                if (len(self.optim_conf["def_start_penalty"]) > k and self.optim_conf["def_start_penalty"][k]):
                    objective = objective + plp.lpSum(
                        -0.001 * self.timeStep * self.optim_conf["def_start_penalty"][k] * P_def_start[k][i] *\
                            unit_load_cost[i] * self.optim_conf['P_deferrable_nom'][k]
                        for i in set_I)

This self.optim_conf["def_start_penalty"] is supposed to be a list of floats.
But it seems to me that you are treating this a list of bool.
For example here if ("def_start_penalty" in self.optim_conf and self.optim_conf["def_start_penalty"]).
Another term I don't understand is len(self.optim_conf["def_start_penalty"]) > k. If that parameter is a list of floats with a single float for each deferrable then when is this condition supposed to True? And again this is treating self.optim_conf["def_start_penalty"][k] as if it were a bool.
Same comment for the def_current_state parameter.
This condition can never hold if that parameter is a list of bool with a single bool for each deferrable load: if ("def_current_state" in self.optim_conf and len(self.optim_conf["def_current_state"]) > k):
I'm a bit lost here.

Hi David, thanks for taking a look and sorry I haven't got to the
documentation yet!
Happy to help you understand the code and of course if there's something
I've done that's hard to understand, please let me know how I can make it
better.
I am relatively confident in the basic functionality because of the
expanded unit tests, but there could be small bugs of course.
— Andrew Garrett
On Mon, 10 Jun 2024 at 05:34, David @.***> wrote:

Hi, this was merged but not correctly reviewed I guess.
I was trying to update the documentation an stumbled on this.
There are some important things that I don't fully understand:
Let's start with this code for def_start_penalty:
    # Add term penalizing each startup where configured
    if ("def_start_penalty" in self.optim_conf and self.optim_conf["def_start_penalty"]):
        for k in range(self.optim_conf["num_def_loads"]):
            if (len(self.optim_conf["def_start_penalty"]) > k and self.optim_conf["def_start_penalty"][k]):
                objective = objective + plp.lpSum(
                    -0.001 * self.timeStep * self.optim_conf["def_start_penalty"][k] * P_def_start[k][i] *\
                        unit_load_cost[i] * self.optim_conf['P_deferrable_nom'][k]
                    for i in set_I)

This self.optim_conf["def_start_penalty"] is supposed to be a list of
floats.
But it seems to me that you are treating this a list of bool.
For example here if ("def_start_penalty" in self.optim_conf and
self.optim_conf["def_start_penalty"]).

This is a common idiom in python - a nonzero value evaluates to true, a
zero value (or empty string, None, etc) evaluates to false. I'm checking
whether there is any value (so I don't just assume it's a list and crash if
there's nothing there). This is what's recommended by the Google style
guide (
https://google.github.io/styleguide/pyguide.html) but happy to follow your
lead if you have another style preference.

Another term I don't understand is len(self.optim_conf["def_start_penalty"])

k. If that parameter is a list of floats with a single float for each
deferrable then when is this condition supposed to True?

Here I'm trying to check whether the list of start penalties is long enough
to have a value for the current one. I don't want to try to get a staff
penalty value for load 2 (3rd in the list) if there are only 2 items in the
list. Len() returns the number of values, and k is the zero based index of
the deferrable load. So if there are 2 deferrable loads, K could be 0 or 1.

And again this is treating self.optim_conf["def_start_penalty"][k] as if
it were a bool.

Right, I don't want to do any processing for start penalty if it's set to
zero.

Same comment for the def_current_state parameter.
This condition can never hold if that parameter is a list of bool with a
single bool for each deferrable load: if ("def_current_state" in
self.optim_conf and len(self.optim_conf["def_current_state"]) > k):

Remember K will take all values from 0 to num_deferrable_loads exclusive,
so this will be false only when the def_current_state list if under sized.

I'm a bit lost here.
—
Reply to this email directly, view it on GitHub
https://github.com/davidusb-geek/emhass/pull/291#issuecomment-2156752354,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACCFXTTMMWZHAP6RPLSWSTZGSU2XAVCNFSM6AAAAABIEWH3ROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJWG42TEMZVGQ
.
You are receiving this because you were mentioned.Message ID:
@.***>


HtmlToWmlConverter throws Duplicate attribute error

I am trying to convert several HTML documents to WML and am getting the Duplicate attribute error in the TransformToLower(XNode node) method when I have a node that contains both lang and xml:lang attributes.  According to the XHTML spec, both attributes may appear.
Do I need to strip out one of the 2 in order for this to work properly?
Thanks

@GarthMartin This may be a bug in the LINQ to XML library.  I have seen other bugs in the LtX lib with regards to duplicate attributes.  I have seen this in multi-threading scenarios.
I recommend striping out one of the two, and seeing if it works.


update to ES 2.4.3

I've updated the build.gradle file to build against ES 2.4.3.

@jprante, Any chance to see this PR merged in the next days ?
Thanks !

yes, if I find time on the weekend, updates to latest ES releases will be done.

@jprante Thanks for your time and your work !

I have released plugin versions <IP_ADDRESS> for ES 2.4.3 and <IP_ADDRESS> for ES 2.4.4 (in a new 2.4 branch)

Thanks !!

Hi @jprante,
it's normal that the plugin is not available on this url https://github.com/jprante/elasticsearch-langdetect/releases/download/<IP_ADDRESS>/elasticsearch-langdetect-<IP_ADDRESS>-plugin.zip ?
I need to build it myself ?
Thanks

Try the URL
http://xbib.org/repository/org/xbib/elasticsearch/plugin/elasticsearch-langdetect/<IP_ADDRESS>/elasticsearch-langdetect-<IP_ADDRESS>-plugin.zip

Closing this pull request.


response.real_url does not exist in my environment

   When i run the example(quotes.py) code, an error " AttributeError: 'TextResponse' object has no attribute 'real_url'" appears. i do not know why.

I used to pass the url of my target website to my parse funtion, then i find that i can not use real_url. But today i find that i can just do it by changing endpint. But how to use real_url or if it can still be used?

@bjtuzpf I can't reproduce it with example quotes.py script.
What are versions of Splash and scrapy-splash you're using?

I'm sorry, i just found that i was wrong.I changed url to your target website, but it just do not use splash at all. So what i said above is wrong.

Thanks for clarifying it!


Unexpected promise rejection while using callback API

Issue Summary
When calling the mail send(..) with a callback, errors are passed to the callback — as expected — and a promise rejection is also produced — which is unexpected and undesirable, as the promise rejection escapes unhandled (other than the unhandled promise rejection handler, by which time it's too late).
Steps to reproduce
The code:
const sendgrid = require("@sendgrid/mail");
sendgrid.setApiKey(API_KEY);
sendgrid.send({
    attachments: [],
    subject: "testing sendgrid",
    html: ""
}, function(error) {
    if (error) {
        console.error(error.message);
    }
});

Produces the following:
(node:87110) UnhandledPromiseRejectionWarning: Error: Provide at least one of to, cc or bcc
    at Mail.addTo (/Users/pc/sendgrid/node_modules/@sendgrid/helpers/classes/mail.js:274
:13)
    at Mail.fromData<EMAIL_ADDRESS>112:12)
    at new Mail (/Users/pc/sendgrid/node_modules/@sendgrid/helpers/classes/mail.js:43:12
)
    at Function.create<EMAIL_ADDRESS>s:615:12)
    at MailService.send (/Users/pc/sendgrid/node_modules/@sendgrid/mail/src/classes/mail
-service.js:96:25)
    at Object.<anonymous> (/Users/pc/sendgrid/sendgrid.js:8:10)
    at Module._compile (internal/modules/cjs/loader.js:721:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:732:10)
    at Module.load (internal/modules/cjs/loader.js:620:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:560:12)
(node:87110) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejectionid: 1)

Looking at the code in /packages/mail/src/classes/mail-service.js:
    catch (error) {

      //Pass to callback if provided
      if (cb) {
        // eslint-disable-next-line callback-return
        cb(error, null);
      }

      //Reject promise
      return Promise.reject(error);
    }

From the eslint comment, it appears that the callback is intentionally not returned, therefore returning a promise rejection — but this clearly results in an unhandled promise rejection in code that is callback-oriented.
Technical details:

sendgrid-nodejs Version: @sendgrid/mail, 6.3.0
Node.js Version: 11.6.0

I have to disagree that this is a community enhancement. It's a pretty obvious defect which, given Node's warning that "[i]n the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code", verges on becoming a security issue.
Also, this seems to be a trivial fix, but I do not wish to get involved, as doing so would mean signing a CLA. Is there anyone actively supporting this project, who could fix this?

I'm happy to open a pr for it
On Wed, Mar 6, 2019, 11:13 juona<EMAIL_ADDRESS>wrote:

I have to disagree that this is a community enhancement. It's a pretty
obvious defect which, given Node's warning that "[i]n the future, promise
rejections that are not handled will terminate the Node.js process with a
non-zero exit code", verges on becoming a security issue.
Also, this seems to be a trivial fix, but I do not wish to get involved,
as doing so would mean signing a CLA. Is there anyone actively supporting
this project, who could fix this?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/sendgrid/sendgrid-nodejs/issues/872#issuecomment-469879315,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAd8QiuxMu7K0spHATqLLtHOHXEwBvNWks5vTuwdgaJpZM4aBDMl
.

I could do that.

Thanks. What about publishing? Last publish is from 9 months ago, so even if this is fixed, I suspect it won't be released.

I see two (potential) issues here:

If there is a client-side issue (e.g, missing to), the callback is executed and the error is thrown (this is what the OP reported)

MailService should return after executing cb(error, null); instead of returning a rejected promise

If there is a server-side issue (e.g., not authorized), the callback is executed with the error and the returned Promise is a rejection

#900 resolves this if it's actually a problem


Error when loading laps of Imola 2021 FP1

Hi,
when I try to load the laps of the FP1 of Imola 2021 I get the following error:
UnboundLocalError                         Traceback (most recent call last)
 in
4
5 session = ff1.get_session(2021, "Imola", "FP1")
----> 6 laps = session.load_laps(with_telemetry=False)
~/anaconda3/lib/python3.8/site-packages/fastf1/core.py in load_laps(self, with_telemetry, livedata)
1335             df = result if 'df' not in vars() else pd.concat([df, result], sort=False)  # noqa: F821
1336
-> 1337         laps = df.reset_index(drop=True)  # noqa: F821
1338         laps.rename(columns={'TotalLaps': 'TyreLife',
1339                              'NumberOfPitStops': 'Stint',
UnboundLocalError: local variable 'df' referenced before assignment
The following block of code is a minimum example that causes the error.
import fastf1 as ff1
from fastf1 import plotting
from fastf1.core import Laps
session = ff1.get_session(2021, "Imola", "FP1")
laps = session.load_laps()
I hope someone can help me :)

There were major technical problems during FP1 in Imola. Therefore, there is actually no data available for this session. See issue #21. I've improved the error handling now though, so this is more obvious.


Many normal operations output what "wsl --help" outputs, under sufficiently new versions.

Version
Microsoft Windows Server [Version 20348.617]
WSL Version

[X] WSL 2
[ ] WSL 1

Kernel Version
5.10.16
Distro Version
No response
Other Software
PowerShell
Docker Desktop 4.7.0
In fact, it is caused by WSL itself.
Repro Steps
When typing "wsl --set-default-version 2", "wsl --set-version  2"
Including when Docker Desktop is started
Expected Behavior
Finished converting the default version to wsl2.
Actual Behavior
no action, only the CLI will output what "wsl --help" outputs
an error when Docker Desktop is started, the content of the error is also the output of "wsl --help"
Specific Contents:

Usage: wsl.exe [Argument] [Options...] [CommandLine]

Arguments for running Linux binaries:

    If no command line is provided, wsl.exe launches the default shell.

    --exec, -e <CommandLine>
        Execute the specified command without using the default Linux shell.

    --
        Pass the remaining command line as is.

Options:
    --distribution, -d <Distro>
        Run the specified distribution.

    --user, -u <UserName>
        Run as the specified user.

Arguments for managing Windows Subsystem for Linux:

    --help
        Display usage information.

    --install [Options]
        Install additional Windows Subsystem for Linux distributions.
        For a list of valid distributions, use 'wsl --list --online'.

        Options:
            --distribution, -d [Argument]
                Downloads and installs a distribution by name.

                Arguments:
                    A valid distribution name (not case sensitive).

                Examples:
                    wsl --install -d Ubuntu
                    wsl --install --distribution Debian

    --set-default-version <Version>
        Changes the default install version for new distributions.

    --shutdown
        Immediately terminates all running distributions and the WSL 2
        lightweight utility virtual machine.

    --status
        Show the status of Windows Subsystem for Linux.

    --update [Options]
        If no options are specified, the WSL 2 kernel will be updated
        to the latest version.

        Options:
            --rollback
                Revert to the previous version of the WSL 2 kernel.

Arguments for managing distributions in Windows Subsystem for Linux:

    --export <Distro> <FileName>
        Exports the distribution to a tar file.
        The filename can be - for standard output.

    --import <Distro> <InstallLocation> <FileName> [Options]
        Imports the specified tar file as a new distribution.
        The filename can be - for standard input.

        Options:
            --version <Version>
                Specifies the version to use for the new distribution.

    --list, -l [Options]
        Lists distributions.

        Options:
            --all
                List all distributions, including distributions that are
                currently being installed or uninstalled.

            --running
                List only distributions that are currently running.

            --quiet, -q
                Only show distribution names.

            --verbose, -v
                Show detailed information about all distributions.

            --online, -o
                Displays a list of available distributions for install with 'wsl --install'.

    --set-default, -s <Distro>
        Sets the distribution as the default.

    --set-version <Distro> <Version>
        Changes the version of the specified distribution.

    --terminate, -t <Distro>
        Terminates the specified distribution.

    --unregister <Distro>
        Unregisters the distribution and deletes the root filesystem.

    --mount <Disk>
        Attaches and mounts a physical disk in all WSL2 distributions.

        Options:
            --bare
                Attach the disk to WSL2, but don't mount it.

            --type <Type>
                Filesystem to use when mounting a disk, if not specified defaults to ext4.

            --options <Options>
                Additional mount options.

            --partition <Index>
                Index of the partition to mount, if not specified defaults to the whole disk.

    --unmount [Disk]
        Unmounts and detaches a disk from all WSL2 distributions.
        Unmounts and detaches all disks if called without argument.```

### Diagnostic Logs

_No response_

We have improved the behavior of wsl.exe output in the version of WSL that is in the Microsoft Store.

We have improved the behavior of wsl.exe output in the version of WSL that is in the Microsoft Store.

@benhillis, I am experiencing the same problem as the OP (@AuroraTea). I am not sure if you have improved the behavior of wsl.exe output in the version of WSL that is in the Microsoft Store because I do not have access to that version as I am similarly on Microsoft Windows Server 2022, which generally lacks access to the Microsoft Store, but would imagine that everyone who recently updated is likewise experiencing this glaring bug.
Version
Microsoft Windows Server
Version 21H2 (OS Build 20348.643)
WSL Kernel version: 5.10.16

I ran into this problem while following the guide on “Manual installation steps for older versions of WSL”, so seemingly everyone who follows this guide, will have in their commands that would allow them to use WSL 2 result in displaying the output of wsl --help instead of executing the desired following commands:

wsl.exe --set-default-version 2
wsl.exe --set-version <Distro> 2
wsl.exe --import <Distro> <InstallLocation> <FileName> --version 2

There are probably others, but from what I can tell, this effectively prevents all users from using WSL 2 at all in its current state and is even causing programs that interface with this CLI (like the latest Docker Desktop as can be seen above) to fail in expected ways.

Would anyone be able to provide a short-term fix or workaround for this? It's a bit of a roadblock.

Exactly the same.
Thank you for your wonderful and accurate presentation.

@DerekNonGeneric Exactly the same.
Thank you for your wonderful and accurate presentation.

Thanks for reporting this @AuroraTea.
WSL2 isn't supported on Windows server (we're working on adding support, but it hasn't shipped yet).
I agree that the help text is misleading though, this should be fixed once WSL2 support comes to Windows server.

@DerekNonGeneric WSL2 was available on Win Server 2022 yesterday!!!


Intel®Embree  Description Details updated in Readme

Description
Intel®Embree is a collection of high performance ray tracing kernels that helps graphics application engineers to improve the performance of their photorealistic rendering application.
Embree is released as open source under the Apache2.0

This README has already been reviewed. The license to Apache is linked at the bottom, to be consistent with the structure of all other READMEs.


Handle CMS data git push -> possible to use another branch or something like that?

DESC

atm netfliy CMS always pushes to the main branch (what is somekind of nasty)

TODOs

[x] CMS should at least push against the current deployment branch
[x] check if a complete seperate branch for storing the data is reasonable or applicable

duplicated + parts implemented


Encounter extract saves period.start incorrect

We use a custom itemControl to represent the encounter start/end time (BTW the encounter period.end should be set once it is completed). It seems this widget does not consider the timezone while parsing the time value, so the value stored in Aidbox looks like 2023-05-16T15:00:00 without timezone (not converted to UTC). As a result we represent the encounter range incorrect, cause as we rely on the fact that all date/time stored on the server are in UTC and convert them to local TZ:

I believe we should update encounter start form field, make it the same as at the new appointment form. So we'll be able to save encounter.start to dateTime answer and then easily parse it in mapper.


Add randomness

I'm not entirely sure yet how to implement this, but probably via a new stage type that can perform one of several random tasks, depending on configuration. Several potentially useful ways to introduce randomness are:

Shuffle each match.
Shuffle all matches while leaving each match intact. (e.g. matching \d+ on ab12cd34ef56gh might yield ab56cd12ef34gh)
Return one of the matches at random.
Use one of several regexes (or stages) at random.
Apply one of several substitutions at random.
Insert one or more random characters.
etc...

Wouldn't it be enough to add an option to the sort stage in order to make it shuffle the matches instead of sorting them? (it would of course be incompatible with the current $,^,# options)

@barbuz That's just one way of randomisation. It would be possible to build all the others on top of it but that's not necessarily very convenient. It might be good to have random versions of other stage types as well, e.g. having M return a random match so you can pick a random element from a list (instead of shuffling the list and then picking the first element).

I was just suggesting what seems the easiest way to introduce randomness. Having just one source of randomness would make Retina usable for a number of tasks which at the moment can't be performed; adding other sources of randomness to optimize convenience/length in different situations can always be done later, but it's not as needed as adding the first one.

Yes, I realise that this is something that needs to be added quite urgently, and it will be part of the next release (with as many other ways of randomisation as I can be bothered to implement at the time), but I can't yet tell how soon that is going to happen.


Provide a better way with dealing with missing modules in debug mode

Currently when a module is missing in debug mode an exception is thrown. That's all well and good, but it prevents T3 from continuing its initialization. We have a situation where multiple engineers will be working on one environment and sometimes they use a proxy to make a local version of their script to the page which may not have a module that the dom for that environment has listed. This is fine in non-debug mode, but when debug is enabled it completely borks the process.
Granted I realize our development process at the moment may be more at fault then how T3 functions, but it would be nice to be able to see the error but not have T3 stop the initialization process when that error occurs.

The intention was to bring immediate problems like missing modules to the developers attention as soon as possible. If a module being initialized is missing, then a part of the page is essentially broken. That can sometimes be a minor inconvenience, but sometimes could be a major problem. The framework really can't differentiate the two so it errs on the side of more caution in debug mode.

I understand. I also completely agree with the concept. It's just proving to be a bit painful in our current working environment. Thanks though.

While I understand the position on this topic, I'd like to try to determine potential alternatives. This has become a reoccurring issue on our team and is costing us valuable dev cycles.
We have a set of environments that we develop against and we use a proxy to swap out our js files during development. When multiple components are being worked on simultaneously on an environment and a developer tries to proxy in his or her code it's usually missing a module (i.e. one that's being developed by someone else). Ultimately this leads to our developers either not developing with debug enabled or having to create temporary files that add empty modules just to get their module to load.
For our purpose, a warning would be much more beneficial. Is there a potential way that we can make the error method itself configurable?
The ideas I've come up with so far aren't necessarily the best. One would be adding an Override object onto the Box namespace that can be used to override error.
// In user code
Box.Override.error = () => ...

// In Application.js
var error = Box.Override.error || fuction() { /* Normal error code */ }

It's a bit ugly, not ideal but it is middle ground.

It's an odd use case. Can you create a stub file that is checked in but .gitignored? i.e. developers have their own version of a stubs.js
// stubs.js
Box.Application.addModule('missing-module', function() {});

Alternatively, there could be something like:
Box.Application.setErrorHandler(function(exception) { ... });

The downside of this is that you'll have to call this before you call Box.Application.init()
Even less optimal would be passing in a callback as a parameter to Box.Application.init({})

I'm good with the second approach. Thanks!

Implemented in #181.

Being as #181 is well under way, I'm going to close this issue. Thanks for working with us!


(HDFS-14541)when evictableMmapped or evictable size is zero, do not throw NoSuchE…

…lementException
Signed-off-by: sunlisheng<EMAIL_ADDRESS>

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
30
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
1390
trunk passed

+1
compile
56
trunk passed

+1
checkstyle
26
trunk passed

+1
mvnsite
55
trunk passed

+1
shadedclient
886
branch has no errors when building and testing our client artifacts.

+1
javadoc
32
trunk passed

0
spotbugs
165
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
161
trunk passed

_ Patch Compile Tests _

+1
mvninstall
53
the patch passed

+1
compile
48
the patch passed

+1
javac
48
the patch passed

+1
checkstyle
19
the patch passed

+1
mvnsite
51
the patch passed

-1
whitespace
0
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
shadedclient
846
patch has no errors when building and testing our client artifacts.

+1
javadoc
32
the patch passed

+1
findbugs
167
the patch passed

_ Other Tests _

-1
unit
63
hadoop-hdfs-client in the patch failed.

+1
asflicense
34
The patch does not generate ASF License warnings.

4082

Reason
Tests

Failed junit tests
hadoop.hdfs.web.TestTokenAspect

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-977/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/977

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 6d73f1cadc93 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 304a47e

Default Java
1.8.0_212

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-977/1/artifact/out/whitespace-eol.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-977/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-977/1/testReport/

Max. process+thread count
307 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-977/1/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
75
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
1120
trunk passed

+1
compile
43
trunk passed

+1
checkstyle
22
trunk passed

+1
mvnsite
45
trunk passed

+1
shadedclient
756
branch has no errors when building and testing our client artifacts.

+1
javadoc
29
trunk passed

0
spotbugs
127
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
126
trunk passed

_ Patch Compile Tests _

+1
mvninstall
39
the patch passed

+1
compile
37
the patch passed

+1
javac
37
the patch passed

+1
checkstyle
16
the patch passed

+1
mvnsite
40
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
807
patch has no errors when building and testing our client artifacts.

+1
javadoc
30
the patch passed

+1
findbugs
142
the patch passed

_ Other Tests _

+1
unit
117
hadoop-hdfs-client in the patch passed.

+1
asflicense
31
The patch does not generate ASF License warnings.

3586

Subsystem
Report/Notes

Docker
Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-977/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/977

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 4b4a9d84104c 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 335c1c9

Default Java
1.8.0_212

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-977/2/testReport/

Max. process+thread count
306 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-977/2/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
30
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
1037
trunk passed

+1
compile
44
trunk passed

+1
checkstyle
24
trunk passed

+1
mvnsite
48
trunk passed

+1
shadedclient
735
branch has no errors when building and testing our client artifacts.

+1
javadoc
31
trunk passed

0
spotbugs
124
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
122
trunk passed

_ Patch Compile Tests _

+1
mvninstall
41
the patch passed

+1
compile
37
the patch passed

+1
javac
37
the patch passed

+1
checkstyle
16
the patch passed

+1
mvnsite
42
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
742
patch has no errors when building and testing our client artifacts.

+1
javadoc
27
the patch passed

+1
findbugs
133
the patch passed

_ Other Tests _

+1
unit
129
hadoop-hdfs-client in the patch passed.

+1
asflicense
45
The patch does not generate ASF License warnings.

3385

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-977/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/977

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 181fc0d8940b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 335c1c9

Default Java
1.8.0_212

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-977/3/testReport/

Max. process+thread count
444 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-977/3/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
34
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
1171
trunk passed

+1
compile
42
trunk passed

+1
checkstyle
19
trunk passed

+1
mvnsite
49
trunk passed

+1
shadedclient
714
branch has no errors when building and testing our client artifacts.

+1
javadoc
25
trunk passed

0
spotbugs
128
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
126
trunk passed

_ Patch Compile Tests _

+1
mvninstall
41
the patch passed

+1
compile
38
the patch passed

+1
javac
38
the patch passed

+1
checkstyle
16
the patch passed

+1
mvnsite
39
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
760
patch has no errors when building and testing our client artifacts.

+1
javadoc
25
the patch passed

+1
findbugs
136
the patch passed

_ Other Tests _

+1
unit
117
hadoop-hdfs-client in the patch passed.

+1
asflicense
25
The patch does not generate ASF License warnings.

3467

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-977/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/977

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux fd467a36ce5a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / b28ddb2

Default Java
1.8.0_212

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-977/4/testReport/

Max. process+thread count
447 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-977/4/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
44
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
1227
trunk passed

+1
compile
41
trunk passed

+1
checkstyle
22
trunk passed

+1
mvnsite
44
trunk passed

+1
shadedclient
797
branch has no errors when building and testing our client artifacts.

+1
javadoc
30
trunk passed

0
spotbugs
136
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
131
trunk passed

_ Patch Compile Tests _

+1
mvninstall
45
the patch passed

+1
compile
38
the patch passed

+1
javac
38
the patch passed

+1
checkstyle
18
the patch passed

+1
mvnsite
41
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
840
patch has no errors when building and testing our client artifacts.

+1
javadoc
28
the patch passed

+1
findbugs
138
the patch passed

_ Other Tests _

+1
unit
109
hadoop-hdfs-client in the patch passed.

+1
asflicense
28
The patch does not generate ASF License warnings.

3725

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-977/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/977

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux b2a98ebf104b 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / b28ddb2

Default Java
1.8.0_212

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-977/5/testReport/

Max. process+thread count
336 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-977/5/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

LGTM

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
0
Docker mode activated.

-1
patch
12
https://github.com/apache/hadoop/pull/977 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/977

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-977/6/console

versions
git=2.7.4

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@goiri ,  I think we need to cherry-pick this feature to all branches, such as branch-2.9 etc. Thanks.

Does github allow cherry-picking? git cherry-pick -x requires -m which is not clear to me.

I saw you have done the cherry-pick, Thanks @goiri for your work :-)

For full disclosure, I messed up the merge of the PR and I had to do the cherry-pick by hand through the patch.
Next time I should do a proper squash and merge and hopefully cherry-pick will go clean.


Refactor typeofEquals usage and remove isInstanceOfDomType

Closes https://github.com/dart-lang/web/issues/77
typeofEquals now takes a String and returns a bool, and instanceOfString is the canonical replacement for isInstanceOfDomType. Removes now unneeded test as well. Updates min SDK version.

This will have to wait until we publish a Dart dev version >= this minimum.

Overall looks good, but I'd also be OK keeping a test like the one deleted in helpers_test. That's because, while technically the instnaceOfString should be tested by the SDK, there is still value in having a test here that ensures that it works for package:web types.

Fair point, added it back and refactored to account for instanceOfString.


Topic recognition

This pull request introduces the implementation of a 2D U-Net model for prostate MRI image segmentation as part of the HipMRI study project. The model is designed to accurately segment the prostate and classify six different tissue types in MRI scans, which is essential for prostate cancer diagnosis. Key features of this implementation include:
Model Architecture: Utilizes a U-Net architecture with an encoder-decoder design, incorporating skip connections to preserve spatial information.
Pre-processing Techniques: Implements normalization and resizing of MRI images to ensure consistent input for the model, enhancing performance and training efficiency.
Evaluation Metrics: Includes a Dice score calculation to assess the model's segmentation accuracy across multiple classes.
Visualization: Provides loss graphs and examples of input and output images for better understanding and assessment of model performance.
ps. This was painful

<This is an initial inspection, no action is required at this point.>
File Organizing: Well-organized files.
Problem Solving:

The algorithm solves the problem appropriately.
Accuracy in testing dataset: 0.78. However, no evidence supports this result.

Model and functions:

Student try to use PyTorch to construct the 2D-UNet models. However, the constructed model is one simplified variant of UNet.
No data augmentation.
No validation application to supervise the model training. Students are expected to use the training dataset to train the model and the validation dataset to observe the model performance in the training phase, then to use the test dataset to test the model performance on the unseen data.
Neither dice similarity coefficient nor dice loss definition in the code.

Code design: Incomplete code design.

Dice loss are expected to be applied in the segmentation task but didn't appear in the project code.

Code comment and docstring:

Good code comments
Minimal function docstrings

Difficulty: Easy.
Additional Comments:

intermediate numbers of commits
Figure link error in the report leading to image disappearance.
The Dice scores in the report cannot be generated from the project code as there is no dice score definition function in the code.
The last line in the "How It Works" section, "PLEASE ADD PATHS IN FOR YOU", indicates that parts or the entire report is suspected generated by the ChatGPT.
An experiment analysis and conclusion are expected to be in the report.
Wrong reference format: reference list should be organized together at the end of the report.

Marking

Good/OK/Fair Practice (Design/Commenting, TF/Torch Usage)
Adequate design and implementation. -1
Spacing and comments.
No Header blocks. -1
Recognition Problem
OK solution to problem. No loss defined, Model is a varient of Unet.-2
Driver Script present.
File structure present.
Good Usage & Demo & Visualisation & Data usage.
Module present.
Commenting missing. Minimal doc strings.-1
No Data leakage found.
Difficulty : Easy. UNet2D (Easy Difficulty)-10
Commit Log
Good Meaningful commit messages.
Good Progressive commits.
Documentation
Readme :Acceptable. Some markup errors. Incomplete.-4
Model/technical explanation :Acceptable. -2
Description and Comments :Acceptable. Wrong reference format: reference list should be organized together at the end of the report.-1
Markdown used and PDF submitted. PDF checked.
Pull Request
Successful Pull Request (Working Algorithm Delivered on Time in Correct Branch). Submitted on time.
No Feedback required.
Request Description is good.
TOTAL-22

Marked as per the due date and changes after which aren't necessarily allowed to contribute to grade for fairness.
Subject to approval from Shakes

Zeke Johnson

Not correct UNet -2 additionally.


Fix: Item introduction categories don't link

Item introduction categories originally linked back to a filtered version of their parent filter, when the item introduction configuration option "Show category" is checked.

We use "Show category" sparingly on item introductions. An example production page with this option checked is this one.
Note that currently the category renders as a plain text "Record" that doesn't link anywhere. With this commit, the category renders as a bold link that links back to the parent filterable list.

Before
After

Notes and todos
It doesn't strike me as obvious that users would know they could click on the category, even if this updated version. I note that the CFPB Design System page on item introductions shows that the category should render with some kind of icon:

I believe the icons come from this hardcoded list; in this case the page category of "Record" isn't in the list so no icon appears. Would it make sense to show some kind of fallback icon in that case? Ping @jenn-franklin. The heading with a fallback information-round matches what is in the DS, and looks like this:

Checklist

[x] PR has an informative and human-readable title
[x] Changes are limited to a single goal (no scope creep)
[x] Code follows the standards laid out in the CFPB development guidelines

Thanks for calling this out @chosak. I agree that the link without an icon does not appear as an obvious link. I would vote to always include an icon. I see in our DS documentation that we say icons are optional, so I would vote to delete this sentence from the DS documentation and make it default to include an icon. I'm happy to make this DS documentation change if nobody objects.

Is "Record" the only category that doesn’t have an icon? Maybe we could set an icon specifically for it? What if we set it as the  “document” round icon? So, this one:

I'm not seeing that the document icon is being used for any other category on that list.

I'm happy to make this DS documentation change if nobody objects.

Thanks @jenn-franklin, that sounds good to me. There are a number of other categories without icons (and a number of icon definitions for categories that don't actually exist anymore). I suspect this might be the cause of internal platform#4114, reported by @anselmbradford, where icons appear inconsistently -- maybe it's not that we're not showing the icon but that we haven't associated an icon with all of our categories. I prefer to merge this fix in as-is but will open a subsequent PR to handle the missing icons!

but will open a subsequent PR to handle the missing icons

Opened at #7378.


Allow override of the StanzaProcessor

In our vysper implementation, we need to manage JPA transactions and Spring security.
Those are transversal feature relying on ThreadLocal.
In order to be able to bind a database transaction and security context to one Stanza processing, I found that the easiest way would be to proxy the StanzaProcessor.
The proposed change allows to override the default StanzaProcessor.

Stanza are not always processed through a StanzaProcessor.
I keep looking for a better solution.


NotImplementedError: Trying to use forward AD with rsub that does not support it

Hi all,
When trying to calculate the Hessian of a function, I've run into the error "NotImplementedError: Trying to use forward AD with  that does not support it", such as rsub. I had two questions:

Is there a list somewhere of functions that are not yet supported?
Is there a way to identify where in my code this is breaking? For instance, stepping into a debugger doesn't seem to allow me to pinpoint where exactly I am using rsub. This would be useful so that I could consider alternatives.

Thanks

It seems like rsub is supported in PyTorch nightly. See https://github.com/pytorch/pytorch/issues/71117

Is there a way to identify where in my code this is breaking? For instance, stepping into a debugger doesn't seem to allow me to pinpoint where exactly I am using rsub.

Could you post the error? If it's in your user-code I think it should be clear-ish. But if it's being called internally in PyTorch, it'll be more difficult.

Switching to the nightly version seems to have worked. Thanks!


Can not remove uncommitted block

Alluxio Version:
2.0.1
Describe the bug
in my test cluster, Disk space is around 100GB, UFS is OBS.
Because space is limited, so when i use MR to generate 1TB data to alluxio, it fails writing in the middle:
2019-11-06 18:11:13,826 WARN [grpc-default-executor-1] alluxio.client.block.stream.GrpcBlockingStream: Received error alluxio.core.client.runtime.io.grpc.StatusRuntimeException: UNKNOWN: Could not find any valid local directory for obs-block-0002- for stream (GrpcDataWriter{request=type: UFS_FALLBACK_BLOCK
id:<PHONE_NUMBER>2
tier: 0
create_ufs_block_options {
  mount_id: 1
  fallback: false
}
medium_type: ""
pin_on_create: true
, address=WorkerNetAddress{host=node-ana-corevzxf, rpcPort=29999, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=node-ana-coreVzxf, rack=null)}})
2019-11-06 18:11:13,827 WARN [main] alluxio.client.file.FileOutStream: Failed to write into AlluxioStore, canceling write attempt.
alluxio.exception.status.UnknownException: Could not find any valid local directory for obs-block-0002- (GrpcDataWriter{request=type: UFS_FALLBACK_BLOCK
id:<PHONE_NUMBER>2
tier: 0
create_ufs_block_options {
  mount_id: 1
  fallback: false
}
medium_type: ""
pin_on_create: true
, address=WorkerNetAddress{host=node-ana-corevzxf, rpcPort=29999, dataPort=29999, webPort=30000, domainSocketPath=, tieredIdentity=TieredIdentity(node=node-ana-coreVzxf, rack=null)}})
	at alluxio.exception.status.AlluxioStatusException.from(AlluxioStatusException.java:154)
	at alluxio.client.block.stream.GrpcBlockingStream.toAlluxioStatusException(GrpcBlockingStream.java:261)
	at alluxio.client.block.stream.GrpcBlockingStream.checkError(GrpcBlockingStream.java:242)
	at alluxio.client.block.stream.GrpcBlockingStream.send(GrpcBlockingStream.java:100)
	at alluxio.client.block.stream.GrpcDataMessageBlockingStream.sendDataMessage(GrpcDataMessageBlockingStream.java:105)
	at alluxio.client.block.stream.GrpcDataWriter.writeChunk(GrpcDataWriter.java:197)
	at alluxio.client.block.stream.BlockOutStream.updateCurrentChunk(BlockOutStream.java:279)
	at alluxio.client.block.stream.BlockOutStream.write(BlockOutStream.java:158)
	at alluxio.client.file.FileOutStream.writeInternal(FileOutStream.java:257)
	at alluxio.client.file.FileOutStream.write(FileOutStream.java:220)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
	at java.io.DataOutputStream.write(DataOutputStream.java:107)
	at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
	at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.writeObject(TextOutputFormat.java:75)
	at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.write(TextOutputFormat.java:88)
	at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:440)
	at org.notmysock.tpcds.GenTable$DSDGen.map(GenTable.java:211)
	at org.notmysock.tpcds.GenTable$DSDGen.map(GenTable.java:163)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:188)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1840)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:182)
Caused by: alluxio.core.client.runtime.io.grpc.StatusRuntimeException: UNKNOWN: Could not find any valid local directory for obs-block-0002-
	at alluxio.core.client.runtime.io.grpc.Status.asRuntimeException(Status.java:530)
	at alluxio.core.client.runtime.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
	at alluxio.core.client.runtime.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at alluxio.core.client.runtime.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at alluxio.core.client.runtime.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at alluxio.grpc.GrpcChannel$ChannelResponseTracker$1$1.onClose(GrpcChannel.java:151)
	at alluxio.core.client.runtime.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at alluxio.core.client.runtime.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at alluxio.core.client.runtime.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at alluxio.core.client.runtime.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at alluxio.core.client.runtime.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at alluxio.core.client.runtime.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at alluxio.core.client.runtime.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:694)
	at alluxio.core.client.runtime.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at alluxio.core.client.runtime.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at alluxio.core.client.runtime.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at alluxio.core.client.runtime.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
	at alluxio.core.client.runtime.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
	at alluxio.core.client.runtime.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
	at alluxio.core.client.runtime.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
	at alluxio.core.client.runtime.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
	at alluxio.core.client.runtime.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
	at alluxio.core.client.runtime.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at alluxio.core.client.runtime.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

After that, i check AlluxioWorker log, it keep printing below logs:
2019-11-06 22:44:20,462 WARN  AsyncBlockRemover - block-removal-service-7: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:21,466 WARN  AsyncBlockRemover - block-removal-service-3: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:22,465 WARN  AsyncBlockRemover - block-removal-service-1: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:23,469 WARN  AsyncBlockRemover - block-removal-service-5: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:24,463 WARN  AsyncBlockRemover - block-removal-service-9: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:25,478 WARN  AsyncBlockRemover - block-removal-service-8: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:26,465 WARN  AsyncBlockRemover - block-removal-service-0: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:27,470 WARN  AsyncBlockRemover - block-removal-service-4: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:28,464 WARN  AsyncBlockRemover - block-removal-service-6: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:29,469 WARN  AsyncBlockRemover - block-removal-service-2: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:30,476 WARN  AsyncBlockRemover - block-removal-service-7: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:31,481 WARN  AsyncBlockRemover - block-removal-service-3: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:32,473 WARN  AsyncBlockRemover - block-removal-service-1: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:33,472 WARN  AsyncBlockRemover - block-removal-service-5: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:34,470 WARN  AsyncBlockRemover - block-removal-service-9: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:35,469 WARN  AsyncBlockRemover - block-removal-service-1: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:36,473 WARN  AsyncBlockRemover - block-removal-service-0: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:37,474 WARN  AsyncBlockRemover - block-removal-service-4: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:38,476 WARN  AsyncBlockRemover - block-removal-service-6: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:39,474 WARN  AsyncBlockRemover - block-removal-service-8: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:40,475 WARN  AsyncBlockRemover - block-removal-service-7: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:41,480 WARN  AsyncBlockRemover - block-removal-service-3: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:42,490 WARN  AsyncBlockRemover - block-removal-service-8: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:43,482 WARN  AsyncBlockRemover - block-removal-service-5: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:44,479 WARN  AsyncBlockRemover - block-removal-service-9: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:45,504 WARN  AsyncBlockRemover - block-removal-service-1: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:46,485 WARN  AsyncBlockRemover - block-removal-service-8: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:47,487 WARN  AsyncBlockRemover - block-removal-service-4: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:48,493 WARN  AsyncBlockRemover - block-removal-service-6: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.
2019-11-06 22:44:49,488 WARN  AsyncBlockRemover - block-removal-service-0: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.

To Reproduce

Use OBS or S3 as UFS, and enable writing the data to local disk first.
generate large data to alluxio, and alluxio space quota is not enough for all the data.

Expected behavior
uncommitted block can be removed, Alluxio worker should not keep deleting the uncommitted block.
Urgency
Normal
Additional context
Both OBS writing and AlluxioWorker need local disk spaces, there is chance to exausted the space.

@Hollyen Could you please clarify how did you achieve enable writing the data to local disk first. ?
Also, what is your write-type?
From the exception it looks like, you might be using ASYNC_THROUGH since I see pin_on_create: true. With this write type, new blocks won't be freed until they're written to UFS. This is to prevent data loss in ASYNC_THROUGH write scheme. So new writes might not find free space and you'd get a exception like you observe.

@ggezer We had a similar problem and the configuration we are currently using is
alluxio.user.file.readtype.default=CACHE
alluxio.user.file.writetype.default=CACHE_THROUGH

In addition to this, there are also a large number of errors reported in the logs.
2021-07-07 09:26:48,214 WARN  BlockTransferExecutor - Transfer-order: BlockTransferInfo{TransferType=SWAP, SrcBlockId=91292012904453, DstBlockId=60348149268480, SrcLocation=BlockStoreLocati
on{TierAlias=MEM, DirIndex=0, MediumType=MEM}, DstLocation=BlockStoreLocation{TierAlias=SSD, DirIndex=0, MediumType=SSD}} failed. {}.
alluxio.exception.WorkerOutOfSpaceException: Failed to find space in BlockStoreLocation{TierAlias=SSD, DirIndex=0, MediumType=SSD} to move blockId<PHONE_NUMBER>4453
        at alluxio.worker.block.TieredBlockStore.moveBlock(TieredBlockStore.java:365)
        at alluxio.worker.block.TieredBlockStore.moveBlock(TieredBlockStore.java:342)
        at alluxio.worker.block.management.BlockTransferExecutor.executeTransferPartition(BlockTransferExecutor.java:136)
        at alluxio.worker.block.management.BlockTransferExecutor.lambda$executeTransferList$0(BlockTransferExecutor.java:97)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2021-07-07 09:26:48,214 WARN  AlignTask - Insufficient space for worker swap space, swap restore task called.

It looks like this error was generated when moving block through the different levels.

Moving between tiers will fail the same since there are no space available for allocation as well.
Since your write type is not ASYNC_WRITE, I believe we need to understand why writes are going with pin_on_create: true in options. Can you check if you have pinning setting enabled for new files?

@ggezer
We did not change the relevant configuration.
Now I can't find the previous log to determine if it exists pin_on_create: true in the first place as this issue. But logs like
2019-11-06 22:44:40,475 WARN  AsyncBlockRemover - block-removal-service-7: invalid block state for block<PHONE_NUMBER>0, exception is Cannot remove uncommitted blockId<PHONE_NUMBER>0.

still exist until now.

@qian0817 The logs you shared are many days apart and are not consistent in their pin_on_create setting. So it seems we're looking at different issues here. Please share complete logs along with configuration and the test description for us to be able to look at it.

The log where the isssue originally appeared is here.
worker.log
and my configuration is
alluxio.master.hostname=hostname
alluxio.master.mount.table.root.ufs=s3://test/
alluxio.master.mount.table.root.option.aws.accessKeyId=xxx
alluxio.master.mount.table.root.option.aws.secretKey=xxx
alluxio.security.authentication.type=SIMPLE
alluxio.security.authorization.permission.supergroup=hadoop
alluxio.security.authorization.permission.umask=027
alluxio.worker.tieredstore.levels=2
alluxio.worker.tieredstore.level0.alias=MEM
alluxio.worker.tieredstore.level0.dirs.path=/mnt/ramdisk
alluxio.worker.tieredstore.level0.dirs.quota=15GB
alluxio.worker.tieredstore.level1.alias=SSD
alluxio.worker.tieredstore.level1.dirs.path=/mnt/ssd/0/alluxio
alluxio.worker.tieredstore.level1.dirs.quota=100GB
alluxio.user.file.readtype.default=CACHE
alluxio.user.file.writetype.default=CACHE_THROUGH
alluxio.underfs.s3.inherit.acl=false
alluxio.underfs.s3.default.mode=0751
alluxio.underfs.allow.set.owner.failure=true
alluxio.integration.worker.resource.cpu=4
alluxio.integration.worker.resource.mem=4096MB
alluxio.user.file.metadata.load.type=ONCE
alluxio.master.metastore=ROCKS
alluxio.master.daily.backup.enabled=true
alluxio.master.daily.backup.time=12:00
alluxio.master.audit.logging.enabled=true

At this point It's not clear why blocks are not selected for eviction/transfer.
Can you add below line to worker's log4j file and try to get a repro?
log4j.logger.alluxio.worker.block.BlockMetadataEvictorView=DEBUG

@ggezer
I can't restart it easily because it's an online environment. but i find
2021-07-03 01:50:33,449 WARN  BlockTransferExecutor - Transfer-order: BlockTransferInfo{TransferType=SWAP, SrcBlockId=71874868412419, DstBlockId=90027614797842, SrcLocation=BlockStoreLocation{TierAlias=MEM, DirIndex=0, MediumType=MEM}, DstLocation=BlockStoreLocation{TierAlias=SSD, DirIndex=0, MediumType=SSD}} failed. {}.
java.nio.file.FileSystemException: /mnt/ssd/0/alluxio/alluxioworker/90027614797842 -> /mnt/ramdisk/alluxioworker/90027614797842: No space left on device
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixCopyFile.copyFile(UnixCopyFile.java:253)
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:471)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at alluxio.util.io.FileUtils.move(FileUtils.java:207)
	at alluxio.worker.block.TieredBlockStore.moveBlockInternal(TieredBlockStore.java:904)
	at alluxio.worker.block.TieredBlockStore.moveBlock(TieredBlockStore.java:352)
	at alluxio.worker.block.TieredBlockStore.moveBlock(TieredBlockStore.java:342)
	at alluxio.worker.block.management.BlockTransferExecutor.executeTransferPartition(BlockTransferExecutor.java:141)
	at alluxio.worker.block.management.BlockTransferExecutor.lambda$executeTransferList$0(BlockTransferExecutor.java:97)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

I think there is a concurrency exception.

@qian0817 At peak capacity, it's possible swaps will fail. Then swap-restore task will activate and will defragment tier spaces. However, in your case swap-restore task is also failing due to bug I just fixed. However, It's still unknown why all your blocks are locked in.
If you can't restart your environment, you can still use Alluxio's setLogLevel command to enable these logs on workers.
https://docs.alluxio.io/ee/user/stable/en/operation/Basic-Logging.html#modifying-server-logging-at-runtime

$ ./bin/alluxio logLevel --logName=alluxio.worker.block.BlockMetadataEvictorView --target=workers --level=DEBUG
$ ./bin/alluxio logLevel --logName=alluxio.worker.block.TieredBlockStore --target=workers --level=DEBUG
$ ./bin/alluxio logLevel --logName=alluxio.worker.block.management.tier.SwapRestoreTask --target=workers --level=DEBUG

Will close it for now, feel free to reopen it and contact us if this is a strong request.


Suggestions on using Apex in OSS Projects?

Greetings all,
Wondering if anyone has any suggestions on best practices of using Apex from some OSS Lambda projects. As the project.json file has the iam policy in it it's probably not the best idea to commit that directly.
I'm wondering if the contents of the repo should be what would normally be in the functions folder and let the user scaffold everything up themselves, as demonstrated below:
[#] git clone github.com/polds/reponame
[#] mkdir new-project && cd new-project
[#] apex init
[#] cp -R ../reponame/ ./functions/

The second option I thought is to gitignore the project.json file and still have the user run apex init
Anywho, ideas?

For single functions I've been just suggesting submods (ex https://github.com/apex/capture), but yea for initializing a non-new project it needs some love

@polds If you're utilizing Terraform then it would just be pulling the role from wherever Terraform set the infrastructure up.

Going to close this issue.


fix indentation in summary-report cronjob

@fjogeleit

Did you test this change? Normally important is the value of nindent which decide for the indentation.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 91.792%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
4574

Relevant Lines:
4983

💛  - Coveralls

No problem, thank you for the quick response and release. It works now 👍


Handle empty spaces in attribute values

We need to support empty spaces in queries, e.g.
./chess_client_amd64 -query="facility:chess" | jq '.[].staffscientist'
"shanks"
"pollock"
"schuller"
"nygren"
"valentin"
"suchismita sarker"
"suchismita sarker, jacob ruff"
"jacob ruff, suchismita sarker"
"valentin"
"jacob ruff, suchismita sarker"
"jacob ruff, suchismita sarker"
"jacob ruff, suchismita sarker"

Then if I use staffscientist key I see the following:
./chess_client_amd64 -query="staffscientist:valentin"
[{"_id":"637784776768e89e30b25e81"...

but I can't look up others:
./chess_client_amd64 -query="staffscientist:suchismita sarker"
[]

fixed with the following commit: 4ffdaa592d9a429dbf44d4278382fb8c1e9ff203, and updated on production server.


0.5.7 breaking change: Previously exhaustive ZipError matches no longer exhaustive

I have code mapping between error types which is no longer exhaustive:
match imp.writer.add_directory(path, FileOptions::default()) {
    Err(ZipError::FileNotFound)             => return Err(VfsError::FileNotFound { path: path.into() }),
    Err(ZipError::Io(e))                    => return Err(VfsError::IoError(e)),
    Err(ZipError::InvalidArchive(e))        => return Err(VfsError::IoError(io::Error::new(io::ErrorKind::InvalidData, e))),
    Err(ZipError::UnsupportedArchive(e))    => return Err(VfsError::Other { message: e.into() }),
    Ok(())                                  => {},
}

    |
46  | match imp.writer.add_directory(path, FileOptions::default()) {
    |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ patterns `Err(PasswordRequired)` and `Err(InvalidPassword)` not covered
    |

It might be worth marking ZipError #[non_exhaustive] going forward, or adding a #[doc(hidden)] _DoNotFullyMatch variant to ZipError to avoid similar breakages in the future.

Whoops! That was a definite oversight. We were considering using a custom error branch for the new methods already, and I'll get those changes out on the 0.5 version instead.
@rylev, that'd mean changing the signatures of the new zipcrypto methods, making 0.5.7 a bad target for anyone to use. Could we yank it now?

I wouldn't yank it since it's a valid version on its own. Marking the enum as #[non_exhaustive] should take care of the issue.

It wouldn't? Anyone depending on version < 0.5.7 already has potentially been broken by the extra variants. We can't add them to a 0.5 version backwards-compatibly

Ah yes you're right. I still not sure yanking is the right approach, but perhaps it is.

It's kinda okay to just publish 0.5.8 and cross our fingers that users skip over 0.5.7. the version number would just be a lie, and I think yanking quickly might be the least bad option

Ok let's push out 0.5.8, and yank 0.5.7. Is this fix to change change the new zipcrypto functions to return a different error type?

I was thinking of returning a ZipResult<ZipFile, InvalidPassword>> from the crypto function, yea. We'd have to replace PasswordRequired with an UnsupportedArchive

Oops, that was my mistake. I didn't think about the possible consequences of adding another value to this enum. I think returning UnsupportedArchive is misleading because the archive is in fact supported. But I guess it does fix the compilation breakage.
If you remove the InvalidPassword error, you also have to fix the test which explicitly expects this error code on using a wrong password: https://github.com/mvdnes/zip-rs/blob/master/tests/zip_crypto.rs#L62
However, I do wonder: Why did you release this as a patch? You did major changes in the library and the entire ZipCrypto part was added. I think this warrants at least a minor version increase, doesn't it? A patch is a small change to fix some minor stuff, not an addition of significant refactorings and new features.

yanking

FWIW, I've added Err(_) => {} fallbacks such that my code in question should be able to handle both 0.5.7 and a future 0.5.8 with the new cases fixed, so I have no strong opinions on this.

Why did you release this as a patch?

0.x.y often guarantees nothing via semver.  By the other common standard, the intent here was fine - that standard being 0.major.minor.patch, with this release being a minor version bump, where extensions to the API and internal backwards-compatible changes would be appropriate.


Can't find handle_caches.jl in post step

With the recent v1.4 release, my workflow that includes this action has broken with this error at the Post Cache Julia step:
Post job cleanup.
Post job cleanup.
/usr/bin/docker exec  e11ee2023a2a40c471adf0aa339aa7bbb9ebb1b339140991e6729d4a4503f510 sh -c "cat /etc/*release | grep ^ID"
ERROR: SystemError: opening file "/home/ubuntu/actions-runner/_work/_actions/julia-actions/cache/v1/handle_caches.jl": No such file or directory

This action is running within a container built using an Ubuntu image on a self-hosted runner. I have a feeling that this issue is related to https://github.com/actions/runner/issues/716, where the ${{ github.action_path }} context variable points towards the incorrect action path. In this case, it's pointing towards the action path on the host that is mounted in the container rather than it's mount point within the container. Mount points are shown below from  docker inspect:
    "HostConfig": {
        "Binds": [
            "/home/ubuntu/actions-runner/_work/_temp/_github_home:/github/home",
            "/home/ubuntu/actions-runner/_work/_temp/_github_workflow:/github/workflow",
            "/var/run/docker.sock:/var/run/docker.sock",
            "/home/ubuntu/actions-runner/externals:/__e:ro",
            "/home/ubuntu/actions-runner/_work/_temp:/__w/_temp",
            "/home/ubuntu/actions-runner/_work/_actions:/__w/_actions",
            "tmp_volume:/tmp_volume",
            "/home/ubuntu/actions-runner/_work:/__w",
            "/home/ubuntu/actions-runner/_work/_tool:/__w/_tool"
        ],

I'll make a PR to fix this as it appears you can use the GITHUB_ACTION_PATH env variable to give the correct path rather than the context variable.

PR #78 added 👍

Closed by https://github.com/julia-actions/cache/pull/78


bufferhint#Popup will corrupt the vim when only one buffer is opened

Step:

Open one file in vim,
Call bufferhint#Popup.

Result: the only buffer will gone and when to some status "a   ## @".

I use temp fix below:
fu! bufferhint#Popup()
if ( 1 == len(filter(range(1, bufnr('$')), 'buflisted(v:val)')) )
return
endif
.....

Sorry, I can't reproduce this bug. Which version of Vim do you use?

It is because the default buffer name "[buffer]" - it conflicts with several plugins under windows platform including CtrlP. After I switch it to "~buffer~", it works correctly. Can you expose it as a preference? This is what I did:
https://github.com/jesse23/bufferhint/commit/60d3166d013812a163fb0a05d7c84d2316f2e945
By the way, this is the best buffer plugin I've met. Keep going!

A patch has been applied! Thanks for your suggestion : )
d129d0c6878428936bb5c2382b6306ffcf4998bf


Introsort Algorithm

Checklist

[X] I've looked at the contribution guidelines.
[ ] This pull request is complete and ready for review.

Description

In this request, I'll implement and explain the introsort algorithm, which is what Swift's .sort() method is based on!
TODO:

Finish introsort tests
Add performance tests
Write README

@rwash8347 looks really cool, looking forward to the readme!


Fixes warning in PullToRefreshView demo

Added a new PropType for Color type verification

cc @nicklockwood @ide

It looks like the logic in the ColorPropType function is inverted, unless I've misunderstood how it works.

Wouldn't it be better to define ColorPropType using the normal proptype composition functions? E.g.
React.PropTypes.oneOfType([React.PropTypes.string, React.PropTypes.number]);

It looks like the logic in the ColorPropType function is inverted, unless I've misunderstood how it works.
@nicklockwood

It actually works. React.PropTypes.* return an Error object if check fails.
The ColorPropType outputs error only if both number and string check fail.
But I think you are right, I just got lazy there.

Wouldn't it be better to define ColorPropType using the normal proptype composition functions?
That is actually a good idea.
Updating PR.

What do you think of https://github.com/facebook/react-native/pull/4866?
 Does it maybe make sense to update them all with the new ColorPropType in the same PR?

What do you think of #4866?
Quite related

Yeah, I think his PR is better. Sorry ¯_(ツ)_/¯

agree


Quote exec params in desktop files

This resolves Issue #4 (FFSSBs with spaces in names are not launched correctly from .desktop files by quoting appropriate parameters in the Exec line when generating the .desktop files. This should mean that FFSB names could include other special characters, including those normally expanded by the shell like asterisks (*), question marks (?), etc., though I wouldn't advise using them.

Oops, my fork's main branch seems to have gotten dirty. I'll rebase and force update this PR, @sebastianappler.

@sebastianappler I've cleaned up my fork & rebased this PR to get rid of the extra commits, so this is now ready for review. No rush on my part.

This fix worked for me and imo feels correct since other edge cases might appear for other breaking characters.
Thank you for the PR, @morgant!


fix(*-ansible-job): Change template registration procedure.

What does this PR do / why we need it
This PR fix a bug in the catalog-info.yaml registration procedure. As stated in the issue, the register action tries to use ${{ steps.publish.output.repoContentsUrl }} where the publish action is publish:github:pull-request that doesn't have repoContentsUrl output.
Which issue(s) does this PR fix
Fixes #161

This looks good to me @kadel can we get this merged?


Avoid copying series IDs for commit log use

This change makes the commit log use the series ID reference directly from the series itself rather than creating a new series ID copied from the written series ID for each commit log write.
We explicitly are allowing series IDs to be garbage collected when reusing a series so that we don't need to pay expensive ownership tracking semantics with contexts during a high frequency calls like writing to the commit log.
cc @xichen2020 @cw9 @prateek @ben-lerner

Coverage decreased (-0.06%) to 75.968% when pulling 8fa4a19b8e1c947b17b4c47f4a87d8574a57188d on master-avoid-copy-id-for-commitlog into 3d1c61958b68e0456c6d4936abb8176c9d710ba5 on master.

Coverage decreased (-0.06%) to 75.968% when pulling 8fa4a19b8e1c947b17b4c47f4a87d8574a57188d on master-avoid-copy-id-for-commitlog into 3d1c61958b68e0456c6d4936abb8176c9d710ba5 on master.


 In particular:

Avoid python object allocations for "InterDataForSeqGroup" objects: These objects have lot of fields, and most of the them were allocated dynamically. In this PR, these objects are pre-allocated and reused between runs. The pre-allocation is done per "number of sequences per group" (to support beam search), where most of the time we simply have one sequence per group.
Avoid python object allocations for "SequenceGroupMetadata" inside scheduler, in the same way as for (1).
Avoid python object allocations for "SequenceGroupToSample" inside SamplingMetadata.prepare(..)
Add an optimization to scheduler to be aware of "one sequence per group" case (since most of them we do not run beam search).
 Command used:
python3 benchmark_throughput.py --model meta-llama/Meta-Llama-3.1-8B-Instruct --backend vllm --input-len 512 --output-len 256 --num-prompts 1000 --tensor-parallel 1
Main branch from 08/05/2024:
Throughput: 19.48 requests/s, 14962.18 tokens/s
This PR:
Throughput: 24.32 requests/s, 18123.51 tokens/s

This is one way to introduce incremental prepare input. Will review soon.
cc @youkaichao @zhuohan123

I took a brief look and have the following impressions:

This optimization applies to continuous decoding steps. In other words, if requests in the batch are added or removed, the cache is invalid.
This optimization doesn't work with chunked prefill as it will schedule prefill and decode requests in the same batch.

Please correct me if I misunderstood anything. IIUC, It seems to me that this optimization should be done with multi-step worker, as it focuses exactly on the scenarios listed above. For general cases, this optimization introduces decoding specific branches (e.g., if no prefill tokens) and we should avoid that as possible.

@comaniac and I discussed the relation to multi-step. In general, a micro-benchmark shows that the benefit from flash-attn decode optimization in this PR is around 2-3%, and most of the benefit comes from python object allocation reductions. There are two possible ways to go about this:

Remove flash-attn optimization from this PR in favor of multi-step (since multi-step should cover flash-attn decode)
Proceed with flash-attn optimization until multi-step is fully enabled.

/ready

Thanks @alexm-neuralmagic this is great!

This optimization applies to continuous decoding steps. In other words, if requests in the batch are added or removed, the cache is invalid.

I think (1) isn't a concern since the proportion of steps in which reqs are added or removed is typically small.

This optimization doesn't work with chunked prefill as it will schedule prefill and decode requests in the same batch.

I didn't look closely enough yet but it presumably it will still work once in this mode once the prefill tokens have been exhausted i.e. once the batch returns to decode-only state?

Remove flash-attn optimization from this PR in favor of multi-step (since multi-step should cover flash-attn decode)
Proceed with flash-attn optimization until multi-step is fully enabled.

Given my comments above I think both parts of this would still make sense to include, but perhaps separate into two separate PRs anyhow?

Thanks @alexm-neuralmagic this is great!

This optimization applies to continuous decoding steps. In other words, if requests in the batch are added or removed, the cache is invalid.

I think (1) isn't a concern since the proportion of steps in which reqs are added or removed is typically small.

Isn't we continue adding/removing decoding requests at every step as long as there are new requests coming?

This optimization doesn't work with chunked prefill as it will schedule prefill and decode requests in the same batch.

I didn't look closely enough yet but it presumably it will still work once in this mode once the prefill tokens have been exhausted i.e. once the batch returns to decode-only state?

I'm not sure if prefill requests exhausting is a reasonable assumption especially for high QPS.
In summary I feel your assumptions make sense in offline batching, but I'm not sure about online serving. Meanwhile, considering the code complexity this PR will introduce, I personally would prefer to isolate it to the scenario it fits into. Of course we could still take the general Python code optimization.

Thanks @alexm-neuralmagic this is great!

This optimization applies to continuous decoding steps. In other words, if requests in the batch are added or removed, the cache is invalid.

I think (1) isn't a concern since the proportion of steps in which reqs are added or removed is typically small.

Isn't we continue adding/removing decoding requests at every step as long as there are new requests coming?

Offline is getting more interest lately with SDG etc. It obviously depends on the particular workload in terms of input/output split sizes and request patterns but I think even in the online serving case the batch constituency doesn't change for a significant proportion of steps. We should probably add metrics for this if we don't have already :)

This optimization doesn't work with chunked prefill as it will schedule prefill and decode requests in the same batch.

I didn't look closely enough yet but it presumably it will still work once in this mode once the prefill tokens have been exhausted i.e. once the batch returns to decode-only state?

I'm not sure if prefill requests exhausting is a reasonable assumption especially for high QPS.

Again I think this is very workload dependent, what about for "Write me an essay about x" type use cases.

In summary I feel your assumptions make sense in offline batching, but I'm not sure about online serving. Meanwhile, considering the code complexity this PR will introduce, I personally would prefer to isolate it to the scenario it fits into. Of course we could still take the general Python code optimization.

I agree about taking care to minimze additional complexity.

@njhill @comaniac thanks for the review and comments. @njhill I think it is a good idea to separate the optimizations. I will remove the flash-attn from this PR and will focus on python optimizations first, to avoid a conflict with mutli-step worker PR from Will.

@youkaichao PTAL and let's try to merge this PR by today or tomorrow.

working on fixing tests

fixed all of the failing tests, should be green now I hope.

Thanks @alexm-neuralmagic for this, it's a huge speedup!
It looks like there's a still couple of unaddressed comments though? https://github.com/vllm-project/vllm/pull/7162#discussion_r1710649611 and https://github.com/vllm-project/vllm/pull/7162#discussion_r1707510994

@njhill no problem, to address the leftover comments I have posted a small followup: https://github.com/vllm-project/vllm/pull/7364


[REFACTOR] Reorganize GenerationConfig DebugConfig and FFI

This PR reorganizes GenerationConfig, DebugConfig and FFI.

Internally, we now directly use the config object instead of json stream.
Request construction turns into engine side so it can make use of debug_config.
Ignore eos now moves to debug_config option.
Removes most string based re-export of gen conifg.

NOTE: this pr introduces more careful behavior to pass in ignore eos

ignore_eos should be passed as {"debug_config": {"ignore_eos": true}}
We need to run mlc_llm serve with mlc_llm serve model --enable-debug to enable debug endpoints


Albums

add basic albums to it

as during upload there's no modification available (right now) this functionality depends on completion of #21.


Radial argument fails to create radial network

Hi,
I've been trying to use the radial function of lomap and have been having issues. I am using the same set of ligands in /tests/radial and the functions in /examples/example_radial_graph.py.
import lomap
db_mol = lomap.DBMolecules('/root/Downloads/Lomap/lomap/tests/radial/', output=True, radial=True, hub = 'ejm44.mol2')
strict, loose = db_mol.build_matrices()
strict_numpy = strict.to_numpy_2D_array()
loose_numpy = loose.to_numpy_2D_array()
nx_graph = db_mol.build_graph()

# Drawing the network
import networkx as nx
import matplotlib.pyplot as plt
pos = nx.spring_layout(nx_graph)
nx.draw(nx_graph, pos)
nx.draw_networkx_labels(nx_graph, pos)
plt.show()

Creates the following graph:

Running with radial=False and no hub ligand selected creates the same, non-radial network.

hi @DreamCykl this is sort of working as intended.  There's a similarity cutoff (default 0.4) which prevents edges that are considered "bad".  So initially a radial graph is constructed, then some edges deleted and a connected graph then built on top of the remnants of the radial graph.
If you want to force a radial graph at any cost, then you can set the cutoff=0.0, e.g.:
db_mol = lomap.DBMolecules('radial/', output=True, radial=True, cutoff=0.0)
this will still create a graph with cycles, so depending on your opinion it might not be a radial graph, but it definitely contains a radial graph


feat: add start and end adornments to TextField

Changes

Added startAdornment and endAdornment as new props

@dexterca I know we chatted on Friday about you coming in and updating some of the styling. Do you have some time to look at this either today or tomorrow? I can rebase this to a feature branch if so.

@dexterca I know we chatted on Friday about you coming in and updating some of the styling. Do you have some time to look at this either today or tomorrow? I can rebase this to a feature branch if so.

Yes, I can take a look today.

I wonder if we should switch some some of the "special" text input fields like UnitNumberFormatField inputs like PercentFormatField or PriceFormatField to use these instead of putting their prefix/suffix inside the text? Worth experimenting with, at least. cc @dexterca

I wonder if we should switch some some of the "special" text input fields like UnitNumberFormatField inputs like PercentFormatField or PriceFormatField to use these instead of putting their prefix/suffix inside the text? Worth experimenting with, at least. cc @dexterca
Just experimenting here's what it looks like so you can kind of compare (I'm not sure I like it):
Today	Adornments

Excellent suggestion!  I dig this a lot personally 👍

@xdumaine @ynotdraw After talking thru some things with @HaleyWardo and looking at Xander's examples, I think we're going to limit adornments to icon, iconButton and iconButtonLink on this PR. We'll then rely on ...FormatFields for text-based "adornments". Thanks Xander for bringing this to our attention!

Converting to draft while @thedoryan works on some styling improvements. 🌟


Is it possible to hide/disable the navigation when only 1 slide is present?

When only one slide is present in the carousel, no navigation is possible, yet the navigation/pagination is still displayed which is a bit confusing for the user. Is there currently a possibility to hide/disable it in this case? Great component btw, really like it. 👍

It sure is! You can pass :paginationEnabled=false to turn off the dots

Thanks! For anyone wondering how I used this:
<carousel :per-page="1" :navigation-enabled="array.length > 1" :pagination-enabled="array.length > 1">
    <slide v-for="item in array" v-bind:key="item.id">
        {{item.text}}
    </slide>
</carousel>

@xJREB, It won't work. Because we'll have more than 1 item in the array even in the smaller width screens. So better we can check if the window width is greater than the mobile width.
<carousel :perPageCustom="[[0, 1], [438, 2], [768, 3], [992, 4]]" :paginationEnabled="window.innerWidth >= 437" :navigationEnabled="window.innerWidth < 437" :loop="window.innerWidth < 437" :autoplayTimeout="3000">     <slide v-for="(item, i) in array" :key="i" :data-index="i">         <!-- Slide 1 Content -->     </slide> </carousel>


Add jnv to homebrew-core

It would be nice if we add jnv to homebrew-core since they build the project in CI for all current macOS platforms.
Here is a formula I wrote. It should work but homebrew-core requires the repo to be >30 days old.
class Jnv < Formula
  desc "JSON navigator and interactive filter leveraging jq"
  homepage "https://github.com/ynqa/jnv"
  url "https://github.com/ynqa/jnv/archive/refs/tags/v0.1.1.tar.gz"
  sha256 "40fdb9de56d7ad55f7f3c52440ba7920f9aaffec07d59881b04e87a6facc5d28"
  license "MIT"
  head "https://github.com/ynqa/jnv.git", branch: "main"

  livecheck do
    url :stable
    strategy :github_latest
  end

  depends_on "autoconf" => :build
  depends_on "automake" => :build
  depends_on "libtool" => :build
  depends_on "rust" => :build

  def install
    system "cargo", "install", *std_cargo_args
  end

  test do
    assert_equal "jnv #{version}", shell_output("#{bin}/jnv --version").strip
  end
end

@pavelzw It seems like someone has taken care of it. Thank you for your cooperation.

https://github.com/Homebrew/homebrew-core/pull/167124

Great 👍🏻