added progress_callback in transcribe method

Hello, I wanted to thank you for your amazing work.
I have added a new Callable argument progress_callback to the transcribe method. This will allow users to track the progress of the transcription process through other frameworks, such as gradio.
Here's an example of how this could be used with gradio:
import gradio as gr

def run_transcribe(progressbar = gr.Progress()):
    def progress_callback(progress_value):
            progressbar(progress_value,desc="Transcribing..")
    progressbar(0,desc="Transcribing")
    model.transcribe(audio=audio,verbose=False,progress_callback=progress_callback)
    return "Done!"

block = gr.Blocks().queue()
with block:
    btn = gr.Button("run_transcribe")
    tb = gr.Textbox()
    btn.click(fn=run_transcribe,[],[tb])
block.launch()

Thank you!

Hi! Thanks for the suggestion. I'm actually thinking of a generator version of the function transcribe(), so the caller can iterate over the transcribed segments as they come in. Would that work for your use case?

I'm actually thinking of a generator version of the function transcribe(), so the caller can iterate over the transcribed segments as they come in. Would that work for your use case?

@jongwook Hey, sorry for jumping in, but I think this progress_callback thing actually can work as generator version as well, because you can also put any sort of other callbacks (for instance queue.put) inside.
Consider the following example:
import whisper
from multiprocessing import Queue, Process

def transcribe_producer(queue: Queue):
    model = whisper.load_model("small")
    audio = whisper.load_audio("javascript.wav")

    model.transcribe(
        audio,
        language="Russian",
        beam_size=5,
        best_of=5,
        segment_callback=queue.put,
    )["text"]

if __name__ == "__main__":
    q = Queue()
    p = Process(target=transcribe_producer, args=(q, ))

    p.start()

    while True:
        print(q.get())

I've changed the transcribe function signature as follows:
def transcribe(
    model: "Whisper",
    audio: Union[str, np.ndarray, torch.Tensor],
    *,
    verbose: Optional[bool] = None,
    temperature: Union[float, Tuple[float, ...]] = (0.0, 0.2, 0.4, 0.6, 0.8, 1.0),
    compression_ratio_threshold: Optional[float] = 2.4,
    logprob_threshold: Optional[float] = -1.0,
    no_speech_threshold: Optional[float] = 0.6,
    condition_on_previous_text: bool = True,
    segment_callback: Callable[[Tuple[float, float, str]], None],
    **decode_options,
):

And added following lines to the add_segment function:
        if segment_callback is not None:
            segment_callback((start, end, text))

For my use-case it works flawlessly and also pretty easy to implement. The cool thing also is that you can define any sort of callback not only queue.put. Look at the attached screenshot to get a feeling of what it looks like when running.

Thank you for comment. I think the idea of making a generator version is also a good one.
By yielding in the transcribe method instead of returning, the following code can be used.
In transcribe
            ....
            yield dict(
                num_frames=num_frames,
                seek=seek,
                text=tokenizer.decode(all_tokens[len(initial_prompt_tokens):]),
                segments=all_segments,
                language=language
            )

it could be used in the following way in gradio:
import gradio as gr

def run_transcribe(progressbar = gr.Progress()):
    progressbar(0,desc="Started Transcribing.")
    result = model.transcribe(audio=audio,verbose=False)
    for progress in result:
        progressbar(progress["seek"]/progress["num_frames"],desc="Transcribing..")
    result = progress

    return f"Done! result is {result}"

block = gr.Blocks().queue()
with block:
    btn = gr.Button("run_transcribe")
    tb = gr.Textbox()
    btn.click(fn=run_transcribe,inputs=[],outputs=[tb])
block.launch()

One issue I encountered during testing was that num_frames and seek needed to be included in the yield in order to track the progress. Calling len(list(result)) to use in for i,progress in enumerate(result) triggers an exhaustion of the generator.
Thanks !

Thank you for comments. I think the idea of making a generator version is also a good one.

You are welcome, in terms of progress tracking I like the yield solution more, because it's more general and you can for instance use it for streaming results. However, I think it's good to be aware of one small disadvantage of generator over the callback approach suggested initially.
Consider this code sample:
import time
import whisper

def some_handler(chunk: dict):
     pass

def time_consumer():
    time.sleep(100000)

model = whisper.load_model("small")
audio = whisper.load_audio("sample.wav")

for result in model.transcribe(audio):
      some_handler(chunk)
      time_consumer() # time consuming process

In that case, model will just sleep and not produce any input, because it runs on the same interpreter and it's locked by the current thread. I think with callbacks you can overcome this issue using some asynchronous iterators and running the generator in a separate process.
If you wish I can make a PR or a gist to show you the concept.

Here's an example of how it could look if you combine callbacks with generators (you may do the same without async, just replace asyncio.sleep with time.sleep and remove async / await keywords.
import asyncio
from multiprocessing import Queue, Process
import whisper

def transcribe_producer(queue: Queue):
    model = whisper.load_model("tiny")
    audio = whisper.load_audio("javascript.wav")
    model.transcribe(
        audio,
        language="Russian",
        beam_size=5,
        best_of=5,
        segment_callback=queue.put,
    )

async def transcribe_async(queue_check_interval: int = 1):
    q = Queue()
    p = Process(target=transcribe_producer, args=(q,))
    process_finished = False

    p.start()
    print(f"Started the process {p.pid}")

    while not process_finished or not q.empty():
        if q.empty():
            print(f"No results available, sleeping for {queue_check_interval} seconds")
            # This will work the same on windows
            # We can use async to make this waiting non-blocking
            await asyncio.sleep(queue_check_interval)
            continue

        yield q.get(timeout=0)

        # Try joining process to see if it's finished
        if not process_finished:
            p.join(timeout=0)

            if not p.is_alive():
                print(f"Process with id = {p.pid} was finished, yielding the rest of the queue")
                process_finished = True

async def io_blocking_task(processing_time: int = 1):
    print(f"IO blocking task for {processing_time} secs")
    await asyncio.sleep(processing_time)

async def main():
    async for item in transcribe_async(1):
        print(f"Got an element: {item}")
        await io_blocking_task(2)

    print("Async task finished")

# This line is crucial to not make processes recursively
if __name__ == "__main__":
    asyncio.run(main())

any update on this?


Missing authentication in documentation

I am migrating from the unmaintained react-contentful and trying to use @contentful/react-apps-toolkit instead.
So I am wondering how to use the authentication for useFieldValue is set up.

react-contentful is not an official Contentful library, so I don't have much context about that.
With useFieldValue you can access the field values of the entry you are currently editing. Therefore you do not need to handle authentication.
To access other entries using the CMA, you can use useCMA and, again, authentication is already handled for you.
Did you run into any issues?


ADD README translate to Chinese

Please  help me merge it, I want to join your, make many people  know this project.
In addition, I will further familiarize myself with the project and make my contribution to the energy level。

Thank you @devon-ye feel free to share with your community this is great and super helpful!


[doc] Fix get() function name

It feels like it's a left-over after some codemod, since there's no at() method on this type.

It's here on live docs: https://doc.rust-lang.org/regex/regex/struct.Regex.html#examples-1

r? @BurntSushi

Ah yup, nice find! @bors r+

:pushpin: Commit e8ecb43 has been approved by BurntSushi

:hourglass: Testing commit e8ecb43c22e8d844c472192a876701085cc0a369 with merge 3806646f4d2022a919a2e0c9a9cd58790c973e59...

:sunny: Test successful - status-appveyor, status-travis
Approved by: BurntSushi
Pushing 3806646f4d2022a919a2e0c9a9cd58790c973e59 to master...


citation source needed

Why not mention you are drawing the contents of the jargon from https://github.com/hemanth/functional-programming-jargon? This some else's writing and should be credited.

@epogrebnyak hey, we already have a link to Rust World pointing to this project on hemanth/functional-programming-jargon project, check out on their README page

The main project is under MIT lisence, so it would good to adhere to it? Maybe a link back to the main repo would be a good thing to do.

@hemanth Ok sure, we agree ;) BTW thanks for your initial work! We've turned it into a usable Rust library based on your idea!

Citation to the original project is now included in https://github.com/JasonShin/fp-core.rs/pull/61 !


Implement property specific validation messages

Now that 0.1.0 is out, I'd like to implement more specific validation messages. These should offer more insight into why the validator might yield a warning; here are some examples:

isColor: rgb(255, 255, 255, 1) - too many arguments, expected 3.
isColor: rgb(255, 255) - too few arguments, expected 3.
isColor: rgb(255, 255, 255,) - unexpected trailing comma.
isColor: rgb(255 255 255) - expected arguments to be comma-separated.

Essentially what we have to do now is to break down the return conditions of the custom validators. Take isLength as an example. The return condition should be extracted into individual conditions that yield a message if the condition is failing. So, for example, the function checks that the value does not end with . - instead of a boolean test, we can return a message here instead:
if (!endsWith(int.number, '.')) {
  return {
    type: 'invalid',
    message: 'At least one number should follow the decimal point.',
  };
}

Suggestions for types of messages to return would be very welcome, as well as pull requests that implement these new messages. 😄

Suggestions for types of messages to return would be very welcome

The only convention we have in stylelint is that messages either start with "Expected" or "Unexpected". Your examples are currently very close. Would it be possible to adopt this convention? Saving us from having to adapt them at the stylelint end.

isColor: rgb(255, 255, 255, 1) - Expected 3 arguments, currently too many.
isColor: rgb(255, 255) - Expected 3 arguments, currently too few arguments
isColor: rgb(255, 255, 255,) - Unexpected trailing comma.
isColor: rgb(255 255 255) - Expected arguments to be comma-separated.

As an aside, it's very exciting to see css-values come together. This depth of linting is stunning.

Would it be possible to adopt this convention?

Certainly! The idea is that no adaptation will be necessary. 😄

As an aside, it's very exciting to see css-values come together. This depth of linting is stunning.

Thanks, this means a lot. 😀
I noticed that currently the generator doesn't handle validation messages properly yet, so I've been working on this - https://github.com/ben-eb/css-values/commit/ad5b7ca14ca2f0a518627628ee0e1b88a481bfd5 implements this checking for single values, just multiple values currently outstanding. Once multiple values are implemented I'll try and work on an example conversion and link it here. I'd like to tackle the rest of the more complex properties myself instead of writing validation messages, which I would like to delegate to anyone interested.

So, I've extended support for validation messages in multiple values, and have done an example conversion which you can check out at https://github.com/ben-eb/css-values/commit/056a394bc9e1156e7c04fdb7575ce713b2ec6d7c.
Would be really great to have some help to convert the rest of the validators to return appropriate validation messages. The general format of this is that each validator might need to pass a value through (returning false) to another validator, or if it looks like a mistake then it should return a message for it.
For example, a validator that validated a hypothetical foo function would return false if the function's name was not foo (such that it could be passed to another validator), but would return a message instead if any of foo's arguments were incorrect. You can use the shouldReturnResult utility to check that either the validation failed and a message was returned, or that the value passed validation. If false, you are expected to pass the value on to the next validator, which most likely will be isVariable.
for(10px); // returns false, passes through to the next validator
foo(10px); // valid, returns true
foo(bar);  // invalid, returns a message "Expected foo to be called with a length value."


[WIP] Remove Fatalf calls in non-test goroutines for tests/benchmarks

Replace goroutines calling Fatalf in tests/benchmarks to address go vet warnings.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @lostluck
CC: @riteshghorse @youngoli
First run at fixing these. Had a lot of trouble with blocking on channel reads piping out of the goroutine, but found a suggestion to replace it with Errorf() + immediate return out instead.


Zesty Hackday Work:

refs zestyzesty/hack-day#8

Add block indentation linting
Log linted items to console
Add linting for bare strings

:clap: Thanks @matteddy!


Controller tests cases failing

Hi,
I have the following in spec_helper.rb. I am trying to login a user before running controller test cases.
RSpec.configure do |config|
  config.include Sorcery::TestHelpers::Rails
  config.include Sorcery::TestHelpers::Rails::Controller, type: :controller
  config.include Sorcery::TestHelpers::Rails::Integration, type: :feature

  config.before(:suite) do
      DatabaseCleaner.strategy = :transaction
      DatabaseCleaner.clean_with(:truncation)
      FactoryGirl.lint
      login_user(FactoryGirl::create(:user))
  end
end

When executing rspec spec/controllers,  fails with error
undefined method login_user' for nil:NilClass (NoMethodError)`
I am not sure why this is failing. Is this a bug with sorcery or I am doing something wrong?

Since there's been no response for a month, I'm closing the issue. @navaneeth if you still have problems, let me know, I'll try to help.


Bump version to v1.0.0-beta.1.0

Closes https://github.com/OpenAssetIO/OpenAssetIO-Manager-BAL/issues/124.

Looks fine, but just calling out you seem to have a different versioning scheme to openassetio betas, e.g. https://github.com/OpenAssetIO/OpenAssetIO/tags shows,
* v1.0.0-beta.2.2

* v1.0.0-beta.2.1

so where is this rev coming from?
Actually, looking closer, looks like you've got one convention in Git tags, but another in Python versions, as I just spotted this here
dependencies = ["openassetio>=1.0.0b2.rev2"

Yeah you got it. When we decided to use semver for beta versions, i.e. -beta.x.y, we found that's not directly supported by PyPI/PEP versioning.  The closest we could get was a "post-release segment" (we prefer the synonym  "rev", but if you look on PyPI it uses "post" regardless :disappointed: )

That concerns me with the reference to 'post', as it's meant to mean no behaviour changes, just like a docs fixup that was found to be in error.
https://peps.python.org/pep-0440/#post-releases
Their note

The use of post-releases to publish maintenance releases containing actual bug fixes is strongly discouraged. In general, it is better to use a longer release number and increment the final component for each maintenance release.

My experiences of pre-releases on PyPi come from cruiz, and I've done a few, but I keep it simple with a semver + aN/bN where N is a monotonically increasing integer.
https://github.com/markfinal/cruiz/releases
I then keep Git tags and PyPi release numbers in sync.
Just saying :)

That concerns me with the reference to 'post', as it's meant to mean no behaviour changes, just like a docs fixup that was found to be in error.
https://peps.python.org/pep-0440/#post-releases
Their note

The use of post-releases to publish maintenance releases containing actual bug fixes is strongly discouraged. In general, it is better to use a longer release number and increment the final component for each maintenance release.

It is a bit of a square peg round hole issue.  We primarily focussed on semver where our scheme is fine, and then shimmed the Python versioning as best we could to fit :cry:
Like I said previously, given the length of the beta period, and that we expected products would be shipping with it and/or pipelines using it, we needed to convey breaking vs. non-breaking changes in the beta version identifier. I think once 1.0.0 is reached proper, future pre-releases, if any, can be shorter-lived, not used in production, and so use a single incrementing number.


Fix 'Display from byte offset' input

Fixes #9483
making sure we initialize the field with 0 offset even if parsed query has undefined offset

Future improvement suggestion - Browse.tsx and Logs.tsx are both similar - they either display a directory or a file. We can create a parent component that houses shared logic. Then each can derive from the parent, adding their own functionality

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4618/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4621/
Test PASSed.

@calvinjia PTAL, thanks

alluxio-bot, merge this please.


feat: add meta info (like title) to share intent object on Android

Summary
When sharing a link on Android, the browser adds the page title to the share intent. This is stored in the Intent.EXTRA_TITLE property. The library currently has no way of accessing this extra information. This PR adds a meta object to the share intent with a title property. This way consumers of the library can access the title property in their code. On iOS the meta object will be empty.
The code has been set up so it's easily extendible. It will be a small change to add another extra property.

Todo

[x] Update README.md

Issue
fixes: #38

@achorein I've implemented your changes. Good call on adding to the other example projects, I forgot about that. Anything else we can optimize here?

Looks good to me 👍

Would it be possible to add the meta image as well?


Add rule-ctl.py script

This is the first draft of rule-ctl.py. This is the first README:
draft
OWASP CRS Rule Control Script
This script aims to help when a bulk change on configuration files is needed. rule-ctl.py can, for example, change the value of an action on all rules, or can add/remove/rename a tag on each rule in a file, or can add/remove a transformation function only in rules that match range<PHONE_NUMBER>90, etc...
Example Usage
There're only two mandatory parameters: --config and --filter-rule-id.
--config set the target config file
--filter-rule-id a regex that matches only rule ids to change
For example, if you want to add a new tag on each rule in file REQUEST-933-APPLICATION-ATTACK-PHP.conf you can do:
python3 util/rule-ctl/rule-ctl.py \
    --config rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf \
    --filter-rule-id ^933.+ \
    --append-tag foo
    --dryrun

--dryrun sends to stdout the result of changes and prevent writing changes on file. It's a good idea to always check all commands with dryrun before overwrite the target configuration file.
You can even alphabetically sort tag list while adding new tags:
python3 util/rule-ctl/rule-ctl.py \
    --config rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf \
    --filter-rule-id ^933.+ \
    --append-tag foo
    --sort-tag
    --dryrun

Variables

--append-variable: Append a variable on the variable list of selected rules
--remove-variable: Remove exact matching variable from selected rules
--replace-variable: Replace variable on selected rules
--replace-variable-name: Replace just variable name part on selected rules

Examples
Replace all ARGS variable with ARGS_GET even if ARGS:foo
python3 rule-ctl.py --config tests/rules.conf \
    --filter-rule-id ^.* \
    --replace-variable-name ARGS,ARGS_GET \
    --dryrun

Tags

--append-tag: Append a new tag to the tag list on selected rules
--remove-tag: Remove tag from tag list on selected rules
--rename-tag: Rename tag on selected rules
--sort-tags: Alphabetically sort tag list on selected rules

Examples
Append a new tag foo and sort tag list
python3 rule-ctl.py --config tests/rules.conf \
    --filter-rule-id ^.* \
    --append-tag foo \
    --sort-tags \
    --dryrun

Transformation Functions

--append-tfunc: Append a new transformation function on selected rules
--remove-tfunc: Remove a transformation function on selected rules

Examples
Append t:lowercase to all selected rules (you don't need the t: prefix)
python3 rule-ctl.py --config tests/rules.conf \
    --filter-rule-id ^.* \
    --append-tfunc lowercase \
    --dryrun

Actions

--replace-action: Replace action on selected rules
--uncond-replace-action: Unconditional replace action on selected rules
--remove-action: remove action from selected rules

Examples
Replace action severity:CRITICAL with severity:INFO and set a new message on rule id 125
python3 rule-ctl.py --config tests/rules.conf \
    --filter-rule-id ^125 \
    --replace-action severity:CRITICAL,severity:INFO \
    --uncond-replace-action 'msg:this is a new message for rule 125' \
    --dryrun

CTL

--append-ctl: Append a new ctl action on selected rules

Examples
Remove rule id 1337 on rule 125 by adding ctl:ruleRemoveById=1337. Do it on main rule (skipping chained rules if present)
python3 rule-ctl.py --config tests/rules.conf \
    --filter-rule-id ^125 \
    --append-ctl ruleRemoveById=1337 \
    --skip-chain \
    --dryrun

Others

--target-file: Set the target file where changes will be saved (default: use file set by --config)
--skip-chain: Skip chained rules
--dryrun: Do not write any changes, just output the results
--debug: Show debug messages
--silent: Used with --dryrun and --debug doesn't write and shows only debug messages
--json: Used with --dryrun return the msc_pyparser JSON output instead of ModSecurity file

This is a very, very useful script that makes msc_pyparser much more accessible. I'll try to get @airween to review this PR and provide some feedback, so this can be finished and merged asap.

Meeting decision June: @airween and @fzipi test this one (already self-assigned as reviewer).

Very minor point... some lines in multi-line command examples in README.md are missing trailing \

Running command (as non-root user) results in warnings:
WARNING: Couldn't open 'parser.out'. [Errno 13] Permission denied: '/usr/local/lib/python3.8/dist-packages/parser.out'
Generating LALR tables
WARNING: Couldn't create 'parsetab'. [Errno 13] Permission denied: '/usr/local/lib/python3.8/dist-packages/parsetab.py'

Running command (as non-root user) results in warnings:
WARNING: Couldn't open 'parser.out'. [Errno 13] Permission denied: '/usr/local/lib/python3.8/dist-packages/parser.out'
Generating LALR tables
WARNING: Couldn't create 'parsetab'. [Errno 13] Permission denied: '/usr/local/lib/python3.8/dist-packages/parsetab.py'

How did you install the msc_pyparser package? (From PyPI, or downloaded the package?)

on ubuntu 20.04 LTS
sudo apt install python3-pip
sudo pip3 install -r util/rule-ctl/requirements.txt

Hmm... it's interesting. What do you get when you type this command?
>>> print(msc_pyparser.__version__)

You should get 1.1. If you get something different, try to add the version in the requirements.txt:
msc_pyparser==1.1

I have cloned theMiddle's repository, and have tried to repdoduce your issue, but I couldn't.

import msc_pyparser
print(msc_pyparser.version)
1.1

Which Python version do you have? I could reproduce your issue on Python3.5 - looks like this version does not install the parsetab.py and parser.out files, or the interpreter looks then under the module's directory, not under the /usr/local.
I have to review it - workaround: try to run as root the script at first time, that will generate the necessary files.

import sys
sys.version_info
sys.version_info(major=3, minor=8, micro=5, releaselevel='final', serial=0)

Looked like warnings were not fatal, script seemed to be producing expected results.
As per your workaround, I can confirm running script as root, has no warnings, and results in the creation of: /usr/local/lib/python3.8/dist-packages/parser.out and /usr/local/lib/python3.8/dist-packages/parsetab.py . Subsequently running as non-root then has no warnings.

I've tried adding/removing transformation functions and tags.
Might be expected behaviour, but attempts to remove multiple tags seems to silently fail to remove all the tags:
python3 util/rule-ctl/rule-ctl.py --config rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf --filter-rule-id ^933.+ --remove-tag platform-multi --remove-tag application-multi --dryrun
above doesn't remove platform-multi tag
whereas it does seem possible to remove multiple transformation functions:
python3 util/rule-ctl/rule-ctl.py --config rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf --filter-rule-id ^933.+ --remove-tfunc lowercase --remove-tfunc urlDecodeUni --dryrun
also I kind of expected (possibly wrongly/unreasonably) that transformation function removal would occur before appending so that it would be possible to strip transformation functions and re-add them in a specific order:
python3 util/rule-ctl/rule-ctl.py --config rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf --filter-rule-id ^933.+ --remove-tfunc lowercase --remove-tfunc urlDecodeUni --append-tfunc urlDecodeUni --append-tfunc lowercase --dryrun
No hardship to have to do this as two commands; but maybe worth a note in the documentation (assuming it's not already there and I haven't failed to spot it)?

@theMiddleBlue : We've had a few comments for this welcome script. Can you incorporate these and finish the PR? Would be nice to merge this in the coming days.

I have found a strange behavior in the rule-ctl.py script: comparing the output with the original file. It seems the line (re)numbering is wrong:
132,133c134
<     SecRule MATCHED_VARS "@pm =" \
<         "capture,\
---
>     SecRule MATCHED_VARS "@pm =" "capture,\

(the first action after the operator and its argument goes into same line, not the next one)
There are few lines where the line increment statements are commented, eg. here. May be this caused the behavior above.
I can take a look it later, if you need.

Meeting decision July:
https://github.com/coreruleset/coreruleset/issues/2141#issuecomment-874288876
@airween will review it and send a fix - @theseion wants to test it

@airween How did you produce that issue? The simple things I've tried all work.

Hi @theseion,
here is how I did it:
$ git branch
* add-script-rule-ctl
  v3.4/dev

$ python3 rule-ctl.py --config ../../rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf --filter-rule-id ^933.+ --append-tag foo

$ git diff ../../rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf
diff --git a/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf b/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf
index 49974f1..271b171 100644
--- a/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf
+++ b/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf
@@ -14,6 +14,7 @@

 SecRule TX:EXECUTING_PARANOIA_LEVEL "@lt 1" "id:933011,phase:1,pass,nolog,skipAfter:END-REQUEST-933-APPLICATION-ATTACK-PHP"
+
 SecRule TX:EXECUTING_PARANOIA_LEVEL "@lt 1" "id:933012,phase:2,pass,nolog,skipAfter:END-REQUEST-933-APPLICATION-ATTACK-PHP"
 #
 # -= Paranoia Level 1 (default) =- (apply only when tx.executing_paranoia_level is sufficiently high: 1 or higher)
@@ -58,12 +59,12 @@ SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAME
     tag:'paranoia-level/1',\
     tag:'OWASP_CRS',\
     tag:'capec/1000/152/242',\
+    tag:'foo',\
...
...
@@ -125,12 +126,12 @@ SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAME
     tag:'paranoia-level/1',\
     tag:'OWASP_CRS',\
     tag:'capec/1000/152/242',\
+    tag:'foo',\
     ctl:auditLogParts=+E,\
     ver:'OWASP_CRS/3.3.0',\
     severity:'CRITICAL',\
     chain"
-    SecRule MATCHED_VARS "@pm =" \
-        "capture,\
+    SecRule MATCHED_VARS "@pm =" "capture,\
         setvar:'tx.php_injection_score=+%{tx.critical_anomaly_score}',\

See the last few lines. The new lines with tag:'foo' are expected, but the SecRule MATCHED_VARS "@pm =" "capture,\ isn't.

@theseion can you confirm (or refute :)) this behavior? I mean after the transformation (eg. append a new tag) the firs action moves into same line as variable and operator are.

Another strange behavior: if there is no any tag at the rule, append-tag has no effect. (I know there no such CRS rule, but if anybody wants to use its own set, it would be a good feature.)

Sorry for the delay.

@theseion can you confirm (or refute :)) this behavior? I mean after the transformation (eg. append a new tag) the firs action moves into same line as variable and operator are.

Yes, I can confirm that. Only appears to affect chained rules.

Another strange behavior: if there is no any tag at the rule, append-tag has no effect. (I know there no such CRS rule, but if anybody wants to use its own set, it would be a good feature.)

The rules_tag_append function is very complicated. I'm not surprised that happens.

Yes, I can confirm that. Only appears to affect chained rules.

only appears in case of chained rules, if there is any non-chained rule before that :)

The rules_tag_append function is very complicated. I'm not surprised that happens.

I think the problem is that the code doesn't follow the number of appended lines (as an offset/shift value), only increments by 1 the lines. I can take a look later.

@theMiddleBlue, @theseion - would you like to review this patch?
This solves only the append-tag formatting, but I think we can follow this pattern to solve the line shift problem (see above).

I'm a bit busy but I'll try to get look at the patch in a couple of days.

@theMiddleBlue, @theseion - would you like to review this patch?
This solves only the append-tag formatting, but I think we can follow this pattern to solve the line shift problem (see above).

thanks for fixing it! I'll try it asap

The patch works. I've started refactoring the script heavily because I didn't understand how it works. If something useful comes out of that I'll contribute that.

The patch works. I've started refactoring the script heavily because I didn't understand how it works. If something useful comes out of that I'll contribute that.

Let me know if I can help you anything.

There is still work going on for this PR and reviews are happening. So I suggest we skip it for the agenda of the meeting tonight. Feel free to merge if you guys think you are done.

FYI: I have a working implementation. I'm writing tests at the moment, which will take me a while but we'll have a proper test suite for all the actions at the end.

Hey @theseion, the expectation of getting a test suite for this is very nice. Do you have a time line for this?

I'm working on it now. I hope to have it ready by monday.

I've opened a new PR with the refactored version of the script. The PR includes a pytest test suite.

Closing this in favour of #2193


[Failed pipeline] ContainerInstance: ContainerGroups

failed run: https://github.com/Azure/ResourceModules/actions/runs/4048059875

There seems to be an issue with the deployment:
  "details": [
    {
      "code": "ResourceDeploymentFailure",
      "message": "The resource provision operation did not complete within the allowed timeout period."
    }

successful run: https://github.com/Azure/ResourceModules/actions/runs/4048059875


Added SpringBoot3-GraphQl sample

Issue #715:
Description of changes:
Added SpringBoot3-GraphQl sample
Commands to test:
sam build
sam local start-api
curl -X POST http://localhost:3000/graphql -d '{"query":"query petDetails {\n  petById(id: \"pet-1\") {\n      id\n      name\n      breed\n      owner {\n          id\n          firstName\n          lastName\n      }\n  }\n}","operationName":"petDetails"}' -H "Content-Type: application/json"
Test results:
{"data":{"petById":{"id":"pet-1","name":"Alpha","breed":"Bulldog","owner":{"id":"owner-1","firstName":"Joshua","lastName":"Bloch"}}}}
By submitting this pull request

[ ] I confirm that my contribution is made under the terms of the Apache 2.0 license.
[ ] I confirm that I've made a best effort attempt to update all relevant documentation.

Hey Fredric do mind putting some description on this pr for context?

Hey @jonife, I just linked the relevant issue and also added instructions to test. Thanks for the feedback.

One suggestion: I think the official name is GraphQL


Spell Misses vs Resists Issue

I've been doing a bit of investigation into this with Shadow Priest work and wanted to get confirmation of a defect I believe is happening here. Under Spell.kt for attackRoll I noticed there are two EventResults out of this one being for RESIST and HIT, however I believe we are missing either MISS or another RESIST in the 3rd roll for resist calculations:
        val resistAvgReduction = spellResistReduction(sp, school)
        if(isBinary) {
            // Make a third roll for full resist or not
            val fullResistRoll = Random.nextDouble()
            val fullResistMod = if(fullResistRoll < resistAvgReduction) { 0.0 } else { 1.0 }
            finalResult = Pair(finalResult.first * fullResistMod, finalResult.second)
        } else {
            finalResult = Pair(finalResult.first * (1 - resistAvgReduction), finalResult.second)
        }

Notice under binary calculations, the fullResistMod could be 0 or 1, however the finalResult will remain with the existing EventResult. What I noticed was some dots (e.g. SiphonLife, Corruption, and some of the new ones I'm working on) would have EventResult.HIT, but do 0 damage in the event logs over the course of the dot (Priest ones I'm working on), which isn't actually possible in the game as this would probably fall under a full resist and the rotation would try to recast the dot, but since the dot is actually applied as 0 the entire dot duration is negated in these binary resists.
I noticed in the existing SiphonLife / Corruption that they could essentially never miss due to never receiving EventResult.MISS (Also side note, I believe there is a second defect here where the dots are not correctly snap shotting as the dot damage is calculated on every tick):
if(result.second != EventResult.MISS) {
  sp.sim.target.addDebuff(CorruptionDot(sp))
}

I wasn't sure if the first EventResult.RESIST in attackRoll was intended to be a EventResult.MISS and this last one should be EventResult.RESIST or have both of them as a EventResult.RESIST though since the game doesn't actually differentiate

I wasn't sure if the first EventResult.RESIST in attackRoll was intended to be a EventResult.MISS and this last one should be EventResult.RESIST or have both of them as a EventResult.RESIST though since the game doesn't actually differentiate

You're definitely right about DoT spells checking for MISS instead of RESIST to apply their DoT!  They should be checking the latter, and that would inflate their connection rate a bit.  Great find!

I've been doing a bit of investigation into this with Shadow Priest work and wanted to get confirmation of a defect I believe is happening here. Under Spell.kt for attackRoll I noticed there are two EventResults out of this one being for RESIST and HIT, however I believe we are missing either MISS or another RESIST in the 3rd roll for resist calculations:

From what I understand about spellcaster mechanics, chance to miss from spell hit rating is distinct from chance to resist due to school resistance. School resistance cannot be mitigated by spell hit rating, so it would not increase the initial spell miss chance.  This implies a distinct roll for resistance, which is what is implemented here.
I think it's a little confusing since the game doesn't really differentiate well between a spell miss and a fully resisted crit/hit, since both just seem to show up as "resist".  Both are possible under the base mechanics, though.
However, the distinction between a fully-resisted hit or crit and a spell miss is important, since one is caused by the target's school resistance, and the other is caused by lack of spell hit rating.
On the other hand, the Spell mechanics are not returning the proper partial resist events.  This doesn't have any impact on damage, but would be needed if deeper analysis into partial resists is desired.

Snapshotting is not implemented - back when I was writing that, I didn't know whether spells were still snapshotted or not for TBC, so elected to just skip it.
It should be fairly trivial to implement - just add a new constructor argument to each DoT implementation class, send the character's current spell power on application, then reference that value instead of the character's current spell power on each tick.
Let me know if you'd like me to fixup the warlock spells!  Excited to see the priest work!

Ok I think with my situation I was expecting a resist event for determining which proc to fire, but I was passing result.first directly to my spells due to snap shot damage. I can easily add zero checks though and just consider it resisted with a miss or true binary resist no problem.


style: improve page style

I think we'd better move markdown link outside the general share button, or users would easily missed it

I think we'd better move markdown link outside the general share button, or users would easily missed it

IMO, The markdown link is also a shared content and I think the same content should be put together🤔🤔. Waiting for other's views

You may request more reviewers..

Is there an online demo?

Is there an online demo?

@moonming https://contributor-graph-oafp202af-apiseven.vercel.app/

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.
🔍 Inspect: https://vercel.com/apiseven/contributor-graph/CSeKUyLBbRs1veiGq1VoKSEzbMNQ
✅ Preview: https://contributor-graph-git-improve-style-apiseven.vercel.app

Here

The search box is too long
The comparison function is not obvious
Remove download button
Enhanced sharing function
All contributors to Apache APISIX repositories are displayed by default

All contributors to Apache APISIX repositories are displayed by default

Is it a new feature request?  Currently the default display is apache/apisix for one repository

Legend font size is still too small for README

@LiteSun need to merge to the download PR

@LiteSun need to merge to the download PR

Merged.


quant keeps going back to LFQ

Describing the issue
When I change the quant to "LFQ+MBR" in the search task and open it again, it has reverted to "LFQ" evefytime.
Reproducing the error
Should happen if you try it. In case its relevant I'm doing top down with all the other suggest param changes or defaults.

Protein database
mouse including isoforms

Mass spectrometry file types
mzML from Orbi Fusion with EThcD

Settings
see above

Example datafiles
I think this is a setting issue, the files that are chosen seem irrelevant

Expected behavior for GUI
should stay on what was chosen
Screenshots

Desktop information
AutoGeneratedManuscriptProse.txt
Additional context

Thanks for bringing this to our attention! It will be addressed in the next release.


[Question] about intersperse function.

Hi!
During preprocessing, when add_blank is True in hparams, some weird intersperse function (here) intersperses an index, which is out of vocabulary bounds (item=len(symbols)), between each pair of adjacent tokens.
My first guess was that this token plays the role of some pauses between tokens, as pause token was not presented in vocabulary. So while training, all pauses sift to this token.
Then, as it's name state, I treated it as some blank token, which is needed to absorb all "noises" between adjacent tokens, as for other tokens to present more clear phonemes. There I thought it may also be used to learn transformations from one phoneme to another, which is not a part of any of two adjacant phonemes itself, but a separate part. but if so, why is it a common token for all gaps?
So, what is the real purpose of this blank token?
this question is more addressed to the authors, but any guesses are welcome.
thanks in advance.

Can someone explain the real purpose of interperse function ? I'm confuse with it a little bit.

Can someone explain the real purpose of interperse function ? I'm confuse with it a little bit.

as long as I have been working with GradTTS I treated the interspersed token, for which item argument stands in the respective function, as kind of "space" token, inserted between each two adjacent phonemes and denoting the amount of, say, "silence" between them which model should learn to pronounce. thus, each non-"space" token should be filled only with sound immediately relevant to this token, while all pauses, skips and spaces should be delegated to this token. moreover, in case of noisy data, all irrelevant (background) buzz could be fed into this tokens hence purging it from other actual phonemes. otherwise, id est if this "space" token is omitted, all the noise and silence would have no other choice but to be memorised as parts of actual phonemes tokens hence contaminating them.


Remove trailing spaces from files

Just a small change to remove all the trailing spaces in some of the files.

Thank you!


Correct issue #160 stating that Swift 5.3 was released

Swift 5.3 is not released yet.

thank you @MaxDesiatov !

@MaxDesiatov you've been making some good contributions lately. would you be interested in being a guest author for an issue?

@jessesquires thanks for the invitation! I'm not a very avid reader of Twitter and Swift Forums, so I'm not sure I'd be able to gather enough links in two weeks for a proper issue. I'd be happy to post a few links to an issue if I stumble upon anything that seems good enough.

While that helps, I would not be able to currently compile an issue without the help of @fassko — so we're here to help. If you would be interested in (helping to) write an issue, I'd love to help!

To add to that, I'd say the lead author is primarily responsible for writing up the content in markdown and filling out the template for the post, not necessarily sourcing all of the links. I think we have a pretty strong group contributing links. 😄
But, no pressure!


Code coloring does not work everywhere

See screenshot of the settings + an actual hx file.

Please copy code here, so we can test it in our environments.

I just took the sample code from the Fonts & Color settings page and corrected minor mistakes.
package;

#if (definedFlag && undefinedFlag)
#error "Error!!"
#else
import util.Date;
#end

/* Block comment */
/**
 Document comment
**/
class SomeClass implements IOther { // some comment
    private var field = null;
    private var unusedField:Float = 12345.67890;
    private var anotherString:String = "Another\nStrin\g";
    public static var staticField:Int = 0;

    public static function inc() {
        staticField++;
    }
    public function foo(param:AnInterface) {
        trace(anotherString + param);
        var reassignedValue:Int = SomeClass.staticField;
        reassignedValue ++;
        function localFunction() {
            var a:Int = $$$;// bad character
        };
    }
}

Verified in 14.1.  Some of the color coding works, but not all of it.


Add option to keep only all childrens but not parent

Take a look on this :

When you select "England" for example, the table of selected values will correspond to the childrends of England, but in the input, it will be marked England.
It would be nice to have an option that instead, in this case, would list all the childrens rather than the parent
Like here https://www.jqueryscript.net/demo/Multi-Select-Checkbox-Tree-treeSelector/

Hi @bastien70, what about grouped prop? This prop allows showing only children without groups if you set it to fasle. In your case, it will be Chelsea and West End.

I tried to use your example and add grouped: false at the end.
const treeselect = new Treeselect({
  parentHtmlContainer: domElement,
  value: [],
  options: options,
  listSlotHtmlComponent: slot,
  placeholder: 'Rechercher ...',
  emptyText: 'Aucun résultat',
  disabledBranchNode: false,
  direction: 'auto', // auto, top, bottom
  id: 'treeselect_input',
  alwaysOpen: false,
  isGroupedValue: true,
  grouped: false
})

slot.addEventListener('click', (e) => {
  e.preventDefault()
  console.log(treeselect.value, treeselect.groupedValue)
})```

![image](https://user-images.githubusercontent.com/58396014/200875655-ae0885dc-f87b-43a9-9207-401cdd41886a.png)

Oooooh okay, yes, you're right now :) Thank you :D

Closed as completed.


chore: update compatibility matrix in docs

@iakovn , I have updated the table for MI versions as users have run into this when using production cloud. I will update the table with open-api versions in another PR


The URL and widgets out of sync

Describe the bug 🐛
Currently, the URL and the state of the UI became out of sync after a page refresh.
To Reproduce 🔍
Steps to reproduce the behavior:

Go to https://87b2c.codesandbox.io/
Search for "Phone"
Click on "Speck"
Click on "OtterBox"
Reload the page
Click on the browser back button
Click on the browser back button
The state of the URL is not synced with the widgets

The refinementList contains two refinements but the URL only contains the query. The URL should drive the search and the UI but it's not the case. The issue occurs for two reasons:

We're not able to compute the parameters of a widget with only getWidgetSearchParameters function. We also depend on the getConfiguration lifecycle. That's why we use the state of the first render to create the new SearchParameters on router.update (popstate). Note that we capture it on the first render and we never update it. It leads to issue once we remove a refinement that was part of this initial state. We could always use the latest up to date state (helper.state) but it won't completely solve the issue because of the point below.
The function getWidgetSearchParameters don't use the uiState as source of truth. At step 8 the uiState does not contain anything related to the refinementList so we return the SearchParameters untouched. Since we get the state of a previous render the refinements are still applied on the SearchParameters. We should clear the refinement for the attribute in any cases and re-apply the refinement from the values in the uiState.

Ideas to fix the issue:
An intermediate solution to fix this issue is to:

always use the latest state inside the router.update callback (like we already do inside onHistoryChange)
update the getWidgetSearchParameters lifecycle to clear the previous state of the part that is controlled by the widget. We can safely then re-apply the refinement with the value stored inside the uiState.

The optimal solution would be to have a getWidgetSearchParameters that is self-contained. Right now it's not the case, the widget depends on getConfiguration + getWidgetSearchParameters to compute the SearchParameters. But with a self-contained function we could compute the SearchParameters from a fresh state and avoid the issue describe above (which also means that we can get rid of getConfiguration at the same time). It requires a breaking change.
Environment:
The issue occurs with all the version of InstantSearch that supports routing.

This is fixed in v4, since we redesigned connectors to be self-contained with getWidgetSearchParameters & getWidgetState.


add a flag that enforces boolean expressions where booleans are expected

a flag that enforces:

any expression with the following operators: ||, &&, ! to be boolean
a condition expression to be of type boolean in the if statement and in the conditional expression (ternary operator)

related: https://github.com/Microsoft/TypeScript/issues/7256

This would be extremely non-idiomatic for JavaScript (as in, I expect almost no codebases to successfully compile under such a flag). I think TSLint would be the place for a rule like this.

Agree with ryan. Just today :

It is highly idiomatic to do such checks in JSX/TSX/React :rose:

@aleksey-bykov https://github.com/palantir/tslint/issues/253 (amongst others)
This is blocked on https://github.com/palantir/tslint/issues/680 however

@myitcv Thanks for letting me know. We at our project stopped using TsLint for this exact reason, the rules can only validate the syntax, not the semantics. We developed a homemade rule runner that gets a type checker and runs against the a type checked code of the whole application rather than an AST of a single file.
It would not be a problem to put such rule together.

@aleksey-bykov in the absence of support for this kind of rule in tslint have you shared your custom rule runner?

I cannot share the real runner, but I can outline how to make one:
import * as ts from 'typescript';
import * as fs from 'fs';

export function run(
    projectFileOrFolder: string
): void {

    const options: ts.CompilerOptions = {
        project: projectFileOrFolder
    };

    const host = tlb.toCompilerHost(options);

    const program = ts.createProgram(
        [],
        options,
        host
    );

    const checker = program.getTypeChecker();
    const rootFileNames = program.getRootFileNames();

    rootFileNames.map(rootFileName => {
        const rootNode = program.getSourceFile(rootFileName);
        analyze(rootNode, checker);
    });
}

function analyze(rootNode: ts.Node, checker: ts.TypeChecker): void {
   // TODO: put your rule here
}

function toCompilerHost(
    options: ts.CompilerOptions
): ts.CompilerHost {
    return {
        readFile: (fileName: string): string => {
            return fs.readFileSync(fileName, 'utf8');
        },
        fileExists: (fileName: string): boolean => {
            return fs.existsSync(fileName);
        },
        getSourceFile: function (filepath: string) {
            try {
                return ts.createSourceFile(
                    filepath,
                    fs.readFileSync(filepath, 'utf8'),
                    options.target,
                    true
                );
            }
            catch (e) {
                return undefined;
            }
        },
        writeFile: () => null,
        getDefaultLibFileName: () => 'lib.d.ts',
        useCaseSensitiveFileNames: () => true,
        getCanonicalFileName: (filename: string) => filename,
        getCurrentDirectory: () => '',
        getNewLine: () => '\n'
    };
}

then comes the simplified rule

function analyze(rootNode: ts.Node, checker: ts.TypeChecker): void {
    visitEachNode(rootNode, node => {
        if (node.kind === ts.SyntaxKind.IfStatement) {
           checkIfStatement(<ts.IfStatement>node, checker);
        } else if (node.kind === ts.SyntaxKind.ConditionalExpression) {
           checkConditionalExpression(<ts.ConditionalExpression>node, checker);
        }
    });
}

function checkIfStatement(node: ts.IfStatement, checker: ts.TypeChecker): void {
    var type = checker.getTypeAtLocation(node.expression);
    if ((type.flags & ts.TypeFlags.Boolean) !== ts.TypeFlags.Boolean) {
        console.warn('if expression is not boolean');
    }
}

function checkConditionalExpression(node: ts.ConditionalExpression, checker: ts.TypeChecker): void {
    var type = checker.getTypeAtLocation(node.condition);
    if ((type.flags & ts.TypeFlags.Boolean) !== ts.TypeFlags.Boolean) {
        console.warn('if expression is not boolean');
    }
}

function visitEachNode(rootNode: ts.Node, visit: (node: ts.Node) => void): void {
    ts.forEachChild(rootNode, child => {
        visit(node);
        visitEachNode(node, visit);
    });
}

also take a look at the official example on how to make a linter: https://github.com/Microsoft/TypeScript/wiki/Using-the-Compiler-API#traversing-the-ast-with-a-little-linter

@aleksey-bykov thanks for the pointers


[Model] Add support for Qwen2 for embeddings

FIX https://github.com/vllm-project/vllm/pull/5611
FIX #5827
This works for ssmits/Qwen2-7B-Instruct-embed-base
>>> from vllm import LLM
>>> model = LLM("ssmits/Qwen2-7B-Instruct-embed-base")
>>> outputs = model.encode("Hello!")
Processed prompts: 100%|█████████████████████████████████████| 1/1 [00:00<00:00,  2.10it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
>>> outputs
[EmbeddingRequestOutput(request_id='0', outputs=EmbeddingOutput(embedding=3584), prompt_token_ids=[9707, 0], finished=True)]

However, this doesn't work for Alibaba-NLP/gte-Qwen2-7B-instruct since its config still says it is a Qwen2ForCausalLM, where we have been relying on embedding models to be of the type XModel: https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct/blob/main/config.json

+1 this would be great

I think ppl can edit the config file to change it from  Qwen2ForCausalLM to Qwen2Model

I think ppl can edit the config file to change it from Qwen2ForCausalLM to Qwen2Model

Qwen2Model is not supported by vLLM.

Should we expect a merge in the near future?

@Semihal thanks for the reminder! I just merged with main so if it is green we can land

Can you pass "pytest tests/models/test_embedding.py"? I got "AssertionError: Not all values are within 0.01 of 1.0" with model "ssmits/Qwen2-7B-Instruct-embed-base" .

@Nickydusk No it does not pass, it seems there is a correctness issue between the implementations so this is not ready to land. This is low on my priority to investigate so if anyone would like to take over this PR, feel welcome and ping me for review.

@youkaichao @WoosukKwon @zhuohan123 @simon-mo @DarkLight1337
Could you please check this PR and merge it? gte-Qwen2 is a Qwen2ForCausalLM embedding model which is quite efficient.

@mgoin @prattcmp @waters222 @Semihal @starmemda
I have opened a new PR to directly support 'gte-Qwen2' embedding models without modifying its Qwen2ForCausalLM architecture. We can collaborate on reviewing the code and merge it in the near future.

Can this feature be merged?

@ybbz The output is not correct/matching HF, as seen in the test. Anybody is welcome to debug it!

Ping in case you forgot about this @mgoin . It should be quite straightforward to add embedding models now.

Closing as superseded by #10184


Even more recipe-ception

Enable this:
wrangles:
  - recipe:
      name: recipe.wrgl.yml

Should also support the other options like the connector:
https://wrangles.io/python/connectors/recipe

Running recipe as wrangle works. For now, all intermediate column will become part of the dataFrame.
Will look into making the output of wrangle-recipes only the desired columns.
wrangles:
  - recipe:
      name: recipe.wrgl.yaml
      output_columns:
        - col1
        - col2

A problem can occur when a wrangle adds or removed rows from the data frame (wrangle-recipe), making it have a non-matching number of rows in original df and wrangle-recipe df


Issue with Reconstructing Bcell clonal lineages

Hi,
I am working with mouse B cell repertoires processed with MIXCR (for consistency for this project I am still using MIXCR v3.0.13). The alignment was done on the VDJregion so I can export all the required fields for the downstream analysis. My file has the following fields present: cloneId, cloneCount, cloneFraction, nSeqCDR3, nSeqCDR1, nSeqCDR2, nSeqFR1, nSeqFR2, nSeqFR3, nSeqFR4, aaSeqCDR3, bestVHit, bestDHit, bestJHit, bestCHit.
I am able to load the data in R with immunarch and look into it in terms of diversity etc... But now I would like to reconstruct Bcell clonal lineages.
When I use the following command I am getting the error below. Any insight would be very helpful, Thanks !
best
Julien
command:
Lineages_align <- immdata_mixcr$data %>%
seqCluster(seqDist(immdata_mixcr$data, .col = 'CDR3.nt', .group_by_seqLength = TRUE), .perc_similarity = 0.8) %>%
repGermline(.threads = 2, .species = "MusMusculus") %>%
repAlignLineage(.min_lineage_sequences = 6, .align_threads = 2, .nofail = TRUE)
error:
Error in map2():
ℹ In index: 1.
ℹ With name: clones_100.
Caused by error in get():
! object 'generate_germline_sequence' not found
Run rlang::last_error() to see where the error occurred.
Warning messages:
1: There was 1 warning in mutate().
ℹ In argument: length_value = map_chr(.y, ~ifelse(all(.x == .x[1]), yes = .x[1], no = glue("range_{min(.x)}:{max(.x)}"))).
Caused by warning:
! Automatic coercion from integer to character was deprecated in purrr 1.0.0.
ℹ Please use an explicit call to as.character() within map_chr() instead.
ℹ The deprecated feature was likely used in the immunarch package.
Please report the issue at https://github.com/immunomind/immunarch/issues.
This warning is displayed once every 8 hours.
Call lifecycle::last_lifecycle_warnings() to see where this warning was generated.
2: In seqCluster(., seqDist(immdata_mixcr$data, .col = "CDR3.nt", .group_by_seqLength = TRUE),  :
Number of sequence provided in .data and .dist are not matching!
System: Mac osx 12.3.1, R 4.2.0 Immunarch 0.9.0

Same problem

Same here


Parent Resolve Correlation to Child Route

Just a question: Do children routes get the resolve value from the parent?

No, resolve is only the results of the matched route's resolve functions. If multiple routes want the same functionality, I would split it out to its own function.
function resolveSomething() {
  return Promise.resolve("something");
};

export default prepareRoutes([
  {
    name: "Parent",
    path: "p/:pID",
    resolve: {
      something: resolveSomething
    },
    children: [
      {
        name: "Child",
        path: ":cID",
        resolve: {
          something: resolveSomething
        }
      }
    ]
  }
]);

Do you have a use case where that doesn't work well?

I figured that was the case. My only concern was making the same API call multiple times even though the parent already got the information. I've solved it already saving info in the app, but I was just wondering if you had implemented something as well. Thanks for the quick responses as usual.

Yes, caching function results is probably the best bet. The once function exported by the @curi/helpers package provides very basic caching. There are also some caching/memoization packages that provide more robust solutions. (or you can always roll your own solution).


Sliding can get stuck in between slides when tapped.

I found that if you swipe then quickly tap the swiped slide, sometimes the slide animation will stop mid-slide and be "stuck" in between slides instead of always snapping to either the next or original slide. Any way to avoid this? (I tested using the default swiper demo)

Sorry to bother  this day.  but what's the params or things I should do to fix this? I got this problem in latest Swiper. Yeah, I tried set 'freeModeMomentum' to false. but the experience is weird. Thanks!


Fix compilation on build targets listed below MIPS

A stray if causes a syntax error leading to failed compilations on targets listed below the line with MIPS

Thank you!
But it is strange that make cross-gcc (which includes builds for MIPS & MIPS64) did not reveal this problem.


[posix] fix echo reply for registered addresses

This commit fixes handling echo request on added manually addresses by BR2. The test case is used in Thread v1.2 Certification for BBR DUT e.g. in C_5.9.20 or C_5.10.2 when DUT acts as Router, REED or FED.
Topology:
-BR1 -----(eth)------HOST--------
/
BR2

Ran the test 3 times:
$ ./script/test build_otbr_docker
$ THREAD_VERSION=1.2 VIRTUAL_TIME=0 FULL_LOGS=0 ./script/test build
$ sudo VERBOSE=1 PACKET_VERIFICATION=1 THREAD_VERSION=1.2 VIRTUAL_TIME=0 ./script/test cert_suite tests/scripts/thread-cert/border_router/test_border_router_reachability.py
...
PASS 3 FAIL 0 tests/scripts/thread-cert/border_router/test_border_router_reachability.py

I've rebased the changes and run the test again. It's passing now. CI is passing as well. It seams there is no problem now.
Should I still add this test to CI? I don't think the test case is covered by other test.

I've rebased the changes and run the test again. It's passing now. CI is passing as well. It seams there is no problem now.
Should I still add this test to CI? I don't think the test case is covered by other test.

Sure it's better to have more test coverage.

@lmaciejonczyk @edmont , is this PR still necessary?


Add SPF 'redirect' mechanism check

SPF can contain a mechanism called redirect. This mechanism indicates that the SPF of that domain must be applied.
Example:
dig txt plus.google.com | grep spf
plus.google.com.	5	IN	TXT	"v=spf1 redirect=google.com"

In this case, the SPF of google.com has to be applied to plus.google.com.
Therefore, I think that redirect is like include but without needing to indicate any all policy.
Feel free to correct me if you think I'm wrong.

Hey @carlospolop ,
Thanks!


Widen default Skeleton page width

Skeleton uses 960 px for content width, but this can cause problems with larger tables leading to a lot of vertical scrolling. A wider reasonable width could be used, maybe 1280 or even 1600.
@lilliputten Please look into this.

PR #13.

@lilliputten Easy mode is to put "Fixes #13" in the PR message which will auto-close this issue on merging.


[flutter_tools] StateError: Bad state: No running isolate during flutter run -d chrome

on beta/3.8.0-10.1.pre
flutter --no-color run --machine --track-widget-creation --device-id=chrome --start-paused --web-renderer html --dart-define=flutter.inspector.structuredErrors=true lib/main.dart
This version is using dwds: 17.0.0 https://github.com/flutter/flutter/blob/beta/packages/flutter_tools/pubspec.yaml#L14
StateError: Bad state: No running isolate (inspector is not set).content_copy
at ChromeProxyService.inspector(chrome_proxy_service.dart:79)
at ChromeProxyService._prewarmExpressionCompilerCache(chrome_proxy_service.dart:222)
at <asynchronous gap>(async)

cc @elliette

FYI @annagrin as well

This should have been fixed by https://github.com/dart-lang/webdev/pull/1938, which is included in DWDS v.18.0.0: https://github.com/dart-lang/webdev/releases/tag/dwds-v18.0.0
Currently 18.0.0 is on head: https://github.com/flutter/flutter/pull/121569
@christopherfujino does this need a cherry-pick or is it alright to roll out through the normal process?

This should have been fixed by dart-lang/webdev#1938, which is included in DWDS v.18.0.0: https://github.com/dart-lang/webdev/releases/tag/dwds-v18.0.0
Currently 18.0.0 is on head: #121569
@christopherfujino does this need a cherry-pick or is it alright to roll out through the normal process?

regular rolling SGTM. I will close this as fixed on master. Thanks!


Worker keeps running out of memory

In file nhl_etl.ipynb
https://github.com/lesterlxy/JDE_final/blob/527474e0c3dc36dad30cef968ab113703ed7ab61/nhl_etl.ipynb#L400

fixed in commit 7c442c0


jpm: icon issue

The latest update added a known problem: the addon icon breaks.
https://bugzilla.mozilla.org/show_bug.cgi?id=1141839
To fix it, install.rdf needs to be edited after "jpm xpi" to remove the icon entry, and icon.png needs to be added to the addon's root.
Edit: same issue on another addon, for reference: https://github.com/antoyo/auto-sort-bookmarks/issues/22

Please upvote the bug too please, mozilla doesn't care enough yet, maybe if it gets more votes...

  Anyway,  there is a workaround for the icon path that I use here too: https://github.com/sblask/firefox-simple-form-fill I'll make sure it works before making another release...

I see what you did there :P Feels kinda icky to me though, putting the url you expect to see after compilation into the source you compile in the first place. Kinda like breaking encapsulation.

There was actually a problem with the icon. While fixing I also changed it to only use svg now. Will be in version 1.0.2 which is now awaiting Mozilla's review. The workaround wasn't my idea, it's documented here: https://developer.mozilla.org/en-US/Add-ons/SDK/Tools/package_json
Not sure it's related to your bug in bugzilla, but I could not use relative urls for the icon in the button description, it had to be a self.data url...


Meta.exclude not always honored with DocumentSerializer/DynamicDocumentSerializer

Issue Summary
Serializers can get into a state where they do not appear to honor the 'exclude' statement in the Meta class.
Simplified Example
Below is a sub-section of the code I am using.
Model
from mongoengine.document import DynamicDocument
from mongoengine.fields import StringField, IntField

class EnvironmentsDocument(DynamicDocument):

    name = StringField(required=True)
    typeName = StringField(required=True)
    version = IntField(required=True)

    meta = {
        'collection': 'environments',
        'indexes': [
            {
                'fields': ['typeName'],
            },
            {
                'fields': ['name'],
            },
            {
                'fields': ['-version'],
                'unique': True,
            }
        ]
    }

Serializer
from rest_framework_mongoengine.serializers import DynamicDocumentSerializer

class EnvironmentsSerializer(DynamicDocumentSerializer):

    class Meta:
        model = EnvironmentsDocument
        exclude = ('id', )

View Set

class NameVersionViewSet(CreateModelMixin,
                          ListModelMixin,
                          RetrieveModelMixin,
                          MongoGenericViewSet):
    """
   list - Get a list of possible environment names
   retrieve - Get latest document for an environment
    """

    lookup_field = 'env_name'

    serializer_class = EnvironmentsSerializer

    queryset = EnvironmentsDocument.objects.all()

    def get_queryset(self):
        qs = super(NameVersionViewSet, self).get_queryset()
        name = self.kwargs.get(self.lookup_field, None)

        if name is not None:
            return qs.filter(name=name)
        else:
            return qs

    def list(self, request, *args, **kwargs):
        qs = self.get_queryset().distinct('name')
        return Response(qs)

    def get_object(self):
        return self.get_queryset().first()

Result
Still always shows 'id' in my JSON output.
{
    "id": "552ec2a12d3f0119ed23da99",
    "name": "oak",
    "typeName": "tree",
    "version": 1
}

Possible Fix
Waiting until the last possible second in get_fields(self) to do exclusions.

class DocumentSerializer(serializers.ModelSerializer):
    """
    Leaving out some of the code here....
    """
    def get_fields(self):
        """
        Leaving out code until just before return...
        """
        for field_exclude in exclude:
            if field_exclude in ret:
                del ret[field_exclude]

        return ret

Should be fixed now.
I've put a test for that
Not sure about DynamicDocument though


[SSoC 2.0] Number to Words Convertor Project Completed

🛠 Issue (Number)

Closes #1296
Description
Please include a summary of the change. Please also include relevant motivation and context. List any dependencies that are required for this change.

Enter the number in the input box
Using Javascript, convert the entered number into words
Number to words conversion is done instantly on entering the number

Type of change

[x] New feature (non-breaking change which adds functionality)

Checklist:

[x] I have made this from my own
[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] The title of my pull request is a short description of the requested changes.

ATTACH SCREEN-SHOTS / DEPLOYMENT LINK

@TusharKesarwani please review this pull request and requested to merge it into the main branch of this repository for a valuable contribution under SSoC 2.0 program
Ans Thanks for assigning me this project.

@TusharKesarwani Sir, this pr is ready to merge.
I requested to merge this pr for a contribution under SSoC 2.0 program

Resolve merge conflicts

@TusharKesarwani Sir, i have resolved the conflicts, please again review this pr and merge it into the main branch for a successful contribution under SSoC 2.0 program

@TusharKesarwani Sir, please also review this pr and merge it also

resolve conflicts first

@TusharKesarwani @zugzwang03 Sirs, please again review this pull request and merge it for a successful contribution under SSoC 2.0 program

@TusharKesarwani @zugzwang03 Sirs, please merge this pr then i will resolve the conflicts of the other pr

@TusharKesarwani Sir, i requested to merge this pull request as now this pr does not have any conflicts at this time.

Project is already present in the repo


Build steps?

This is another speculative issue for discussion, not something that's necessarily going to be implemented. The question is whether & how to support build steps to run when producing a wheel.
Should we?
So far, I have always said that flit is a simple tool for simple packages, i.e. those with no build step. I think these are the vast majority of packages we publish,  and the simplicity argument is still an important one against adding such features. I also don't know much about compilers, which are one of the most common sorts of build steps.
The best argument I see in favour is that you may start packaging something with flit, then add something that requires a build step. With no support for that, you have to throw away your packaging metadata and workflows to switch to another tool.
I expect that the majority of people looking at this issue will be those who want such features, so I'm not going to pay too much attention to +1s.
How would it work?
Briefly, the wheel build would copy files (the files in the package and... some others?) to a temporary location, and invoke external tools there to do the build.
It would need some way to figure out the compatibility effects of the hooks:

Some do not restrict compatibility (e.g. minifying JS code)
Some restrict compatibility towards the platform they're run on (e.g. standard compilation)
Some restrict compatibility towards a specific platform regardless of where they're run.
Some can be asked to target a particular platform (cross compilation)

It would also need to elegantly combine multiple build hooks - e.g. if you have a C extension module and a Qt designer .ui file to compile, it should be possible to specify those as two independent build steps, without either needing to be aware of the other. Or maybe this composition should be the responsibility of a proper build system which we invoke.

Not rocket science to convert your metadata format to mine either. Or a standardized toml metadata based on setup() arguments could be used. Or flit could provide a service to write the dist-info folder and perhaps the manifest, and the build system could do everything else. Or flit could provide useful command line tools like 'publish' and the build step could do almost everything else including writing the wheel.
Tricky part would be communicating which files had been built and where they go between flit and build tool. Collaboration could be "write your .so to a new folder that is zipped up to be a wheel", build step may or may not do the .py copy itself, and flit could handle slightly tricky issue of wheel manifests and metadata.
https://bitbucket.org/dholth/enscons/src/f31e60b7e29a9dc1bc4e314b7457727e5619db9e/pyproject.toml?at=default&fileviewer=file-view-default

Also enscons doesn't really require the metadata to be in toml, it's all dicts internally.

Yeah, there could be a flit2encsons tool which would help people upgrade to the more flexible packaging when they need it. :-)

I expect that the majority of people looking at this issue will be those who want such features, so I'm not going to pay too much attention to +1s.

I am actually looking to see an SVG diagram of those "build steps" whatever they are.

I'm not quite sure what you mean. The discussion is about supporting whatever build steps people might want (compiling, cython, JS minification...), not writing any specific build steps.

The cargo build script is a decent example. http://doc.crates.io/build-script.html
Cargo runs the script, the script does anything it wants to, and then cargo goes back to its own strength which is building Rust packages. Seems analogous to the place flit wants to occupy.
@techtonik here are a couple of examples of non-C-compiler build steps, with ascii art instead of SVG.
My pysdl2-cffi wrapper has a build dependency tree excerpted in part here, printed with SCons --tree=all:
    +-sdl/ttf.py
      +-sdl/__init__.py
      | +-builder/__init__.py
      | +-builder/build_image.py

ttf.py needs to be generated after sdl/init.py and they both depend on the contents of a special purpose code generator in builder/*.py (since editing the code generator is a big part of doing the project). The build system knows how to run the code generator to transform the inputs to the output by traversing this graph.
  +-pysdl2_cffi.egg-info
  | +-pysdl2_cffi.egg-info/PKG-INFO
  | | +-pyproject.toml
  | +-pysdl2_cffi.egg-info/entry_points.txt
  | | +-pyproject.toml
  | +-pysdl2_cffi.egg-info/requires.txt
  |   +-pyproject.toml

The .egg-info directory is built by generating PKG-INFO, entry_points.txt and requires.txt, which depend on the contents of pyproject.toml.

Yup, the cargo build script is the sort of thing I'm thinking of.

One option to consider would be to re-use PEP 517 and delegate the entire wheel building step (and get_requires_for_build_wheel), while leaving flit in control of the sdist generation and other aspects of the project's local development experience.
This would align with Daniel's suggestion of "flit could provide useful command line tools like 'publish' and the build step could do almost everything else including writing the wheel."

One option for the 'build script' model is for flit to allow build steps which don't limit platform compatibility, like minifying Javascript, or generating Python UI code. But you would still need to use another tool if you have build steps like compiling native code, where the result is less portable than the source.
I'm still not sure whether this is worth the added complexity, though.

There a plenty of build systems out there. Why not to make flit a plugin for those tools? For example, https://github.com/SCons
The reason for that is to increase transparency into what flit does. Integrating with other build systems will show which things are reinvented, which things are really hacks, and which best practices can be reused.

I'm warming slightly to the idea of a cargo-style build script: Flit would make the sdist, unpack to a temporary directory, and then run the build script there before packing up a wheel. In theory, this isn't much extra complexity, but... what does theory know?
Other complexities that have occurred to me:

Caching: should flit attempt to provide any support for caching, so that the build step doesn't have to do 100% of the work every single time? Or is it entirely up to whatever tools you're invoking to figure out caching if they need to?
Build steps that want to deal with the VCS: this won't work from the unpacked tempdir. I don't think packages should need VCS info, but there may be a desire to do things like embedding a commit ID in the package to precisely identify the version.

For the latter concern, you may want to define a set of environment variables that flit makes available to the build script, and have one of them be FLIT_VCS_DIR.
That could also work if you receive requests for help with managing artifact caching - pass in a FLIT_CACHE_DIR setting. However, it's probably simpler to just let tools manage their own caches (e.g. someone wanting to speed up C/C++ builds is going to get more benefit out of setting up ccache than they are from learning a flit-specific artifact caching regime)

One word of caution about copying things to a temporary directory for building... it takes several minutes for projects with large git histories, lots of node_modules, etc. Pip suffers from this: https://github.com/pypa/pip/issues/2195

I'd copy only the current VCS checkout - no history, no files that aren't version controlled - so the speed hopefully wouldn't be an issue. Of course, relying on and working with VCSs brings its own set of problems, but I've already chosen that path for building sdists.
It would need a fallback path for when there's no VCS info available, though; either because it's not a VCS checkout, or because the VCS software is not found. In that case, it would probably have to copy all the files and hope.

It's probably a tradeoff. Thinking of an example webapp, either you copy node_modules or count on the build script to run npm install or whatever to recreate them. Both are slow, but a build script gives the developer more control to optimize the process.

Yup, precisely. And a build script is also more predictable, because we can be sure that it was run on these files, not the files as they were two weeks ago.
Build tools can also cache data outside the package directory. I don't think npm does, but if a build script used pip install --target to bundle other packages, it would benefit from pip's caches.

Because of read or because of write?
If flit could detect if tmp is memory based - can this speed things up? https://superuser.com/questions/45342/when-should-i-use-dev-shm-and-when-should-i-use-tmp

Cargo copies to build?
On Mon, May 7, 2018, at 6:33 PM, anatoly techtonik wrote:

Because of read or because of write?

If flit could detect if tmp is memory based - can this speed things
up?
https://superuser.com/questions/45342/when-should-i-use-dev-shm-and-when-should-i-use-tmp> — You are receiving this because you were mentioned. Reply to this
email directly, view it on GitHub[1], or mute the thread[2].>

Links:

https://github.com/takluyver/flit/issues/119#issuecomment-387227223
https://github.com/notifications/unsubscribe-auth/AAMskl--GbU_cicKvwgXe8v43Co8pyMnks5twMuwgaJpZM4OUCec

It might be worth exploring 2 builds -- one before sdist, one before wheel.
I was thinking about Cython modules. Usually, one would want to compile the Cython code to C code before creating an sdist. And then before creating a wheel, compile the C files to libraries.

I have very complicated compiling workflow for one of my packages: first step is code generation, second is launching a virtual machine with different OS to actually compile binaries, then I use setuptools+wheel to pack wheels. I build binaries manually and "attach" them to a package using package_data.
I can't ship source dists, they would be absolutely useless, only wheels, it would be helpful to disable sdists in flit config entirely. In my case it is far better to prevent people installing fallback source dists and complaining why the package doesn't work for them.
Another thing to note is managing tags for several wheels. Currenly I have to edit setup.cfg every time I build wheels:
[bdist_wheel]
python-tag = cp36  ; cp35 cp37 etc
plat-name = win32  ; win64 etc

What I expect from flit is just making set of wheels (for different pythons and platforms) and uploading them to pypi, given set of binaries I want to "attach". It would be helpful if flit could make some validation, e.g. remind me if I didn't (re)build something to prevent uploading incorrect wheels (python hook for flit?).
Variant A
[tool.flit.binaries]
disable-sdist = true
python-tag = ["cp35", "cp36", "cp37"]
plat-name = ["win32", "win64"]
exclude = [
  { python-tag = "cp35", plat-name = "win64" },
]
paths = []
check-binary-hook = "project:check_binary"

def check_binary(binary_path):
    s = binary_path.stat()
    if not s.st_size:
        return False  # have to rebuild
    if time.time() - s.st_mtime > 300:
        return False
    return True

Variant B
[tool.flit.binaries]
disable-sdist = true
wheel-list = "project:wheel_list"
file-mapper = "project:file_mapper"
file-checker = "project:file_checker"

def wheel_list():
    for python_tag in ('cp35', 'cp36', 'cp37'):
        for plat_name in ('win32', 'win64'):
            if python_tag == 'cp35' and plat_name == 'win64':
                continue
            yield python_tag, plat_name

def file_mapper(python_tag, plat_name):
    folder = Path('{}_{}'.format(python_tag, plat_name))
    yield folder / 'module.pyd'
    yield folder / 'submodule.pyd'

def file_checker(binary_path):
    s = binary_path.stat()
    if not s.st_size:
        return False  # have to rebuild
    if time.time() - s.st_mtime > 300:
        return False
    return True

The goal of PEP517/PEP518 is making things more automatized for end user/developer, making less assumptions of build time requirements. But these PEPs work entirely at python side, they cannot pull compilers, visual studio, virtual machines with different OSes, external binary packages and third party applications, e.g. blender, firefox, even build systems like make/scons/cmake. It is unavoidable for developers to install all these packages manually following some guide in project's readme.
Before wheels pip could compile some packages for you from source dists. It requires installed compiler, python headers, may be cython, etc. But pip fails if your package depends on something external, like blender headers: you have to install prerequisites first, then pip install again. I think you can't automatize all possible building workflows and it is unavoidable to add ability of including raw binary files to wheels.

I have very complicated compiling workflow for one of my packages...

Flit is probably not going to be the right tool for you, then. Even if we add build steps, it's focused on the simple use cases, not the complicated ones. You might want to look at a system like encsons instead.

But these PEPs work entirely at python side, they cannot pull compilers, visual studio, virtual machines with different OSes, external binary packages and third party applications, e.g. blender, firefox, even build systems like make/scons/cmake.

The PEPs don't specify a mechanism for that, but there's no reason that a build backend complying with these PEPs couldn't download and run a VM or a container to do the build. In fact, I'd be surprised if no-one makes a docker build backend for PEP 517.

Probably I wrote an overloaded post, but for me (regarding changes only in flit) it's quite enough to have:

Ability to disable sdists entirely
Ability to inject arbitrary files into wheels (package_data-alike)
Ability to produce several wheels for different architectural tags (flit build and I have multiple files in dist folder)

I have no plans to distribute buildable sdists with scripts, makefiles and docker.

@neumond you might actually be part of the small target "market" for enscons. It provides a few tools for building a wheel, which can contain anything, as a result of arbitrarily complex build steps. You don't have to include the sdist target if you don't want one. There is also the pre-enscons approach taken by https://bitbucket.org/dholth/sdl2_lib/src/default/, a library that only packages sdl2 dlls in a wheel and cannot have a reasonable sdist, which just implements a standalone wheel in a 130 line waf script.

@dholth Nice idea of building on top of scons, but for me scons is kinda counter-intuitive, I spend too much time searching APIs for every single function I want to use. Have you considered building on top of Waf?

@neumond In addition to being useful in its own right for folks that like Scons, enscons is considered an illustration of the principle of adapting an existing full-featured build system to the Python ecosystem's build system interface expectations, rather than writing a fresh build system from scratch. The entire purpose of the PEP 517/518 build system abstraction layer is to make that easier to do.
However, this discussion has aspect of the discussion has now shifted to be entirely off-topic for flit's issue tracker - this issue is about asking whether or not there's a useful middle ground between flit supporting full PEP 517 style build delegation to arbitrary build backends, and flit not supporting build steps at all.

@ncoghlan Right now I feel the need to have abstract wheel assembling library, not tied with flit and enscons. It looks like wheel package should do this, but I couldn't find clear and easy to use API there, which would clarify wheel contents & metadata, enforce constraints, warn on metadata contradictions (for humans™ ☺). Nevertheless I find wheel's source code very helpful at filling gaps of PEP425 and 427.

What I wonder about is whether we can (and should) make things easier for people who are going from a pure Python package with no build step to a package that maybe bundles some minified Javascript, or compiles a Cython file. It seems rough to make them throw away their flit packaging and start again with another tool, as we currently do.

I started a thread about using Meson as build-system https://discuss.python.org/t/should-python-packaging-aim-for-meson-as-build-system-in-case-of-extension-modules/2579 which would function just like enscons.
See e.g. https://github.com/FRidh/mesonpep517examples/blob/master/pyproject.toml and the meson.build file in the root and subfolder. The pyproject.toml looks a lot like flit already and I think the steps that are needed to use meson here are small. We just need more and better examples probably.

I've now got a project that fits the use case of "minifying the JS/CSS". Given that I now do have experience with the use case, here's what I can say we'd need for that usecase.

Specify a build script. I want to run npm install then gulp build.
Need to include the minified files (which are NOT tracked in version control)
Should exclude the non-minified files (which are tracked in version control)

I'm imagining a new tool.flit.wheel table, that has keys for each of these (following tool.flit.sdist):
[tool.flit.wheel]
build = "build.py"
include = ["src/projectname/scripts/bundle.min.js", "src/projectname/styles/bundle.css"]
exclude = ["src/projectname/assets/*"]

With this, I can then have a build.py file like:
from subprocess import run

run(["npm", "install"], check=True)
run(["gulp", "build"], check=True)

I do think there's a few not-yet-decided-on items, so... well, here's my take on them: :P

Re caching: I think I'm personally 100% OK with flit saying "you manage the caching".
Re "Build steps that want to deal with the VCS": maybe we can use git worktree for this? We use it for pip's build pipeline: https://github.com/pypa/pip/blob/8d549f2cf87bff6ee0512638403d6f4e9c55e809/tools/automation/release/init.py#L165-L186
Re build: I imagine we'll need to decide whether we want to allow "drop in a build.py file and flit will run it" behavior around it. I don't have a good feel for this. :)

file ↓ | build key →
specified
not specified

exists
run given
???

not exists
fail
status quo

@takluyver do you think this looks like a reasonable overview of the approach that flit could take for this?

Another option is the build key be a table instead, that has an optional "requires" key whose value is returned by https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel:
build = { script = "build.py", requires = ["pylibsass"] }

If we make the requires key optional, we could also get away with:
build.script = "build.py"

which reads really nicely IMO. :)

@takluyver gentle nudge If you have express any interest in this, I'm happy put up a PR for this. :)

With the coming of PEP 621 - implemented but undocumented in Flit 3.2, hopefully to be documented once people have kicked the tyres a bit - I'm leaning towards saying no to build steps. I originally wrote in the issue description that:

The best argument I see in favour is that you may start packaging something with flit, then add something that requires a build step. With no support for that, you have to throw away your packaging metadata and workflows to switch to another tool.

PEP 621 should make the packaging metadata portable between build tools, so it will be easier to switch from Flit to something like enscons or setuptools, assuming they also get support. It probably won't be entirely seamless - changing build processes never is - but it should avoid having to rewrite the same metadata in a slightly different format. And that metadata can be most of what you give Flit.
I've also come to appreciate that building software is a massive, complicated topic, and I would rather not be responsible for a build system. We could limit the scope to make it an easier problem, e.g. only allowing build steps which don't affect platform compatibility, like minifying JS. But I think there are relatively few use cases like this between the much more common ones where there's either no build step at all, or we want to build native code for a specific target platform. So I don't think there's much value in adding build steps without support for the complex cases.
So I'm planning to close this issue in a week or two, unless someone makes a compelling argument in favour of supporting build steps despite PEP 621. Of course, that won't mean it's written in stone - we can always revisit the question if circumstances change or if there's a brilliant new idea for how to implement it.

Closing for the reasons described in my previous comment. Maybe we'll revisit this some day in different circumstances, but for now, I think the drawbacks outweigh the advantages, and I don't think it's useful to have questions like this hanging in uncertainty indefinitely.


Add tags to metric sender

Add tags field to UniqueMetric. Send tags to hawkular per key.

This was +1d in STG PR #1640.
@zgalor fyi, I created and approved this PR to get your changes into our production branch.


Added compilability for target_arch=wasm64

Thanks! While you're at it could this be switched to use target_family = "wasm" instead?

Sure no worries - rebased.


bug "This key pair is invalid! Please select keys for different parties."

When entering the Public and Private key, it does not allow you to take the next step and writes "This key pair is invalid! Please select keys for different parties."

That's right, after generating keys on the site and saving them to disk, during encryption after selecting these keys, it turns out that these keys do not match.

How to make it work correctly?
Вс, 14 янв. 2024 г. в 20:58, x71coment @.***>:

That's right, after generating keys on the site and saving them to disk,
during encryption after selecting these keys, it turns out that these keys
do not match.
—
Reply to this email directly, view it on GitHub
https://github.com/sh-dv/hat.sh/issues/97#issuecomment-1891018863, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AYEJTBVT6BZIQXUBTCVGDZDYOQMFJAVCNFSM6AAAAABBTBLQ76VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJRGAYTQOBWGM
.
You are receiving this because you authored the thread.Message ID:
@.***>


Gatsby v2 complete ground-up rewrite

Deploy preview for blog-thibmaekelbergh failed.
Built with commit 3b0442f7479a7f35a02f5047cafdce2225886d36
https://app.netlify.com/sites/blog-thibmaekelbergh/deploys/5d97c5ca9a8b6700089cad9e


Not Updating Graph

I have the fetcher continuously updating in the background, and running graph.py gets me great csv's, but when I run app.py the web app does not use the updated csv's.  I tried deleting the generated_graphs folder and then re-generating it with graph.py, but that didn't help.  How do I make the web app use the latest data?

You're browser is caching the information it looks like; I had the same issue.
If you run the app in an incognito browser, you'll be fine.


Update/openmm 7.7

This merge includes:

Update to OpenMM version 7.7.
No longer using parmed to read simulation input files.


Unhandled Exception - failed to launch browser

Hey Thomas,
I am working on a project and hit a small snag, was wondering if you could lend a hand.
I have a program, that fetches JSON data, feeds the JSON response into a queueing mechanism, and then queueing mechanism opens a puppeteer instance (code below for the puppeteer instance) which does a variable find/ replace and takes a screenshot.
navigationSource = 'url';

FunctionName: function(Data, var2, var3) {
    return new Promise(function(resolve, reject){
        (async () => {
          const cluster = await Cluster.launch({
            concurrency: Cluster.CONCURRENCY_BROWSER,
            puppeteerOptions: { args: ['--no-sandbox'] },
            maxConcurrency: 2,
            workerCreationDelay: 500,
            retryLimit: 1,
            timeout: 5000,
          });

          await cluster.task(async ({ page }) => {
            await page.goto(navigationSource);

            await page.evaluate(x => {
                data = x;
                replaceRCValues(data['content'], rcScoreValues);
            }, Data);

            await page.emulateMedia('screen');
            await page.pdf({ path: var2 + var3,
                      format: 'letter',
                      printBackground: true,
                            margin: {
                          top: ".2in",
                          bottom: ".1in",
                          left: ".75in",
                          right: ".75in"
                      },
                      });
          });

          // In case of problems, log them
          cluster.on('taskerror', (err, data) => {
            console.log(`DEBUG - Error with cluster ${data}: ${err.message}`);
          });

          cluster.queue();

          await cluster.idle();
          await cluster.close();

          resolve({success:true})
        })()
        .catch(err => {
          console.log(`DEBUG - CAUGHT ERROR FROM MainFunction -- ${err}`)
          reject({success:false})
        })

      });
  },

Issue
I was first running into issues directly with puppeteer, in that the browser instance was failing to launch due to resource issues, and I was getting a timeout error 30000ms exceeded
I then looked into your program, to handle said error, and my success rate skyrocketed with the way you cluster the puppeteer instances, thank you for that.
I still however, am getting the following error
(node:64000) UnhandledPromiseRejectionWarning: Error: Unable to launch browser for worker, error message: waiting for target failed: timeout 30000ms exceeded
    at Cluster.<anonymous> (/Users/crosa/Cylance_Repo/CylanceReportsTwoOhNew/reportProcessor/node_modules/puppeteer-cluster/dist/Cluster.js:129:23)
    at Generator.throw (<anonymous>)
    at rejected (/Users/crosa/Cylance_Repo/CylanceReportsTwoOhNew/reportProcessor/node_modules/puppeteer-cluster/dist/Cluster.js:5:65)
(node:64000) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:64000) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I've tried to put various catch blocks in my code, and I also changed cluster.on('taskerror', (err, data)) to cluster.on('UnhandledPromiseRejectionWarning', (err, data)) in hopes of handling those errors, but that was un-successful too.
Solution I am looking for
Ultimately, I'd like to
(1) Attempt to ingest JSON data and push to puppeteer for variable find/replace and screenshot generation
(2) If the browser fails, retry once.  If it fails the second time, catch the error and reject it, so that I can update the report attempt as failed
Any thoughts on how I can handle that rejection?  So that I can pivot off of its failure
Bug Number 2
I would like the browser to timeout after 5000ms rather then 30000ms, as you can see by my above code, there is a timeout set for 5000ms, but regardless I get the 30000ms timeout for the browser.  Any thoughts there?  I tried to add puppeteerOptions: { args: ['--no-sandbox'], timeout:5000 }, as well and that did not work.  Still timesout at 30000ms

The timeout in options is for the full task, you can still set custom timeouts to page.goto (and other puppeteer functions) as you want. Regarding the waiting for target failed: timeout 30000ms exceeded. When exactly does that happen? Can you provide a minimal example, please?
Tagging this as question, as this does not really look like a bug to me. Feel free to re-open or re-tag to continue the discussion.