fix contrast limits

closes #262
Fixes the issue in #262 regarding calculating contast limits for MultiScale data. This PR checks whether data is multiscale and if so calculates the contrast limits based on the lowest resolution. Furthermore, an additional check is added to see whether contrast limits are monotonically increasing and if this is not the case it sets the contrast limits to the dtype range.

LGTM!


Conversion rules for polars.

@Wainberg - are you available to review the PR?

Thanks for taking a crack at this! I think you'll need some additional logic to handle the conversion between pl.Categorical/pl.Enum and R factors. I seem to remember this was also an issue for the pandas to R conversion.
For what it's worth, here are the functions I've written for my lab members to convert between Python and R data frames.

I don't support Enums yet because pl.col(pl.Enum) doesn't work yet and I haven't bothered coming up with a workaround because they're fixing it.
I run signal.signal(signal.SIGINT, signal.default_int_handler) after from rpy2.robjects import r because the import messes up the Ctrl + C behavior - it triggers a full traceback. Would be great if you could fix this :)

>>> # before, pressing Ctrl + C
KeyboardInterrupt
>>> from rpy2.robjects import r
>>> # after, pressing Ctrl + C
Traceback (most recent call last):
  File "/home/wainberg/miniforge3/lib/python3.12/site-packages/rpy2/rinterface.py", line 94, in _sigint_handler
    raise KeyboardInterrupt()
KeyboardInterrupt

Because polars DataFrames don't have row names but R data.frames do, I give the user the option to specify the row names (when converting polars -> R) or whether to add the row names as the first column of the DataFrame (when converting R -> polars). Converting string columns is like 100-1000x more expensive (I forget the exact amount) than converting numeric columns, so converting the rownames is off by default.
If some things look unintuitive or roundabout, it's probably because I benchmarked them and they're more efficient than the more straightforward way.

As a general point, I find it cleaner, more intuitive and less error-prone to import a function that does the conversion (which can then be included in a method chain using df.pipe()) rather than using a with statement and auto-converting. This has always bugged me about rpy2.
Anyway, here's the code:
def df_to_rdf(df, *, rownames=None):
    """
    Converts a polars DataFrame to an R data frame via rpy2.

    Args:
        df: a polars DataFrame
        rownames: an optional polars Series (or tuple, list, etc.) of rownames
                  for the R data frame; column names will be copied from df

    Returns:
        The corresponding R data frame.
    """
    import rpy2_arrow.arrow as pyra
    from rpy2.robjects import r
    signal.signal(signal.SIGINT, signal.default_int_handler)
    if not isinstance(df, pl.DataFrame):
        raise TypeError(f'input to df_to_rdf should be a polars DataFrame, '
                        f'but you gave it a {type(df)}')
    # Handle categoricals by separating off the categories, converting the
    # codes, and adding the categories back at the end
    # TODO also handle pl.Enum
    categories = df\
        .select(pl.col(pl.Categorical).cat.get_categories().implode())
    df = df.with_columns(pl.col(pl.Categorical).cast(pl.Int32))
    # Convert
    rdf = r['as.data.frame'](pyra.pyarrow_table_to_r_table(df.to_arrow()))
    # Add the categories back
    for col_categories in categories:
        col_index = rdf.colnames.index(col_categories.name)
        rdf[col_index].rclass = 'factor'
        rdf[col_index].slots['levels'] = series_to_rvector(col_categories[0])
    if rownames is not None:
        rdf.rownames = series_to_rvector(pl.Series(rownames))
    return rdf

def rdf_to_df(rdf, *, keep_rownames=False):
    """
    Converts an R data frame to a polars DataFrame via rpy2.

    Args:
        rdf: an R data frame
        keep_rownames: if True, adds the R data frame's names as the first
                       column of the output polars DataFrame (called rownames,
                       or rownames_ if there's already a rownames column)

    Returns:
        The corresponding polars DataFrame.
    """
    import rpy2_arrow.arrow as pyra
    from rpy2.robjects import DataFrame, r
    signal.signal(signal.SIGINT, signal.default_int_handler)
    if not isinstance(rdf, DataFrame):
        raise TypeError(f'input to rdf_to_df should be an R data frame, '
                        f'but you gave it a {type(rdf)}')
    # Remove any classes rvector has (except factor) to avoid conversion errors
    df = {}
    for col_name, rvector in rdf.items():
        original_rclass = rvector.rclass
        try:
            rvector.rclass = 'factor' if 'factor' in rvector.rclass else ()
            df[col_name] = \
                pyra.rarrow_to_py_array(r('arrow::Array$create')(rvector))
        finally:
            # Put the original classes back
            rvector.rclass = original_rclass
    df = pl.DataFrame(df)
    if keep_rownames and rdf.rownames:
        rownames_column_name = 'rownames'
        while rownames_column_name in df:
            rownames_column_name += '_'
        df.insert_column(0, rvector_to_series(rdf.rownames)
                         .rename(rownames_column_name))
    return df

Again, I'd like to reiterate that it would be fabulous if you could incorporate rpy2-arrow directly into rpy2, because then the polars folks would be on board with doing polars_df.to_r() and pl.from_r(r_df). I wish they were less particular about whether rpy2 and rpy2-arrow are two separate libraries versus one, but they seem to care a lot.

Thanks for taking a crack at this! I think you'll need some additional logic to handle the conversion between pl.Categorical/pl.Enum and R factors. I seem to remember this was also an issue for the pandas to R conversion.

Thanks. I'll look at what you have and what might be missing in the PR (as well as performance).
pandas's Categorical should be handled.
https://github.com/rpy2/rpy2/blob/e0f2155e4857c61b1bf9bed4aecc7650e7ffb6d3/rpy2/robjects/pandas2ri.py#L89
https://github.com/rpy2/rpy2/blob/e0f2155e4857c61b1bf9bed4aecc7650e7ffb6d3/rpy2/robjects/pandas2ri.py#L343
Please open an issue if not the case.

For what it's worth, here are the functions I've written for my lab members to convert between Python and R data frames.
* I don't support Enums yet because `pl.col(pl.Enum)` doesn't work yet and I haven't bothered coming up with a workaround because they're fixing it.

pandas can have features that don't completely work yet. I am also staying away from them as much as possible.

* I run `signal.signal(signal.SIGINT, signal.default_int_handler)` after `from rpy2.robjects import r` because the import messes up the Ctrl + C behavior - it triggers a full traceback. Would be great if you could fix this :)

Thanks.
IIRC R will modify SIGINT handling during initialization. rpy2 is trying to revert that right after R is initialized (see here) with
def _sigint_handler(sig, frame):
    raise KeyboardInterrupt()

which I had understood to be Python's typical behaviour for KeyboardInterrupt. If there is default interruption handler in Python this would work as well.

>>> # before, pressing Ctrl + C
KeyboardInterrupt
>>> from rpy2.robjects import r
>>> # after, pressing Ctrl + C
Traceback (most recent call last):
  File "/home/wainberg/miniforge3/lib/python3.12/site-packages/rpy2/rinterface.py", line 94, in _sigint_handler
    raise KeyboardInterrupt()
KeyboardInterrupt

* Because polars DataFrames don't have row names but R data.frames do, I give the user the option to specify the row names (when converting polars -> R) or whether to add the row names as the first column of the DataFrame (when converting R -> polars). Converting string columns is like 100-1000x more expensive (I forget the exact amount) than converting numeric columns, so converting the rownames is off by default.

String management in Python and R is different, and IIRC both use some form of caching / pointer strategy to minimize object creation and memory usage when identical strings are present.

* If some things look unintuitive or roundabout, it's probably because I benchmarked them and they're more efficient than the more straightforward way.

More static, less modular code can bring efficiencies. In the case of rpy2, there is also the notion of convenience vs performance. The robjects layer bring a few convenience features that abstract conversion or  make wrapper for R objects more pythonic but that can come at the cost of performance. The rinterface layer can then be used. I prefer to have specifics about the performance issue motivating odd-looking code (say, as code comments).

As a general point, I find it cleaner, more intuitive and less error-prone to import a function that does the conversion (which can then be included in a method chain using df.pipe()) rather than using a with statement and auto-converting. This has always bugged me about rpy2.

Several things here. Conversion rule sets spare users a lot of tedious explicit calling of conversions, or all re-implementing dispatch or casting logic. You are actually leveraging the default conversion rule set in your code. Even the rinterface-level to interface with R is applying some for conversion. Thinking about conversions as a rule set with a built-in dispatch mechanism allows things like "Python list of Python polar DataFrames -> R" to "just work" by adding "Python polar DataFrame -> R" to an existing default rule set handling "Python list -> R".
The use of a context manager (and the with statement) comes from the fact that conversion rule sets have to be implemented as "global" but should facilitate their handling in a context-dependent fashion.
I hear you that having to wrap calls into with block can be annoying in some contexts. However, creating a function like you find more intuitive is rather trivial.
def df_to_rdf(df):
    with polars2ri.converter.context() as conversion_ctx:
        return converstion_ctx.py2rpy(df)

That function definition could be in the module with the conversion rules set, this way all use-cases are covered.

Anyway, here's the code:
def df_to_rdf(df, *, rownames=None):
    """
    Converts a polars DataFrame to an R data frame via rpy2.

    Args:
        df: a polars DataFrame
        rownames: an optional polars Series (or tuple, list, etc.) of rownames
                  for the R data frame; column names will be copied from df

    Returns:
        The corresponding R data frame.
    """
    import rpy2_arrow.arrow as pyra
    from rpy2.robjects import r
    signal.signal(signal.SIGINT, signal.default_int_handler)
    if not isinstance(df, pl.DataFrame):
        raise TypeError(f'input to df_to_rdf should be a polars DataFrame, '
                        f'but you gave it a {type(df)}')
    # Handle categoricals by separating off the categories, converting the
    # codes, and adding the categories back at the end
    # TODO also handle pl.Enum
    categories = df\
        .select(pl.col(pl.Categorical).cat.get_categories().implode())
    df = df.with_columns(pl.col(pl.Categorical).cast(pl.Int32))
    # Convert
    rdf = r['as.data.frame'](pyra.pyarrow_table_to_r_table(df.to_arrow()))
    # Add the categories back
    for col_categories in categories:
        col_index = rdf.colnames.index(col_categories.name)
        rdf[col_index].rclass = 'factor'
        rdf[col_index].slots['levels'] = series_to_rvector(col_categories[0])
    if rownames is not None:
        rdf.rownames = series_to_rvector(pl.Series(rownames))
    return rdf

def rdf_to_df(rdf, *, keep_rownames=False):
    """
    Converts an R data frame to a polars DataFrame via rpy2.

    Args:
        rdf: an R data frame
        keep_rownames: if True, adds the R data frame's names as the first
                       column of the output polars DataFrame (called rownames,
                       or rownames_ if there's already a rownames column)

    Returns:
        The corresponding polars DataFrame.
    """
    import rpy2_arrow.arrow as pyra
    from rpy2.robjects import DataFrame, r
    signal.signal(signal.SIGINT, signal.default_int_handler)
    if not isinstance(rdf, DataFrame):
        raise TypeError(f'input to rdf_to_df should be an R data frame, '
                        f'but you gave it a {type(rdf)}')
    # Remove any classes rvector has (except factor) to avoid conversion errors
    df = {}
    for col_name, rvector in rdf.items():
        original_rclass = rvector.rclass
        try:
            rvector.rclass = 'factor' if 'factor' in rvector.rclass else ()
            df[col_name] = \
                pyra.rarrow_to_py_array(r('arrow::Array$create')(rvector))
        finally:
            # Put the original classes back
            rvector.rclass = original_rclass
    df = pl.DataFrame(df)
    if keep_rownames and rdf.rownames:
        rownames_column_name = 'rownames'
        while rownames_column_name in df:
            rownames_column_name += '_'
        df.insert_column(0, rvector_to_series(rdf.rownames)
                         .rename(rownames_column_name))
    return df

Again, I'd like to reiterate that it would be fabulous if you could incorporate rpy2-arrow directly into rpy2, because then the polars folks would be on board with doing polars_df.to_r() and pl.from_r(r_df).

My reticence is around maintenance of rpy2 when a proliferation of optional dependencies when the resources to maintain rpy2 are very limited and the dependencies cross-languages are not handled by package management for Python. In the case of conversion to polars it relies on an R package that is not even available in the standard package repository for R. I am fine with including into rpy2_arrow, even though polars will be an optional dependency. At least there is a containment of optional features into thematic packages.
I guess that we are in a situation where there is no perfect solution, but a choice between trade-offs. For example, the pandas model seems to include features that might be optional, or not yet fully functioning, while the jupyter model is to have extension modules. The former can be argued to make it more likely that users will try it, but this can at the cost of entropy. The latter can help independently-maintained extensions to develop at a faster pace.

I wish they were less particular about whether rpy2 and rpy2-arrow are two separate libraries versus one, but they seem to care a lot.

I also unsure about the rationale of this when an optional dependency in rpy2 would not mean be more used, or necessarily better maintained.
However, I don't understand either what is the importance if having those to_r() and from_r() part of polars when something like rpy2_arrow.to_pypolars() or rpy2_arrow.from_rpolars(), or any similar functionality in another package than rpy2_arrow would work just as well.

Yeah it's really just about convenience. A surprisingly large number of people aren't even aware that Python-to-R conversion is possible, probably because it's not integrated into pandas/NumPy.
Would be great if you can fix the sigint thing since rpy2's current behavior is quite annoying.

Yeah it's really just about convenience. A surprisingly large number of people aren't even aware that Python-to-R conversion is possible, probably because it's not integrated into pandas/NumPy.

Speaking of awareness, IIRC you mentioned that you did not know about the ggplot2 and dplyr wrapper in rpy2. IIUC R polars can be used in combination with dplyr in R. This could be used to used the dplyr API from Python while the data is in polars: https://rpy2.github.io/doc/v3.5.x/html/lib_dplyr.html

Would be great if you can fix the sigint thing since rpy2's current behavior is quite annoying.

I am looking into it. This is tracked here: https://github.com/rpy2/rpy2/pull/1085

The  issue with a test seems to be cause by a inconsistency or issue with the R package arrow. I reported it (https://github.com/apache/arrow/issues/39603). I will have to find a workaround for that test.

I hit another issue. This time it seems to be with the R package polars (reported here: https://github.com/pola-rs/r-polars/issues/725).

@Wainberg The equivalent to rdf_to_df() and df_to_rdf() is called rpl_to_pl() and pl_to_rpl() with the intent to make them handle as many polars object types as possible.
This is now merged and will be in the release 0.1.0 of the package.

Awesome!
I tried out pl_to_rpl with a DataFrame (doesn't seem to support polars Series) but got this error:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "rpy2_arrow/polars.py", line 108, in pl_to_rpl
    with polars.converter.context() as conversion_ctx:
         ^^^^^^^^^^^^^^^^
AttributeError: module 'polars' has no attribute 'converter'. Did you mean: 'convert'?

I think you meant with converter.context() instead of with polars.converter.context() here:
def pl_to_rpl(df):
    """Convenience shortcut to convert a polars object to a R polars object."""
    with polars.converter.context() as conversion_ctx:
        return conversion_ctx.py2rpy(df)

def rpl_to_pl(df):
    """Convenience shortcut to convert a R polars object to a polars object."""
    with polars.converter.context() as conversion_ctx:
        return conversion_ctx.rpy2py(df)

Thatnks. It looks like I thought that I had tests but but tests are not using the functions they were meant to test.
https://github.com/rpy2/rpy2-arrow/blob/main/rpy2_arrow/tests_polars.py#L154
:facepalm:

Thatnks. It looks like I thought that I had tests but but tests are not using the functions they were meant to test. https://github.com/rpy2/rpy2-arrow/blob/main/rpy2_arrow/tests_polars.py#L154
🤦

Fix in progress with PR #15

Should be fixed in 0.1.1 (now on pypi).


Fixed the db query alias prefix to distinguish between metrics and di…

…mensions to avoid collisions when an aliases is shared.

Coverage increased (+0.06%) to 93.59% when pulling ef084f886933ff80974d94f6adbdc90cc8530f3e on fix_metric_dimension_alias_collision into 19527f737beae340d4eba201e20fc06d5ae31914 on fireant1.0.


How to use the custom_bucket_policy?

Can you add some examples for the custom_bucket_policy ?

The v1.7.3 release fixes this issue and allows a custom policy to be added to the bucket.


extract: Fix missing '--git-revision' in the usage banner

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.
[ ] Have you successfully run brew style with your changes locally?
[ ] Have you successfully run brew typecheck with your changes locally?
[ ] Have you successfully run brew tests with your changes locally?

Thanks @Kentzo!


Images are pixelated in kitty terminal

This is likely something in my local configuration, but hoping someone here might know what that could be.
I can display images in kitty fine, and they look like I would expect:

But when I try to display images in a presentation, they are rendered horribly pixelated:

Am I missing some configuration or setting needed to make this rendering work well?

Kitty version: 0.35.1
Presenterm version: 0.7.0

OK, turned out I needed to explicitly add --image-protocol kitty-local when launching. Is this expected? I assumed it would automatically detect that, but maybe something else is still missing?

Are you using tmux? It seems like the kitty detection is failing, which is a known issue under tmux. Only in that scenario you currently need to explicitly use --image-protocol, so you shouldn't need to set it unless you're using it.
You can make it the default in the config file so you don't have to pass it in every time if you are actually using tmux.

I was not using tmux, and I opened a fresh instance of kitty just to make sure there wasn't some sort of leftover settings in my environment.
I did get around it by explicitly setting kitty-local in the config file, so things seem to be working well for me now. But if there are anything that would be useful to debug from this environment, just let me know. I'll close this issue for now.
Thanks for such a great tool!


Reshape SKGLView when Window.BackingScaleFactor changes (#1853)

Description of Change
Reshape SKGLView when Window.BackingScaleFactor changes, ensuring that the output is displayed at the proper scale. If it is possible to create a test for this type of visual change, I don't know how.
Bugs Fixed
[BUG] macOS: When display scale changes SKGLView does not adjust appropriately
https://github.com/mono/SkiaSharp/issues/1853
API Changes
None
Behavioral Changes
None
PR Checklist

[ ] Has tests (if omitted, state reason in description)
[x] Rebased on top of main at time of PR
[x] Changes adhere to coding standard
[ ] Updated documentation

Is this the only way to check? Or rather, is this sufficient?
I am thinking if you have moved the window, does it always trigger a paint - even without a render loop? Is there some window based event that we should check? What happens if you change the display scale in the system preferences?
The code looks good, I am just wondering if we should be listening to some event...

Yes, moving the window between two displays of different scales triggers a paint even without a render loop. A paint is also triggered when then the display scale is changed via System Preferences.
There is an event that is fired when display scale is changed via System Preferences (NSApplication.Notifications.ObserveDidChangeScreenParameters) but it does not fire when moving the window between displays. If there is another event that fires when moving the window, I am not aware of it.

I moved the code as suggested. Great change. The result is visually perfect now, eliminated a brief flash that happened before.

I may have just discovered a better way to do this... don't merge yet.

Nevermind... There is an event that fires - NSView.DidChangeBackingProperties and the fix can be implemented there however doing so results in the content being redrawn at the wrong scale momentarily. Handling it in DrawRect is much better.

I believe this code is ready to merge.

OMG, 7 months. Sorry! Let me re review and merge. Thanks for fixing this.

Actually, I am looking at this and see the logic in draw rect basically cancels and triggers a redraw.
What happens with the reshape event?
Is the chance basically to not have a flash?

If we don't track whether the scale has been changed and call Reshape directly, it is never called unless the view is resized.
When tracking the scale from inside of OnPaintSurface where I had it originally, there was a flash. Moving it to DrawRect and bailing out when a changed is detected prevents a flash entirely.
FWIW, we have already been using this in production with thousands of users for the last 6 months.


support psc-package

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

This is for my forked repository change, so I close it.


Ability to export more than 2000 records

Ability to export more than 2000 records for csv/shapefile

@emjohnst - Emails are sent out by the system to notify users that the process has completed. What email address would you like it to come from? Does<EMAIL_ADDRESS>work, or would you like to use another one?

Currently working in DLVR for Heritage Sites. Other resource types are causing an issue due to an Arches Core bug causing the recrord sorting to fail. I also need to do a bit of work to silo the tasks between BCRHP & BCFMS, and DLVR & TEST instances.
@emjohnst  - Can you please confirm what email address you'd like automated messages to come from?

@bferguso I got the email - but is the download supposed to be in the email? or where is the link to get the download results?

@bferguso It is possible to set the export so that the user has to define what resource type they want to export? This may help gatekeep people from going to the site and simply downloading the entire database

No, there isn't any way to do that by default, it would require additional customization to provide that functionality.

@bferguso I see where the download is now, is there a point to providing an email if the email doesn't have a link to the download? How do the downloads remain in the notification tab for public users that are not logged in?

I believe it uses session ID so if the anonymous user's session times out it would no longer be available. I'd need to confirm this for sure, and dig into how long the sessions last for.

@bferguso

Can you please make the email come from<EMAIL_ADDRESS>There is no link in the email (see image). Can you include link to downloads in email? Or remove option to enter email in Arches.
If using email please update first line of message: "Hello, Your request to download a set of search results from the BC Register of Historic Places is now ready."
Internal users get an error message. This is likely because heritage site submission and sandcastle don't have define csv exports - can we make them have csv exports just simply on all fields?

@emjohnst  - I'd like to spin off #4 as a separate ticket as that's unrelated to this and should be tracked independently.

@bferguso sounds good, let's remove the email

Emails disabled. They can be re-enabled by internal users by editing their profile. We should be able to provide a link to the download if desired, but lets do that after we upgrade to 7.6.x.


Demo link on github doc broken

On the documentation page at https://github.com/opencast/annotation-tool/blob/master/documentation/Overview.md you can find a demo link to https://entwinemedia.github.com/annotations/ what is broken / 404.
Would be great to have a working demonstration for the OC annotation tool to show (and perhaps test) the tool, simmilar to develop.opencast.org/

@cgruber-uos If you would like to test the annotation tool we are running at the University of Bern, please contact me (david.graf@unibe.ch). I could provide you with a test account. Feature wise we are similar to the version in the master branch.


Issue: Trying to debug an embedded web application at subpath

Hi there, everyone! I hope it's a correct place to file this issue, if not, it's not a problem for me to file it at a correct location.
The issue I have is from a very special use-case, involving embedding Flutter Web into an existing application using iframes and conditionally using a reverse proxy that points to flutter run at a non-root path like /flutter/.
When trying to debug, on either master, beta or stable debugging with breakpoints never works correctly. Tried using both VSCode and DevTools. It seems like there is an issue with source file name resolution, but I don't have a way to confirm it, except for DevTools throwing an issue with _stripBasePath(...).
When just using a reverse proxy without a sub-path, breakpoints work on master and beta, but show an error in VSCode on trying to dig into variables and objects.
Steps to Reproduce

Setup an nginx reverse proxy, where /flutter points to a pre-defined port running flutter run
Have an embedding web app at / in the reverse proxy
Try to debug using VSCode

Expected results:
Being able to continue developing the application normally, using breakpoints and hot-restart with the embedded (in the iframe) application behaving correctly.
Actual results:

At first, issue with the application connecting to the debugging system, which requires this workaround (port 42422 is proxy's endpoint, 42420 is flutter run) which goes into index.html

if (location.href.includes('42422')) {
  Object.defineProperty(window, "$dwdsDevHandlerPath", {
    value: 'ws://localhost:42420/$dwdsSseHandler',
    writable: false
  });
}

When trying to debug, breakpoints are marked as unavailable
Have other issues with debug socket disconnecting, restart hanging, breakpoints showing up in wrong locations

Code sample
NGINX configuration

#user  nobody;
worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       42422;
        server_name  localhost;

        location /flutter/ {
            proxy_pass http://localhost:42420/flutter/;
            proxy_pass_header Content-Type;
            proxy_redirect   off;
            proxy_set_header Host            $host;
            proxy_set_header X-Real-IP       $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            # kill cache
            add_header Last-Modified $date_gmt;
            add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
            if_modified_since off;
            expires off;
            etag off;
        }

        location / {
            proxy_pass http://localhost:62768/;
            proxy_pass_header Content-Type;
            proxy_redirect   off;
            proxy_set_header Host            $host;
            proxy_set_header X-Real-IP       $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            # kill cache
            add_header Last-Modified $date_gmt;
            add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
            if_modified_since off;
            expires off;
            etag off;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

flutter run parameters
"args": [
    "--web-port", "42420",
    "--web-launch-url", "http://localhost:42422/mvc/Flutter/Container/step-by-step",
]

Logs

Hello @skyne98. Thank you for filing this issue. Unfortunately, the information you provided is not enough for me to try to reproduce this issue. Could you provide the below information:

A minimal, reproducible example. For an issue this complex to reproduce, a repository that contains all the required files to verify the issue is ideal. If possible, could the sample be void of any third party SDKs or libraries? For example, replacing the nginx proxy with a Dart one.
Detailed reproduction steps. Mainly, what's missing is the "parent" web app. Could you provide, in your code sample, a sample web app with the iframe as well?
Is this issue reproducible in IntelliJ/Android Studio as well?
Could you also provide the entire log of running flutter doctor -v as well as the crash logs, if any?

@exaby73, sure thing, as soon as I have a moment I will make a repo.
Can you give an example of a reverse proxy in Dart I can use?

You can look at how a proxy is implemented in webdev_proxy for example :)


Fix # Issue with toggle button

I've translated it to CoffeeScript and slight rewritten to make toggleButton working.

The only issue here is that checkbox and radio directives now run on the parent container rather than the inputs, which causes problems with transclusion (often a input won't be found when the JS runs)
$($element[0]).find('label > input[type=checkbox]')

Instead could you look at the parent / container elements to detect toggle support?


cmd/compile: rangegen.go test killed on ppc64

#!watchflakes
post <- pkg == "cmd/internal/test" && test == "Test/rangegen.go" && goarch ~ `ppc64`

Test/rangegen.go is occasionally getting killed on ppc64 builders.
e.g., https://ci.chromium.org/ui/p/golang/builders/ci/gotip-linux-ppc64le/b8756201558164082785/test-results?sortby=&groupby=
=== RUN   Test/rangegen.go
=== PAUSE Test/rangegen.go
=== CONT  Test/rangegen.go
    testdir_test.go:142: exit status 1
        command-line-arguments: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64le/compile: signal: killed

--- FAIL: Test/rangegen.go (33.68s)

From the history, this seems to be a recent regression: https://ci.chromium.org/ui/test/golang/cmd%2Finternal%2Ftestdir.Test%2Frangegen.go?q=V%3Ago_branch%3Dmaster+V%3Agoos%3Dlinux+V%3Ahost_goos%3Dlinux+
Note that this test is very large and has caused OOMs before (#64789).
cc @golang/ppc64

I changed /home/swarming on the LUCI builders to a tmpfs mount on 2/13, that seemed to consistently fail during this test as it went over the 8G RAM limits of the container. The LUCI builders are to using quite a bit more space than the buildbot instances.
I bumped it later that day to 10G, and set up a 5G swap file on the ppc64le builder. Hopefully that is enough to keep things running, hopefully faster.

Thanks for the update. For what it's worth, I got this failure on https://go.dev/cl/564137 at around Feb 14, 6pm EST. It sounds like that may be after your latest change? (This could be a bug in that WIP CL, but it seems unlikely to fail just this one test)

Looking at the VM, it did trigger an OOM on a container. The syslog is claiming 7.5G of that is "file" usage.
Is there any way to make LUCI more conservative with it disk usage? This seems like a pretty big jump from the old CI which also ran entirely on a tmpfs.

That's a good question. I wonder how much disk this test used on the old infra. I know this test does generate an absolutely massive source file.
cc @golang/release

I didn't realize the ppc64 LUCI builder also needed a bump too. It is bumped to 10g.
If this doesn't crash again in the next week, this issue can be closed.

Poking around at the idle ppc64le builders, there is a folder "/home/swarming/.cache/gopls" which is consuming between 2.5G and 3.3G on each instance.
That's a problem. Is it possible to update LUCI to cleanup caches at the end of a test run?

I recall @adonovan used a GOPLSCACHE mechanism to help with that in the previous infrastructure (CL 494297). I'm not seeing that in the LUCI infrastructure—perhaps it needs to be ported over. There's some relevant discussion in this thread. CC @mknyszek.

For reference, this is what has accrued on linux-ppc64le-power8--05 since the last container reboot:
832K	.cache/gopls/013c47f3
259M	.cache/gopls/01a3ac08
136M	.cache/gopls/0634c015
88M	.cache/gopls/0d9f1c10
64M	.cache/gopls/1045283d
258M	.cache/gopls/159d94d7
142M	.cache/gopls/17fdaa63
144M	.cache/gopls/18c46665
89M	.cache/gopls/1c4c60e1
64M	.cache/gopls/223a02a4
128K	.cache/gopls/273a5340
6.7M	.cache/gopls/299fee7f
128K	.cache/gopls/29a597a4
26M	.cache/gopls/387f9f03
138M	.cache/gopls/4e2cae36
32M	.cache/gopls/575df253
128K	.cache/gopls/65191266
1.7M	.cache/gopls/694e1ff5
1.4M	.cache/gopls/6fc45050
6.7M	.cache/gopls/70bf6d67
89M	.cache/gopls/78e17b72
61M	.cache/gopls/8a118055
27M	.cache/gopls/8a786b24
6.7M	.cache/gopls/a786fe67
1.4M	.cache/gopls/a806c003
1.4M	.cache/gopls/aa30a907
33M	.cache/gopls/ac020416
832K	.cache/gopls/b0ace309
257M	.cache/gopls/b0bcd7cf
832K	.cache/gopls/b6fa81e7
42M	.cache/gopls/bcc7bedc
27M	.cache/gopls/c8caba22
41M	.cache/gopls/c99017d5
128K	.cache/gopls/cd2810e5
144M	.cache/gopls/d4694109
41M	.cache/gopls/d558ebe0
43M	.cache/gopls/d56f4029
1.9M	.cache/gopls/d67d19d4
33M	.cache/gopls/d93c051c
41M	.cache/gopls/e3926c21
138M	.cache/gopls/eeb12931
41M	.cache/gopls/f224d393

That list sums to around 3GB, which is higher than the cache soft limit, but that's to be expected in a test environment as it is constantly generated fresh garbage. It doesn't seem like a large amount of data compared to the capacity of a typical temp volume though.

I recall @adonovan used a GOPLSCACHE mechanism to help with that in the previous infrastructure (CL 494297). I'm not seeing that in the LUCI infrastructure—perhaps it needs to be ported over.

Yes, that was an effective fix for the problems of this kind we saw in the older builders. It should be as simple as setting GOPLSCACHE to a temp directory for the entire run.

I've set GOPLSCACHE=/home/swarming/.swarming/w via the container image on all ppc64x luci builders while waiting for a more generic fix.

It should be as simple as setting GOPLSCACHE to a temp directory for the entire run.

@adonovan @pmur Can you clarify what you mean by "a temp directory"? Is the TMPDIR set in the builds sufficient (https://logs.chromium.org/logs/golang/buildbucket/cr-buildbucket/8755540096649505841/+/u/step/9/log/1)? That gets cleared after every run.

Is the TMPDIR set in the builds sufficient? That gets cleared after every run.

Yes, a temp directory that lasts for a complete run of tests at a single CL is ideal. Thanks.

Sent crrev.com/c/5314212. I created an explicit subdirectory in the workdir for it which will have the same effect, and I thought it might be a bit clearer to have it next to the GOCACHE directory.

I confirmed that the environment variable is now set in new builds to a directory that will definitely get wiped on each run. Hopefully this should be resolved.


Add pause support

Add Controller2 message support
Add set packet-in-format message support
Add PacketIn2 message support
Add Resume message support

In file nx_action.go:
In NXActionController2PropUserdata, Userdata should be []byte instead of a byte.
Accordingly, the Marshal/Unmarshal function should be changed to handle a slice of byte.
We are not further decoding the properties of Continuation using DecodeContinuationProp. This function is defined but not used. Is this intentional?
Thanks,

Thanks for the review @ashish-varma .

I tried to use ovs-ofctl to add a flow with controller action and the userdata can only be from 0x00 to 0xff which is one byte?
Yes it's intentional. IMO, we don't need or shouldn't parse Continuation normally. At the same time, I think those decoding code will be helpful if users are interested in the content of Continuation in some situations.

In file nx_action.go:
In NXActionController2PropUserdata, Userdata should be []byte instead of a byte.
Accordingly, the Marshal/Unmarshal function should be changed to handle a slice of byte.
We are not further decoding the properties of Continuation using DecodeContinuationProp. This function is defined but not used. Is this intentional?
Thanks,

Thanks for the review @ashish-varma .

I tried to use ovs-ofctl to add a flow with controller action and the userdata can only be from 0x00 to 0xff which is one byte?
Yes it's intentional. IMO, we don't need or shouldn't parse Continuation normally. At the same time, I think those decoding code will be helpful if users are interested in the content of Continuation in some situations.

Oh ok, let me re-confirm the ovs-ofctl and ovs side code to check the size of userdata.


feat: support auto create mcs with flags

What type of PR is this?
/kind feature
What does this PR do?

Which issue(s) does this PR fix?

Fixes #
Special notes for your reviewer:

Does this PR introduce a user-facing change?

/assign @wangyizhi1

/lgtm

/approve
/lgtm


Unlink\Link GMaven project problem

Sometimes the gmaven project does not unlink successfully. And it remains in the tool window. The problem is floating. Reproduced on OS Windows.

@kriegaex Hi
I tried to reproduce this issue on Windows and it worked correctly for me. I added logs to version 233.27.1 . If you have time, could you reproduce the problem again on version 233.27.1 and send me the IDE logs?

@grisha9, I sent the log to you via Telegram.

fixed in 233.28


How to control the size of the icon button and the distance between the left and right

How to control the size of the icon button and the distance between the left and right

can you please help me add icons? I dont know what and how to modify xml or add icon reference

try add some padding in the images themselves, and set toolbar height


[Feature Request]:  Add Kadanes Algorithm's Documentation

Is there an existing issue for this?

[X] I have searched the existing issues

Feature Description
Greetings !!
Abhay here :) I would like to add Kadane's Algorithm Documentation for the website and also the code in 4 languages.
I hope this finds useful.
Thank You
Use Case
Kadane's algorithm is used for optimizing the problem ,mainly for max subarray question
Benefits
No response
Add ScreenShots
No response
Priority
Medium
Record

[X] I have read the Contributing Guidelines
[X] I'm a GSSOC'24 contributor
[X] I have starred the repository

Sir where should i add this Doc ?

@Abhay182005dat Please complete your task otherwise I will close it.

Yeah sir I will do it as soon as possible,I am having internals so ...
Pardon me
Thank you
On Sun, 9 Jun, 2024, 9:38 pm Ajay Dhangar, @.***> wrote:

@Abhay182005dat https://github.com/Abhay182005dat Please complete your
task otherwise I will close it.
—
Reply to this email directly, view it on GitHub
https://github.com/CodeHarborHub/codeharborhub.github.io/issues/571#issuecomment-2156675356,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BCYSJ3DAN5PRDAL3ATCTXM3ZGR4YDAVCNFSM6AAAAABI3D3FP2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJWGY3TKMZVGY
.
You are receiving this because you were mentioned.Message ID:
@.***>


font and text

Hello, please add the feature of changing the font and font size and face to the middle text.

@kojofosu Should I do this ??

@kojofosu Should I do this ??

hi, yes if you have the bandwidith to work on it. We will all appreciate the help greatly 🙏🏾

@kojofosu I have added this method wating for the merge of the pull request.

@mahdiShamsPoour the method is been added now you can use it.


Add obofoundry topic to repo metadata

The OBO Foundry has recently created a topics page at https://github.com/topics/obofoundry. It would be great if you could add the obofoundry topic to your repository, since it's currently listed as active in the OBO Foundry. If you're not sure how to do that, follow the instructions here.   The main issue for this task is at https://github.com/OBOFoundry/OBOFoundry.github.io/issues/1538 in case you want some more context or to join the larger discussion.

@cthoyt Added to the ARO repo, thanks for reaching out :)

Thank you very much @arpcard  :))


Closes #1336 Feature Request: Replace smq_select() and sdg_select() with basket_select()@devel

Thank you for your Pull Request! We have developed this task checklist from the Development Process Guide to help with the final steps of the process. Completing the below tasks helps to ensure our reviewers can maximize their time on your code as well as making sure the admiral codebase remains robust and consistent.
Please check off each taskbox as an acknowledgment that you completed the task or check off that it is not relevant to your Pull Request. This checklist is part of the Github Action workflows and the Pull Request will not be merged into the devel branch until you have checked off each task.

[x] Place Closes #<insert_issue_number> into the beginning of your Pull Request Title (Use Edit button in top-right if you need to update)
[x] Code is formatted according to the tidyverse style guide. Run styler::style_file() to style R and Rmd files
[x] Updated relevant unit tests or have written new unit tests - See Unit Test Guide
[ ] If you removed/replaced any function and/or function parameters, did you fully follow the deprecation guidance?
[x] Update to all relevant roxygen headers and examples
[x] Run devtools::document() so all .Rd files in the man folder and the NAMESPACE file in the project root are updated appropriately
[x] Address any updates needed for vignettes and/or templates
[ ] Update NEWS.md if the changes pertain to a user-facing function (i.e. it has an @export tag) or documentation aimed at users (rather than developers)
[ ] Build admiral site pkgdown::build_site() and check that all affected examples are displayed correctly and that all new functions occur on the "Reference" page.
[x] Address or fix all lintr warnings and errors - lintr::lint_package()
[x] Run R CMD check locally and address all errors and warnings - devtools::check()
[x] Link the issue in the Development Section on the right hand side.
[x] Address all merge conflicts and resolve appropriately
[x] Pat yourself on the back for a job well done! Much love to your accomplishment!

@tamarasenior , sorry, I think I was a bit to quick with merging the pull request.
Could you add an item to NEWS.md?
You can use the same branch but you need to create a new pull request.


(AzureCXP) fixes MicrosoftDocs/azure-docs#39294

Created a variable to store the output.

The requested updates are in master. We'll close the pull request.


[Question] Error handling - How to catch all unhandled errors?

Hi,
First of all, thanks for this great library. I am planning to use it in production and before I do, I need to figure out how to catch all unhandled errors. For this, Telegraf documentation suggest the following -
bot.catch((err, ctx) => {
  console.log("Ooops, encountered an error for", ctx.updateType, err)
})
bot.start((ctx) => {
  throw new Error('Example error')
})
bot.launch()

On these lines, I do the following in my service:
constructor(@InjectBot() private bot: TelegrafProvider){
       this.bot.catch((err, ctx) => {
            console.log("Encountered an error for", ctx.updateType, 'Error:', err);
            ctx.reply("Something went wrong. Please try again later");
        });
}

However, this doesn't seem to work. What am I doing wrong?

Hey @Dr-Phone, thx! Module already has hardcoded catch:
https://github.com/bukhalo/nestjs-telegraf/blob/a309d8640d5a46e7a32d1298c9c1548adff7f1cf/lib/telegraf.provider.ts#L24
That's not good, i know)
The only thing I can offer at the moment is to import the sources of this module to your project and make it work the way you want it to. OpenSource power 🔥
I will think about how to solve this problem in the next release. I think Exception Filters built-in functionality of will work well for that.
And, feel free for make PR.

Hey @bukhalo, thanks for your quick reply. Exception Filters might actually work.
For now, I've littered the code with try-catch blocks everywhere due to the deadline pressure. I guess will have to think about it over the weekend.


vscode: Fix icon name

Looking at official vscode *.deb packages, the icon are named as vscde, vscode-insiders, etc. Let's follow that to make third-party icon theme users happy.
ref: https://az764295.vo.msecnd.net/stable/2ccd690cbff1569e4a83d7c43d45101f817401dc/code_1.80.2-1690491597_amd64.deb
ref: https://az764295.vo.msecnd.net/insider/9800cf6dd6bf4634889d60720ef46a400f3a7298/code-insiders_1.81.0-1690547214_amd64.deb
ref: https://github.com/VSCodium/vscodium/releases/download/<IP_ADDRESS>09/codium_<IP_ADDRESS>09_amd64.deb
Example:

After this change, with Adwaita icon theme (which itself does not provide a vscode icon):

With Colloid icon theme:

Looking how gentoo packages vscode, they are doing the same:

https://github.com/gentoo/gentoo/blob/bd298bdb1d8ee6ab61f3fadb9cc2d5169daf84c0/app-editors/vscode/files/vscode.desktop
https://github.com/gentoo/gentoo/blob/bd298bdb1d8ee6ab61f3fadb9cc2d5169daf84c0/app-editors/vscode/vscode-1.80.2.ebuild#L115

Fixes: #243513
passthru.tests are failing on master and are unrelated.
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

@ofborg test vscodium.xorg


User-defined printing function for Elements?

Given that Elements is a generic constructor for throughput which doesn't have any particular semantics attached to it, would it be possible to expose an API to print say MLoc instead of MElem? It would be a small but nice QOL improvement.

Hey, thanks for trying Criterion.rs, and thanks for the suggestion!
I suppose this would be nice in some cases, but this seems pretty niche. I'm not sure it would be worth the (admittedly small) maintenance and API-complexity burden. I'd be willing to change my mind if there's a lot of interest though. In the meantime, perhaps you could pipe the command-line output through a sed script or something similar?


AF5: sometimes a request just disappears after internal createTask event

[x] I've read, understood, and done my best to follow the *CONTRIBUTING guidelines.

NOTE: this is almost the same issue as https://github.com/Alamofire/Alamofire/issues/3392, but using plain HTTP and only one concurrent request, and we are not creating sessions per request -- just the one Session for all the requests now.
What did you do?
Our iOS app was using Alamofire 4 successfully; as part of upgrading to iOS14, we moved to Alamofire 5. Our application connects to a backend from iPad on wifi. Connection goes over local network to a locally hosted apache webserver.
Our application isn't doing anything very complicated. It now makes plain HTTP requests, and only one at time (that is limited by an NSOperationQueue with a concurrency limit of 1), using a single Session object which we keep as long as we want to make those requests.
Pseudocode for making our requests in AF5:
// executed just once
session = Alamofire.Session([...])

// then later, executed many times, only one active at once:
session.request(url, method: method, parameters: parameters, encoding: encoding, headers: httpHeaders)
        .debug()
        .validate()
        .responseData(emptyResponseCodes: emptyBodyAllowedForCodes) { response in
                   // handle result
        }

What did you expect to happen?
We expect all HTTP requests to result in the completion block being called.
What happened instead?
Very occasionally (like, once in thousands of requests), the completion block isn't called, and as far as I can tell, the request is never made to the server (since the logs are missing a corresponding entry around that time).
I attached an AF Logger and logged everything I could. From that I can see the following in the log for one of the 'dropped' HTTP requests:
HTTPAlamofireRequester.request called...
in AlamoFireRequester: url = http://redacted/indexedDocument/64317069, method = HTTPMethod(rawValue: "GET"), headers = ["Date": "Wed, 13 Jan 2021 23:02:47 GMT",
AHAFLogger session became invalid  = <__NSURLSessionLocal: 0x121110f60>
AHAFLogger requestDidResume: GET http://redacted/indexedDocument/64317069
AHAFLogger didCreateInitialURLRequest: http://redacted/indexedDocument/64317069
AHAFLogger didAdaptInitialRequest: req = http://redacted/indexedDocument/64317069 adapted req =http://redacted/indexedDocument/64317069
AHAFLogger didCreateURLRequest: http://redacted/indexedDocument/64317069
AHAFLogger didCreateTask: task = LocalDataTask <5F05B7E2-C6C0-4F4B-9F42-DBC9C3CEF521>.<1> orig req = Optional(http://redacted/indexedDocument/64317069)

The AHAFlogger lines are reporting Alamofire.Logger callbacks, so are a good sign of progress inside AF.
After the above log lines, there are no more lines mentioning the URL indexedDocument/64317069, and the completion block for that request is never called. So didCreateTask: task =  is the last we hear about the request from AF.
When I check the server logs for recent requests, I can't find any trace of the request that seemed to disappear.
For comparison, here's the same verbose logging output for an HTTPS request that completed as expected:
NMNM alamoFireRequester: url = http://redacted/indexedDocument/64950467, method = HTTPMethod(rawValue: "GET"), headers = ["Date": "Wed, 13 Jan 2021 23:02:41 GMT", "X-Nonce": "(redacted)"]
AHAFLogger requestDidResume: GET http://redacted/indexedDocument/64950467
AHAFLogger session became invalid  = <__NSURLSessionLocal: 0x12101bd80>
AHAFLogger didCreateInitialURLRequest: http://redacted/indexedDocument/64950467
AHAFLogger didAdaptInitialRequest: req = http://redacted/indexedDocument/64950467 adapted req =http://redacted/indexedDocument/64950467
AHAFLogger didCreateURLRequest: http://redacted/indexedDocument/64950467
AHAFLogger didCreateTask: task = LocalDataTask <C30B71C0-5EFF-4DC5-9666-476B43B05B58>.<1> orig req = Optional(http://redacted/indexedDocument/64950467)
AHAFLogger didResumeTask: LocalDataTask <C30B71C0-5EFF-4DC5-9666-476B43B05B58>.<1>
[connection] nw_resolver_start_query_timer_block_invoke [C810] Query fired: did not receive all answers in time for redacted:443
AHAFLogger didReceiveChallenge: task = LocalDataTask <C30B71C0-5EFF-4DC5-9666-476B43B05B58>.<1>, orig req = Optional(http://redacted/indexedDocument/64950467)
[connection] nw_resolver_start_query_timer_block_invoke [C811] Query fired: did not receive all answers in time for redacted:443
AHAFLogger dataTask: didReceive data: origReq = Optional(http://redacted/indexedDocument/64950467)
AHAFLogger dataTask: didReceive data: origReq = Optional(http://redacted/indexedDocument/64950467)
    [ ... multiple didReceive data lines removed for brevity...]
AHAFLogger dataTask: didReceive data: origReq = Optional(http://redacted/indexedDocument/64950467)
AHAFLogger dataTask: willCacheResponse: origReq = Optional(http://redacted/indexedDocument/64950467)
[Request]: GET http://redacted/indexedDocument/64950467
    [Headers]:
        Authorization: Basic (redacted)
        Date: Wed, 13 Jan 2021 23:02:41 GMT
        X-Nonce: (redacted)
    [Body]: None
[Response]:
    [Status Code]: 200
    [Headers]:
        Content-Length: 121529
        Content-Type: application/octet-stream
        Date: Wed, 13 Jan 2021 23:02:40 GMT
        Server: (redacted)
        X-Powered-By: (redacted)
    [Body]: 121529 bytes
[Network Duration]: 0.7587810754776001s
[Serialization Duration]: 2.0541716367006302e-05s
[Result]: success(121529 bytes)
2021-01-13 23:02:41.855298+0000 CMIC PP[9284:9086514] url = http://redacted/indexedDocument/64950467, response code = 200 headers = Optional([AnyHashable("Server"): (redacted), AnyHashable("Content-Type"): application/octet-stream, AnyHashable("Content-Length"): 121529, AnyHashable("X-Powered-By"): (redacted), AnyHashable("Date"): Wed, 13 Jan 2021 23:02:40 GMT])
SGSG response headers: Optional([AnyHashable("Server"): (redacted), AnyHashable("Content-Type"): application/octet-stream, AnyHashable("Content-Length"): 121529, AnyHashable("X-Powered-By"): (redacted), AnyHashable("Date"): Wed, 13 Jan 2021 23:02:40 GMT])
SGSG response error: nil
SGSG underlying error: nil
 QOQO req done for http://redacted/indexedDocument/64950467, status = 200
AHAFLogger didGatherMetrics: req = GET http://redacted/indexedDocument/64950467 (200)
 NMNM caseFileDocumentOperation:didFinishForCaseFile: id = 64950467
AHAFLogger didValidateRequest: req: Optional(http://redacted/indexedDocument/64950467)
AHAFLogger didValidateRequest: req: Optional(http://redacted/indexedDocument/64950467)
No request created yet.
AHAFLogger didCompleteTask: LocalDataTask <C30B71C0-5EFF-4DC5-9666-476B43B05B58>.<1> orig req = Optional(http://redacted/indexedDocument/64950467)
AHAFLogger requestDidFinish: GET http://redacted/indexedDocument/64950467 (200)
AHAFLogger request(Value): req: GET http://redacted/indexedDocument/64950467 (200) didParseResp: success(121529 bytes)

Of note this is that we get more AF events after the AHAFLogger didCreateTask:. And of course the web server log shows these successful requests as 200 OK codes.
Alamofire Environment
Alamofire version: 5.4.1
Xcode version: 12.3
Swift version: Swift 5 (as per Xcode build settings)
Platform(s) running Alamofire: iOS 14.2 on iPad
macOS version running Xcode: 11.1
Alamofire integration method: Carthage
Project language: mixed ObjC + Swift
Demo Project
Apologies, I don't have a demo project at the current time, but wondering if the above info is useful in seeing what the issue could be.

Thanks for the detailed report. How many requests are we talking about before a failure occurs? And your server logs show that the request is actually made, it just never calls back to the Session?
I would suggest adding more logging, in addition to what's available through EventMonitor, so you can track requests more closely, especially for the state of the Request. It's odd that you never get additional callbacks for requests that are actually made, since didCreateTask is only a few lines away from didResumeTask.

Thanks for the detailed report. How many requests are we talking about before a failure occurs? And your server logs show that the request is actually made, it just never calls back to the Session?
I would suggest adding more logging, in addition to what's available through EventMonitor, so you can track requests more closely, especially for the state of the Request. It's odd that you never get additional callbacks for requests that are actually made, since didCreateTask is only a few lines away from didResumeTask.

Failure to call the completion can happen relatively quickly - in the first 500 requests -- or it can take 10000 or more. It's rally unpredictable.
Server log shows request doesn't reach it. It almost looked like after createTask, AF wasn't auto-calling resume internally. I actually disabled auto-resume and manually called resume myself on each request after making it, just to explore that idea, but the problem still occurred.
So it feels like request isn't even leaving the iPad, but I'd need to use Wireshark or similar to make absolutely sure of that.
Thanks for the state logging idea, I'll do some logging on that.

Failure to call the completion can happen relatively quickly - in the first 500 requests -- or it can take 10000 or more. It's rally unpredictable.
Server log shows request doesn't reach it. It almost looked like after createTask, AF wasn't auto-calling resume internally. I actually disabled auto-resume and manually called resume myself on each request after making it, just to explore that idea, but the problem still occurred.
So it feels like request isn't even leaving the iPad, but I'd need to use Wireshark or similar to make absolutely sure of that.
Thanks for the state logging idea, I'll do some logging on that.

Sorry, how do I get the state? task.currentRequest.state doesn't work.

Sorry, how do I get the state? task.currentRequest.state doesn't work.

Can you share the code you're using to enqueue requests onto the OperationQueue? I set up a simple test and repeatedly ran 100,000 requests to a local test server without issue.

Can you share the code you're using to enqueue requests onto the OperationQueue? I set up a simple test and repeatedly ran 100,000 requests to a local test server without issue.

Hi, unfortunately I've run out of time with the client for investigating AF side, and they want to see what the app will do if we just use the usual iOS comms directly.
I'll certainly update you here with the results as it might be useful.

Hi, unfortunately I've run out of time with the client for investigating AF side, and they want to see what the app will do if we just use the usual iOS comms directly.
I'll certainly update you here with the results as it might be useful.

Hi again, we're still looking at AF.
I've found that if I run our app in the profiler (using Allocations instrument), we get the issue really fast - within hundreds of attempts, not thousands. Does your example run ok in the profiler?
Btw our app is currently configured to build in Debug mode for the profiler.

Hi again, we're still looking at AF.
I've found that if I run our app in the profiler (using Allocations instrument), we get the issue really fast - within hundreds of attempts, not thousands. Does your example run ok in the profiler?
Btw our app is currently configured to build in Debug mode for the profiler.

It would be useful if you posted your enqueueing code, otherwise I'm just guessing at how you're enqueueing all of those requests. I'll give it a try in Instruments.

It would be useful if you posted your enqueueing code, otherwise I'm just guessing at how you're enqueueing all of those requests. I'll give it a try in Instruments.

I've created a simple app which runs 10,000 requests in an OperationQueue one at a time and haven't been able to replicate a failure. It would be great if you could create a test project which replicates the issue (I can provide my own request URL).

I've created a simple app which runs 10,000 requests in an OperationQueue one at a time and haven't been able to replicate a failure. It would be great if you could create a test project which replicates the issue (I can provide my own request URL).

I'll do it if I have time, but client has other ideas. They're asking me to submit a ticket to Apple support about this, against my advice. We all know the result of a TSI will be "Not our API, not our problem!"

I'll do it if I have time, but client has other ideas. They're asking me to submit a ticket to Apple support about this, against my advice. We all know the result of a TSI will be "Not our API, not our problem!"

Have you solved it? I also have the same result. very occasionally and randomly. Do you use Alamofire 4 instead of 5?

Hi, I forgot to provide a information that, I cancel  a request everytime before session request( Base on our app).  I don't know if this will help.

Mark. I also have this question.

Hi Team, Any update for this issue, we are commonly facing the same issue on iPhone 13 so please let us know how we can deal with and resolve it from our side.
When we checked and debugged Alamofire task session LocalDataTask did not execute. Our API and Server both are working properly.
 [LocalDataTask
<5127CF78-4526-4AE5-9A23-1C2F6C3BDA09>. <70›, LocalDataTask
<6784E32E-4FF1-49E5-A77F-CA43906F5C64>.<71>,
LocalDataTask
<7CA70A66-0B44-434C-9AC9-E7B6897DCBDD>. <72›, LocalDataTask
<2099A752-4499-4524-B144-DC4BBE6DCDF5>. <74>, LocalDataTask
<D10BAEED-CE41-423C-AE1C-01F7D7E50CB8>.<77>, LocalDataTask
<2BD6AE21-DFE9-4F2C-931A-48CD3BEC6511>. <78>, LocalDataTask
<AEF68696-7385-4545-90EA-8EAE62214A92>.<79>,
LocalDataTask
<79E98006-D02E-42DB-BA08-D706EC30CE33>. <80>,
LocalDataTask
<9568184D-8658-4971-82C6-9D7E6C8F1280>.<81>,
LocalDataTask
<7212BADF-C4DF-4E31-A5D3-80F60883D584>. <82>,
LocalDataTask
<A23108C1-E527-460F-9FB1-CDC6AF7FE37A>. <83>, LocalDataTask
<B0C30D69-36A0-4906-A7CE-549A259B014B>. <84>, LocalDataTask
<B706454C-A4A7-4D61-859E-48A12EFAD736>.<85›, LocalDataTask
<014E64D4-2571-4875-9BC4-6FDABF@CAC15>. <86>, LocalDataTask
<BOD&CADE-D88E-4D99-B915-36092524A185>.<87>, LocalDataTask
<AD3B5D9D-4B3C-448A-A944-C90D6AB1D8F1>) <88>,
LocalDataTask
<34900E16-187-4A85-AE27-ABAE801E5D0B>.<89>,
LocalDataTask
<5E81D5EF-F18F-4CE7-BCFE-281597C68EC1>, <90> ]


update pyspec dir

it looks like the dir in the Makefile was not updated during some file tree reorg

looks like this is fixed in #2346.


Are the models pre trained on imageNet ?

Dear
Thanks for sharing the code of this great idea, I was wondering if the models are trained from scratch on cifar 10 or build upond pretrained weights from image net ?
Best Regards

Thanks. The models are trained from scratch and not pre-trained on any
larger datasets. The objective was to test performance in low data
situation.
Pranav.
On Thu, 30 Sep, 2021, 2:33 am Oussamab21, @.***> wrote:

Dear
Thanks for sharing the code of this great idea, I was wondering if the
models are trained from scratch on cifar 10 or build upond pretrained
weights from image net ?
Best Regards
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pranavphoenix/VisionXformer/issues/1, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADYZSJUQW2FDAMEJTX2WRN3UEN5LNANCNFSM5FAZUZMA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.


Double, weirdly titled books in list

When pushing highlights to readwise, I get double books, once with their title and the highlights and once with a string of characters for a title and the same highlights. (like : 7420a5e5-a596-41cc-9a4a-2fd2f0860dc4)
I can delete them in readwise but they show up again when I do another sync with October. Both bought books and documents I put on the reader directly are affected.

Hi @timbruls,
Hmm, books with GUIDs are books that have been purchased from the Kobo store to my knowledge although I haven't done extensive testing with Kobo purchased titles.
Are you also using the official Kobo extension from Readwise as well? If you were, it seems reasonable to expect that you'd get duplicates (both via Readwise direct and from October) although I hadn't considered it before now so October should probably not try to process officially purchased books.
As for items you've put on the reader yourself, that doesn't sound quite right.
If you're on the latest version, would you be able to go to the Settings page, click the button to open your logs folder and email the last few logs (or if you know which one corresponds to your recent duplicate uploads) to<EMAIL_ADDRESS>and I'll see if I can spot anything that might give away the cause?
Thanks for the heads up

I will test with the official extension turned off, I hadn't considered that. If it still persists, I will mail you the logs :)

Hi @timbruls, I've just released v1.3.2 which should ignore store-bought Kobo titles now. You'll need to delete any duplicate uploads one time however.

Hey @timbruls, just letting you know that v1.4.0 has just released which has a toggle in Settings for opting into syncing store-purchased titles.
v1.3.2 also introduced some bugginess in querying so not all sideloaded titles may have synced with earlier versions.


refactor(@angular-devkit/schematics-cli): use latest version of jasmine for schematics

Schematics project install jasmine, but its of very old version.
PR Checklist
Please check to confirm your PR fulfills the following requirements:

[ ] The commit message follows our guidelines: https://github.com/angular/angular-cli/blob/main/CONTRIBUTING.md#-commit-message-guidelines
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[x] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[ ] Other... Please describe:

What is the current behavior?
Currently jasmine version installed is very old
Issue Number: N/A
What is the new behavior?
Install the latest version of jasmine
Does this PR introduce a breaking change?

[ ] Yes
[x] No

Other information

Hi Santosh, check out our commit message guidelines for pass the linting rule 😉

Hi Santosh, check out our commit message guidelines for the linting rule to pass 😉

Sorry raised a PR after years, let me fix this

@JeanMeche should be ok now. But I was wondering if we should keep jasmine, my first thought was to remove it, but we can discuss and decide.

The error looks not related to this PR

In May, the Angular Robot tried to upgrade and got rejected because Jasmine 5 doesn't run on Node 16 anymore: https://github.com/angular/angular/pull/50284
@santoshyadavdev I am all ears regarding dropping Jasmine. What are your reasons?

Got you.
I'd like to add that there are also reasons to use Jasmine instead of Jest:

Sound testing framework: The Angular team uses Jasmine as well. I guess the same is true for the rest of the Google Angular applications. In general, though, I also confirm that the majority outside of Google uses Jest.
Legacy Code: Especially Angular CLI-based applications that started out with Jasmine and already have a good testing code base might not have the need to switch to Jest.
Full browser environment: Certain types of tests need a full browser environment. Jest runs on JSDOM, which is not a browser and doesn't give you that kind of support.
Better support for asynchrony: There are some edge cases where Jest fails on asynchrony management. Jasmine doesn't have those obstacles.
Better Integration: Maybe the main argument for Jasmine. Jasmine is integrated into the build system of Angular. Because of that, Jest had many performance issues with ESM.

Either way, official support for Jest will finally come to Angular as well. There is already experimental support since version 16. So devs will be able to choose.
If you are interested in more, I've written a whole article about it: https://dev.to/this-is-angular/angular-testing-in-2023-past-present-and-future-j5m


strict type check not working

Environment

Operating System: Windows_NT
Node Version:     v20.10.0
Nuxt Version:     3.11.2
CLI Version:      3.11.1
Nitro Version:    2.9.6
Package Manager<EMAIL_ADDRESS>Builder:          -
User Config:      devtools
Runtime Modules:  -
Build Modules:    -

Reproduction
https://github.com/PurpleDevIO/air-hockey/
This issue happens on any nuxt repo in my machine
Describe the bug
Strict mode does not work with vs-code or nuxi typecheck

Additional context
I have tested manually setting strict to true in both tsconfig.json and nuxt.config.ts but still errors in implicit anys
Logs
No response

It works once the variable is actually read


Update spotify.config.json.example

Changed domain and port to use default Spotify settings in https://developer.spotify.com/dashboard/applications
Else we should document how to add the new URI to the application list.

Don’t forget to change instruction of installation

yes will write all readme once we will pull the PROD version .


⚠ Adacta-fintech.com has degraded performance

In 263cfd2, Adacta-fintech.com (https://www.adacta-fintech.com) experienced degraded performance:

HTTP code: 200
Response time: 932 ms

Resolved: Adacta-fintech.com performance has improved in 563e547.


Bug: QOS profile is not rendered on member interfaces when using "LACP Fallback Individual"

Discussed in https://github.com/aristanetworks/avd/discussions/3794

Originally posted by PHipnarowicz March 29, 2024
Hi AVD Developers,
When creating update_ethernet_interface_cfg function important element is missed.
This is service_profile getting from qos_profile and that's why port channel members can't inherit QoS settings.
For non-port-channel member, service_profile is extra updated, but for port-channel member with lacp_fallback this step is missed.
Is this something that can be fixed?

Move this line: https://github.com/aristanetworks/avd/blob/8dd875a4174dfb446d2e882c8ada071afc21d05a/ansible_collections/arista/avd/roles/eos_designs/python_modules/connected_endpoints/ethernet_interfaces.py#L199
To this shared function:
https://github.com/aristanetworks/avd/blob/8dd875a4174dfb446d2e882c8ada071afc21d05a/ansible_collections/arista/avd/roles/eos_designs/python_modules/connected_endpoints/ethernet_interfaces.py#L81

Also update a test to expose this.


README: add deprecation notice

What changed?
Added a deprecation notice to the README and linked the new grafana/grafana-plugin-examples repo as the place to go to.
Check out the updated README

Totally agree, let's wait with these deprecation notice PRs for now 👍

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Add Redis configuration when integrating with Redis charm

Overview
After integrating with Redis, the charm will change the Synapse configuration with host and port information from relation data.
Rationale
Redis integration is required by horizontal scaling configuration.
Juju Events Changes
redis_relation_updated changes Synapse configuration via pebble_service.
Module Changes
N/A
Library Changes
N/A
Checklist

[x] The charm style guide was applied
[x] The contributing guide was applied
[x] The changes are compliant with ISD054 - Manging Charm Complexity
[x] The documentation is generated using src-docs
[ ] The documentation for charmhub is updated.
[x] The PR is tagged with appropriate label (urgent, trivial, complex)

Documentation for charmhub will be updated after adding integration test.

Lint checks failed for 6210385

Success: no issues found in 47 source files ************* Module backup src/backup.py:253:20: R1737: Use 'yield from' directly instead of yielding each element one by one (use-yield-from)

Your code has been rated at 10.00/10

To be fixed by https://github.com/canonical/synapse-operator/pull/222


fix invalidate_processor_cache length to match mmap() len

The old memory data currently used may be an illegal value, causing famfs to fail to initialize correctly (possibly causing a segfault).

I need to think a bit about this; Looking through the superblock and log lengths, the handling is a bit inconsistent. The intent is:

Superblock length is fixed at 2MiB
Log length is variable, but always a multiple of 2MiB (but currently we have only tested with the default log length of 8MiB)

Currently there are log_length fields in both the superblock and the log header; having the log length in the superblock is convenient because it always needs to be verified before the log can be mapped - and this can handle a non-invariant log length.
Unless I'm missing something, this change is basically a NOP except that it won't handle a different log length.

I think there are two reasons for the modification of this PR:

When initialized for the first time, the length field in the memory pointed to by pmem may be a random value? Flush the cache based on this length does not meet expectations.
After changing the log size of famfs, the cache cannot be flushed as expected. (When the log size becomes larger, part of the cache will not be cleaned. When the log size becomes smaller, the mkfs tool will always core dump, because it accesses a virtual address range outside mmap.)

Regards! :)

The branch/PR I'm working on will be consistent as to how it references superblock and log lengths...

Please take a look at https://github.com/cxl-micron-reskit/famfs/pull/74. It generalizes the configurable log length, and I think it fixes the cache related issues.
It also fixes some buggy code related to FAMFS_ALLOCATION_UNIT, although it does not make that a configurable/superblock parameter (that can happen soon).

Close this PR, this issue will be fixed in #74


[Android] Fix the launch screen not work correctly

The launch screen dialog has its own lifecycle. The main Activity's attributes
should not be directly apply to the launch screen dialog. So remove these here.
As to XWALK-4688, if you want to apply custom on xwalk_launch_screen, you need
to call window.screen.show(); in your javascript.


chore(starknet_gateway): use the get nonce method of the stateful validator trait

This change is

#2948 : 2  dependent PRs (#2404 , #2911 )
#2982  👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.


bpf: add multicast in MAX_OVERLAY_OPTIONS

Fix mismatched brackets in multicast code and add ENABLE_MULTICAST in MAX_OVERLAY_OPTIONS to check such mismatches during compile time in future.

/test

ci-ginkgo: https://github.com/cilium/cilium/issues/32075


Output of Flutter tool, when iOS can't be built, doesn't fit 80 columns

Terminals seem to default to 80 chars wide, however the error message for when flutter tool can't build the iOS app is wider the 80 chars:

Long term we should implement automatic detection of the width and wrap to that width. (We need args to do that too.)


Inconsistent EXPIRE: EXPIRE with empty string as TTL should report an error

Steps to reproduce
Run the commands mentioned in the test on Line 81 in the file https://github.com/AshwinKul28/dice-tests/blob/main/tcltests/unit/expire.tcl#L81.
Expected output
The expected output when the above set of commands when run on Redis
'*not an integer*'

Observed output
The observed output when the above set of commands when run on DiceDB
'ERR invalid expire time in 'expire' command'

The steps to run the test cases are mentioned in the README of the dice-tests repository.
Expectations for resolution
This issue will be considered resolved when the following things are done:

Changes in the dice repository code to meet the expected behavior.
Successful run of the tcl test behavior.

You can find the tests under the tests directory of the dice repository and the steps to run are in the README file. Refer to the following links to set up DiceDB and Redis 7.2.5 locally

Setup DiceDB locally
Setup Redis 7.2.5 locally
Setup DiceDB CLI

@lucifercr07
can i take this up?

@Abh-ay Thanks for picking this up. Assigned!

Hi @AshwinKul28 @lucifercr07
Just verify this command in redis and it accept -ve value also refer below ss

Should we handle this or keep as it is?


Discussion: Replacing or complementing StaticString with String in APIs

(Hello and thank you for all of your work on PlaydateKit, it's really great!)
I wanted to start a discussion around either 1) moving from StaticString to String now that some basic support has landed in Swift Embedded or if StaticString is still desirable (for performance/simplicity/memory/? reasons), 2) complementing any initializers/functions that accept StaticString with an overload for String.
I've been experimenting with this in my working fork of PlaydateKit and so far things seem to be working out pretty well. I'm curious to hear if there's interest in this, and if so, which direction would be preferred. If there's interest and a direction is determined, I would be happy to open pull requests and help move things along! Thanks for reading 🙏!
As just one example, for System.log():
Current
public static func log(_ log: StaticString) {
    let logToConsole = unsafeBitCast(
        system.logToConsole.unsafelyUnwrapped,
        to: (@convention(c) (UnsafePointer<CChar>?) -> Void).self
    )
    log.utf8Start.withMemoryRebound(
        to: CChar.self,
        capacity: log.utf8CodeUnitCount + 1
    ) { pointer in
        logToConsole(pointer)
    }
}

With String instead:
public static func log(_ log: String) {
    let logToConsole = unsafeBitCast(
        system.logToConsole.unsafelyUnwrapped,
        to: (@convention(c) (UnsafePointer<CChar>?) -> Void).self
    )

    // No idea if this is the best way to go about this, by the way!
    var utf8Log = log.utf8CString
    withUnsafeMutablePointer(to: &utf8Log[0]) { pointer in
        logToConsole(pointer)
    }
}

Which, of course, allows for potentially useful things like:
var message = ""
if Bool.random() {
  message = "Yay!"
} else {
  message = "Oh no!"
}

System.log("A logged message: \(message)")

(This is a silly example but a more concrete one is e.g. using a dynamically generated String as an argument to Graphics.Bitmap(path: ...) and I'm sure there are potentially many more!)

Hi! That's definitely next on the list of things to add, I'm just waiting for the fix for https://github.com/apple/swift/issues/74386 to land in a nightly build as it was causing issues (should be in the next few days)

Ah, wonderful! Thank you so much 🙌 ! I hadn't noticed the crash because in my working fork I had just been replacing StaticString in the signatures rather than overloading them 😅.


Fix observing changes in local data provider

Fixes #398

This one needs to reverted, it break many things.

Partial revert: https://github.com/vaadin/px-data-grid/commit/0bbae74b897f96d07b7e2e21eb63e28d37aedcd5


Adds Persian corpus

Translators:

Saeed Torabzadeh<EMAIL_ADDRESS>Alireza Davoudi<EMAIL_ADDRESS>Ebrahim Soroush<EMAIL_ADDRESS>

Hi @adavoudi, thank you for this impressive contribution. I'm going to check over the additions and if everything checks out I'll merge them in.


Handle already released source files that dont match scriptKind

In LS we have document registry which maintain the cache by the scriptKind but there is not such concept at program level.
So when we try to reuse program there is no fail-safe way to check if scriptKind doesn't match between program reuse unless we create the new source files.
But this also means that we could try to get new sourceFile two times from the host.

when trying to see if we can reuse the program structure using old source file list and checking if new source file makes it such that program structure can be reused or does it need to be constructed from scratch.
When we decide that we cant reuse the structure so we try to build program by scratch by processing root files, dependencies etc.

In LS when we try to get the source file, we see if we already have the source file from old program and if yes and their script kinds don't match we just get updated source file from the document registry,
But if the old Source file's scriptKind doesn't match then we release that source file (removing our ref count) and acquiring the new source file with correct script kind.
What this resulted was, when we try to create source file when trying to determine if program structure can be reused, we would release the old document with script kind and acquire new document with correct script kind. But when the program structure is not reused, we would try to release the document with old Script kind but the bucket had the document with new script kind.
Fixes #54381

Awesome investigation @sheetalkamat ! Thank you for looking into this
@RyanCavanaugh Would we be able to get this in a TS 5.1.x recovery build? Copilot seems to be triggering this crash a fair amount by opening and closing documents quickly, but it's not a copilot specific issue

@typescript-bot cherry-pick this to release-5.1


[Gradle Plugin] Option to make detekt system resource availability aware

Your feature request may already be reported!
Please search in the issues before creating a new one.
Expected Behavior

Configuration option to control amount of detekt processes scheduled in parallel. Similar to maxFork option from Gradle work api.
Current Behavior

Having multi-module project with 50 modules. Configured like this:
subprojects { subProject ->
            subProject.apply plugin: 'io.gitlab.arturbosch.detekt'

            detekt {
                debug = false
                toolVersion = versions.detektPlugin
                input = subProject.files("src")
                config = files("$rootDir/code_quality_tools/detekt-config.yml")

                reports {
                    xml.enabled = false
                    txt.enabled = false
                    html.enabled = true
                }
            }
        }
    }
}

Running ./gradlew detekt results in (a try to spawn) 50 java processes at the same time. Which hits the OOM wall on a CI containirezed machine.
Practically, we have to disable running detekt in parallel and run detekt module-after-module.
Checking every module in a multi-module project looks like unsupported use case for now?
Context

Related issues I found:

https://github.com/arturbosch/detekt/issues/1295
https://github.com/arturbosch/detekt/issues/2035
https://github.com/arturbosch/detekt/issues/2119

Thanks for the feedback and for describing your use case in detail!
I can understand the limitations in some environments and how it hinders further adoption of detekt.
I'm not sure whether the value/effort for supporting this use case is given here. @arturbosch might want to assess first.

For the past 2-3 years android community shifts into modularizing the app into pieces approach. Its movement has been recognized by Google dev team. And these days Google dev relation team is encoraging application development by splitting its code into separate gradle projects as well. https://www.youtube.com/watch?v=PZBg5DIzNww
My understanding that the same target audience that is interested in having healthy code base and having clean code by setting up static anylizers is also the same audience which tends to follow suggested best practices.
And so I think projects which install detekt are likely to be projects organized with submodules. And supporting multi-module case is not that uncommon. But of course you as maintainer knows better your core supported use case.
Would you recommend to drop detekt gradle pluging and switch to execute jar directly? Possibly with some custom helper scripts on top for multi-module projects?

@plastiv starting from detekt 1.7.0 we won't spawn any java processes anymore as we have moved to a cached classloader approach.
Please test 1.7.0-beta2 on your multi module project and share your experience.
Note that this also boosts performance by a factor of 2-4.

If you use org.gradle.parallel=true gradle should be processor aware and only start #cores of detekt tasks?

Correct. I am using a CI machine with 64 cores. If I limit parallel option explicitly org.gradle.workers.max=8 OOM is still reproducable for me. Which was one of the reasons I have created this issue, since other tasks (kotlin compile, android assemble, ktlint, checkstyle) seems to have no issues running in 64 workers.
So it looks like my issue is triggered by detekt. But now I'm not sure. I have learned recently that org.gradle.jvmargs= isn't the total build memory budget. Instead org.gradle.jvmargs= controls gradle daemon only. Kotlin daemon, jvm test forks and any external gradle plugin code (like detekt) is free to ignore and use it's own settings. Which makes sense, of course. But it limits control options to profile and monitor total memory consumption per CI run.
I have tried but so far I couldn't have consistent results with 1.7.0-beta2 either. Mostly due to lack of profiling tool. There is a noise in build output triggered by other changes. Maybe you can suggest what to use to fine tune memory settings?
I also plan to make more investigation next week and will report my findings.

@plastiv how large is your codebase?
I know of a specific "memory leak" in detekt which got deprecated in 1.7.0 and needs to wait until 1.8.0 to get removed. It basically does not free memory (from the Config objects) until all files are analyzed.

I have tried but so far I couldn't have consistent results with 1.7.0-beta2 either.
Does it mean that sometimes still OOM occur?

I guess if you have 64 cores you have increased the gradle daemon memory to minimum of 8gb?
What kind of OOM occurs? Metaspace OOM maybe (with 1.7.0-beta2)?

I've invested more into my issue research and I think now that I have misinterpreted original error message
 java.lang.OutOfMemoryError: unable to create new native thread
 	at java.lang.Thread.start0(Native Method)
 	at java.lang.Thread.start(Unknown Source)
 	at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
 	at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)

as an error triggered by exhausting memory budget. Which I later connected with known classpath issue.
Turns out the same exception is thrown when reaching out PIDs max limit. Enforcing PID limit is (new) feature on kubernetes https://kubernetes.io/blog/2019/04/15/process-id-limiting-for-stability-improvements-in-kubernetes-1.14/ which is managing our CI nodes. I can confirm that build is capable to start 500 threads per run by checking VisualVM locally.
I do still think that spawning over 500 threads with org.gradle.workers.max=8 is smelly. But I'm almost certain that it is a coincidence that thread # 501 happened to be a detekt's thread. My issue is outside of the detekt project.
Looking forward to the days when android projects could be built on JDK11. Which has Java flight recorder built in and which is container aware out of the box. This would save my investigation time considerably.

how large is your codebase?

2.5k files, 100k lines of code, 60% kotlin, 40% java

Does it mean that sometimes still OOM occur?

Yes. But I currently think that OOM swallows out of PID exception. Looks like it was not considered as important detail to specify.

I guess if you have 64 cores you have increased the gradle daemon memory to minimum of 8gb?

Build peaks 2.7Gb only for gradle daemon, 1.2Gb for kotlin daemon and if thread stack size is 8Mb (apparently, default these days) then 500 threads == 4Gb just for the stacks (hello kotlin coroutines 👋where are you?)

Metaspace OOM maybe

Gradle daemon peaks 300Mb metaspace and Kotlin daemon 100Mb. Unless specified metaspace has no limit. But if reached have clear error message that metaspace is the issue (checked before).

Thanks for the detailed feedback and analysis @plastiv. It's always interesting to see how detekt is used in different scenarios.
I guess this can be closed now, since there's (atm) nothing we can do about it, right?

@plastiv thanks for sharing your profiling experience.
I find it interesting and challenging to profile applications.
Lastly I hunted some classloader memory leaks in detekt with good old VisualVM :).


Create staging and intermediate models for bootcamps application related tables

What are the relevant tickets?

https://github.com/mitodl/ol-data-platform/issues/578
Description (What does it do?)

Creating  the following staging models for bootcamps application related tables

stg__bootcamps__app__postgres__courses_personalprice
stg__bootcamps__app__postgres__courses_installment
stg__bootcamps__app__postgres__django_contenttype
stg__bootcamps__app__postgres__applications_applicationstep
stg__bootcamps__app__postgres__applications_applicationstep_submission
stg__bootcamps__app__postgres__applications_courserun_application
stg__bootcamps__app__postgres__applications_courserun_applicationstep

Creating  int__bootcamps__applications to combine these staging tables for replacing https://bi.odl.mit.edu/queries/969/source

How can this be tested?

dbt build -select +int__bootcamps__applications

and compare it with https://bi.odl.mit.edu/queries/969/source (Note that I didn't include users profile fields in this intermediate as that's in int__bootcamps__users)

The builds all ran clean and the definitions all look good to me I'm signing off

sorry meant to approve not close...


CloseButton does not work

I set the value for closeButton = true in all of my steps, but the close button doesn't show, and the user can't close the turorial until the end, Am I wrong with something?

Did you find a solution to this problem? I have the same issue right now.


feat: Use the requests Session interface for runtime assertions

          I think it would be wise to lean on the Python requests Session interface. Which does runtime assertions to verify the types. Our testing strategy will reveal any issues related to the incorrect type being used.

Originally posted by @tdstein in https://github.com/posit-dev/posit-sdk-py/pull/300#discussion_r1833127353

@schloerke - should we close this as not planned?


Vm ci fixnetbsd

I'm not sure what's happened here, it seems to have not run CI at all.  It looks as if you're applying three commits, one of which is the top of the branch you're trying to merge into.  Can you rebase on vm-ci and force push to see if that fixes it?

@devnexen I have just rebased your branch on top of main to get a clean CI run.

@devnexen I have just rebased your branch on top of main to get a clean CI run.

please do :-) thanks


Fixing a typo

fixing @ oasislabs/docs too

Apologies for not being more clear.
Can you move this here, as the docs are being removed from this repo: https://github.com/oasislabs/docs/pull/89

yeap did both. same typo appears here too: packages/client/README.md


Change number of coco classes in detection recipe

🚀 The feature
Infer the number of classes from the dataset without hard-coding it, for example with
def get_dataset(name, image_set, transform, data_path):
    paths = {"coco": (data_path, get_coco), "coco_kp": (data_path, get_coco_kp)}
    p, ds_fn = paths[name]

    ds = ds_fn(p, image_set=image_set, transforms=transform)
    num_classes = len(ds.coco.cats)
    return ds, num_classes

This means that we need to remap the classes with continuous indexes, for example with
class CocoDetection(torchvision.datasets.CocoDetection):
    def __init__(self, img_folder, ann_file, transforms):
        super().__init__(img_folder, ann_file)
        self._transforms = transforms
        self.mapping = self.map_coco()

    def map_coco(self,):
        mapping_coco = {}
        for idx, c in enumerate(self.coco.cats):
            mapping_coco[c] = idx+1
        return mapping_coco

    def __getitem__(self, idx):
        img, target = super().__getitem__(idx)
        target = copy.deepcopy(target)
        for t in target:
            t['category_id'] = self.mapping[t['category_id']]
        image_id = self.ids[idx]
        target = dict(image_id=image_id, annotations=target)
        if self._transforms is not None:
            img, target = self._transforms(img, target)
        return img, target

Lastly, the number of classes for the model could be num_classes+1
model = torchvision.models.detection.__dict__[args.model](
     weights=args.weights, weights_backbone=args.weights_backbone, num_classes=num_classes+1, **kwargs
)

Motivation, pitch
The number of classes to train coco is hard coded and is 90+1.
https://github.com/pytorch/vision/blob/18b39e362e7ae4b58f9638f90246f7651fe3de98/references/detection/train.py#L37
This value does not reflect the number of classes that coco has (80) but the highest index since some indexes are not used.
Since the number of classes defines the head of detection architectures
https://github.com/pytorch/vision/blob/18b39e362e7ae4b58f9638f90246f7651fe3de98/torchvision/models/detection/faster_rcnn.py#L354-L357
it results in a small increase of memory usage (and may result in strange behaviors).
Alternatives
No response
Additional context
No response

@rvandeghen Thanks for the proposal.
I agree it's annoying the fact that all of TorchVision's models keep the 10 unused categories. This has been like this for years and unfortunately "fixing" it now means breaking some models. There might be a mechanism that could allow us to keep the old models as is and fix it for new ones, but I suspect it will introduce quite a lot of extra code.
So my OCD is definitely annoyed but this issue whenever I see it but I am tempted to say it's probably a nofix issue because of the implications that this can have on BC.
Let me know what you think.

@datumbox I fully agree that there is a feeling of “it's too late now“. However I thought that it would have been the good timing since we have done #2707 and #5307.
There are currently 8 detection models so I think retraining them should not be a huge effort but it could be worth to use that strategy for the upcoming ones.
I can retrain some of them if necessary and provide the weights.
Let me know.

The challenge is that we will need 2 versions of meta-data for the labels, then custom mechanisms on the reference scripts so that one can validate/do-inference with the old models while train the new ones with reduced classes. Then we would need to update our documentation and code examples to indicate that some models are on a reduced category list etc. Perhaps there are more things we need to change.
Unfortunately we can't really replace the existing 8 models with the new ones. This would violate the strong BC guarantees that TorchVisio offers. The use-case in mind is transfer learning. Someone might have already fit new models on top of our weights and now they wont work or they wont be reproducible.
As much as I would like to fix this, I think this is not something we can fix without breaking BC or adding a lot of extra code to tackle corner-cases. I think I'll leave this proposal open, in case we can adopt it if we introduce a major BC-breaking change on the area of Detection. There are quite a few things we need to do (such as supporting multiple losses, removing transforms from within the model etc) and many of them break BC. Perhaps we can sneak in this change then.


Failure in updating VM Scale Set stage

Unfortunately, never could I successfully run the pipeline due to the failure below. It happens consistently to Ubuntu 2204, Windows 2022 and Windows 2019 (I have not tried Ubuntu 2004 but I think it will exhibit the same issue). Any idea how to make it work? (I hid the actual id values with MASKED word)

2023-02-27T09:14:10.0761208Z ##[section]Starting: Update Virtual Machine Scale Set
2023-02-27T09:14:10.0900984Z ==============================================================================
2023-02-27T09:14:10.0902043Z Task         : PowerShell
2023-02-27T09:14:10.0902719Z Description  : Run a PowerShell script on Linux, macOS, or Windows
2023-02-27T09:14:10.0903432Z Version      : 2.212.0
2023-02-27T09:14:10.0904033Z Author       : Microsoft Corporation
2023-02-27T09:14:10.0904725Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
2023-02-27T09:14:10.0906234Z ==============================================================================
2023-02-27T09:14:11.2617757Z Generating script.
2023-02-27T09:14:11.2670658Z Formatted command: . 'C:\a_w\109\s\azuredevops-buildagents\scripts\update-vmss.ps1' -ClientId MASKED -ClientSecret MASKED -ResourceGroup MyBuildAgents -SubscriptionId MASKED -TenantId MASKED -VmssNames win2022buildagents -ManagedImageId "/subscriptions/MASKED/resourceGroups/Packer-Configuration-RG/providers/Microsoft.Compute/images/windows2022-29087"
2023-02-27T09:14:11.3085143Z ========================== Starting Command Output ===========================
2023-02-27T09:14:11.3330685Z ##[command]"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'C:\a_w_temp\3cfe15c5-f726-4a1b-8488-a165c8b246a9.ps1'"
2023-02-27T09:14:17.2493233Z ERROR: The Resource 'Microsoft.Compute/virtualMachineScaleSets/win2022buildagents' under resource group 'MyBuildAgents' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
2023-02-27T09:14:17.4683893Z Updated Virtual Machine Scale Set ManagedImageId: win2022buildagents - /subscriptions/MASKED/resourceGroups/Packer-Configuration-RG/providers/Microsoft.Compute/images/windows2022-29087
2023-02-27T09:14:17.6757754Z ##[error]PowerShell exited with code '1'.
2023-02-27T09:14:17.7218071Z ##[section]Finishing: Update Virtual Machine Scale Set

Apparently the VM set should pre-exist. I guess we can close this issue.


feat: remove if-else blocks in chart for k8s compatibility

no-work-item

@kushthedude the ** syntax seems to be not available.


Fix issues 1-5-2022

Changes for:
https://github.com/towhee-io/towhee/issues/398
https://github.com/towhee-io/towhee/issues/397

/lgtm