reduce npm script indirection and surface area

a somewhat aggressive approach to shaving off some of the indirection and complexity of working with packages in the monorepo.
largely inspired by recent bundling and packaging work where many commands were inconsistently stacked together, split, and realiased in root with new names.
goals

eliminate anything that isn't documented, meta/build/lint flows can be documented separately outside of package.json.
merge build and bootstrap flows
remove build only scripts
remove npm hooks (prebuild/postclean etc..) in favor of stacking together commands with && where necessary
avoid building send-request in the app dx
use --prefix packages/name for root directory dx rather than offering individual package dx

I think there's a bigger team discussion to be had here around what we want out of our package.json scripts, as well as duplication of commands in eg CI files, before making any changes like these.

Agreed :handshake:

There are also some fairly bad practices being introduced here, eg relying on ts-node being installed globally, and directly referencing node_modules/.bin instead of using npx.

I'd considered using npx rather than node_modules/.bin thanks for the encouragement!
I'd argue relying on a top-level package like ts-node is a good workspace practice that allows us to avoid needing a duplicated devDep.
Unfortunately we have some big boulders to move to get full workspaces going with plugins, packages and the inso abstraction. Although a gradual migration to workspaces seems possible from a recent spike.

The NPM hook removal (pre/post scripts) also likely need further work, as a lot of the packaging relies on it when publishing to NPM, which would need to be handled in the publish flow instead if removed or by using more specific hooks like prepublishOnly.

You're spot on with your concerns here lets to discuss further synchronously.

after discussion with @DMarby and @filfreire we are going to split this PR into a few passes

bootstrap -> build
make clean+build consistent between packages, eliminate npm hooks in packages
prefix build only scripts with CI when context is needed (ts-node ./something.js) otherwise use --prefix to set context
playwright test naming and dx need work
only one cli test needs the express server, elimate the with-mock and concurrently

Closing spike, Thanks @DMarby and @filfreire for your thoughts.


Make screen transition smooth on android

Currently showing background when the user is changing screen on android app.

#62


Issues upgrading to 0.17.0 - Explore migrations not functioning as expected

Superset version
Upgrade from 0.15.4 to 0.17.0
Expected results
Migration from explore v1 to explore v2, introduces nice new features but core functionality remains the same.
Actual results

Time filters, field choices, where clauses, having clauses and format arguments all migrate fine.
Filters don't survive when first opened in explore v2, resulting in no filters being applied, these all have to be recreated.
Within the NVD3 charts, most checkboxes migrate in the opposite setting than they should. e.g. Contribution being unchecked before becomes checked, likewise with reducing X ticks and the other checkbox-like fields.

Steps to reproduce
Take productions database running 0.15.4 with slices which have filiters and checkboxes applied and then upgrade to 0.17.0

Wait how did that work for us?!?!?!

Running tests...

I started with a fresh DB, pointed to 0.15.4, upgraded the db and loaded examples, moved to 0.17.0 and upgraded again, and things look good.
Is it possible that you've been pointing to [bad versions of] master on this particular database?

Some of our slices are quite old so it's possible that it's specific to our data structures. Here's an example of the params on one of our slices generated on or before 0.15.4, but which renders wrong on 0.17.0. In this case specifically the contribution flag.
{
    "add_to_dash": "false",
    "bar_stacked": "y",
    "bottom_margin": "50",
    "collapsed_fieldsets": "",
    "contribution": "false",
    "datasource_id": "41",
    "datasource_name": "shipment_package",
    "datasource_type": "table",
    "flt_col_0": "courier",
    "flt_eq_0": "",
    "flt_op_0": "in",
    "goto_dash": "false",
    "granularity_sqla": "dispatch_date",
    "groupby": [
        "courier",
        "patched_courier_delivery_service"
    ],
    "having": "",
    "limit": "0",
    "line_interpolation": "linear",
    "metrics": [
        "count"
    ],
    "new_dashboard_name": "",
    "new_slice_name": "",
    "num_period_compare": "",
    "period_ratio_type": "growth",
    "rdo_save": "overwrite",
    "reduce_x_ticks": "false",
    "resample_fillmethod": "",
    "resample_how": "",
    "resample_rule": "",
    "rich_tooltip": "y",
    "rolling_periods": "",
    "rolling_type": "None",
    "save_to_dashboard_id": "",
    "show_bar_value": "false",
    "show_brush": "false",
    "show_controls": "false",
    "show_legend": "y",
    "since": "45 days ago",
    "slice_id": "243",
    "slice_name": "Packages by Courier Service",
    "time_compare": "",
    "time_grain_sqla": "day",
    "timeseries_limit_metric": "",
    "until": "now",
    "userid": "1",
    "viz_type": "bar",
    "where": "",
    "x_axis_format": "smart_date",
    "x_axis_label": "",
    "x_axis_showminmax": "y",
    "y_axis_format": ",",
    "y_axis_label": "",
    "y_axis_zero": "false",
    "y_log_scale": "false"
}

post migration to 0.17.0 the params field remains the same:
{
    "add_to_dash": "false",
    "bar_stacked": "y",
    "bottom_margin": "50",
    "collapsed_fieldsets": "",
    "contribution": "false",
    "datasource_id": "41",
    "datasource_name": "shipment_package",
    "datasource_type": "table",
    "flt_col_0": "courier",
    "flt_eq_0": "",
    "flt_op_0": "in",
    "goto_dash": "false",
    "granularity_sqla": "dispatch_date",
    "groupby": [
        "courier",
        "patched_courier_delivery_service"
    ],
    "having": "",
    "limit": "0",
    "line_interpolation": "linear",
    "metrics": [
        "count"
    ],
    "new_dashboard_name": "",
    "new_slice_name": "",
    "num_period_compare": "",
    "period_ratio_type": "growth",
    "rdo_save": "overwrite",
    "reduce_x_ticks": "false",
    "resample_fillmethod": "",
    "resample_how": "",
    "resample_rule": "",
    "rich_tooltip": "y",
    "rolling_periods": "",
    "rolling_type": "None",
    "save_to_dashboard_id": "",
    "show_bar_value": "false",
    "show_brush": "false",
    "show_controls": "false",
    "show_legend": "y",
    "since": "45 days ago",
    "slice_id": "243",
    "slice_name": "Packages by Courier Service",
    "time_compare": "",
    "time_grain_sqla": "day",
    "timeseries_limit_metric": "",
    "until": "now",
    "userid": "1",
    "viz_type": "bar",
    "where": "",
    "x_axis_format": "smart_date",
    "x_axis_label": "",
    "x_axis_showminmax": "y",
    "y_axis_format": ",",
    "y_axis_label": "",
    "y_axis_zero": "false",
    "y_log_scale": "false"
}

but when we open that slice in the explore view, these are the options which are applied, and you can see that contribution and a load of the other options are checked rather than using the false property within the parameters:

As an example for filters, here's another one where I've trimmed it down to just he filter fields:
as at 0.15.4:
...
    "flt_col_0": "template",
    "flt_col_1": "template",
    "flt_eq_0": "",
    "flt_eq_1": "delivery-impending-ongoing,delivery-shipping-soon",
    "flt_op_0": "in",
    "flt_op_1": "in",
    "goto_dash": "false",
    "granularity_sqla": "date",
    "groupby": [
        "template"
    ],
   ...

post migration to 0.17.0 (unchanged)
...
    "flt_col_0": "template",
    "flt_col_1": "template",
    "flt_eq_0": "",
    "flt_eq_1": "delivery-impending-ongoing,delivery-shipping-soon",
    "flt_op_0": "in",
    "flt_op_1": "in",
    "goto_dash": "false",
    "granularity_sqla": "date",
    "groupby": [
        "template"
    ],
...

as rendered in 0.17.0:

It doesn't seem to be as simple as malformed JSON because the groupby argument migrates just fine. Any ideas?

Found the issue.
Migrations hadn't run properly. You can tell from the fact that the params field for the slice before and after migrations hadn't changed. As part of the migrations all of the JSON blobs are upgraded rather than it being performed on read.
Running migrations manually worked a charm - the reason they didn't run is related to our own deployment methods and not related to superset. Good work on this update guys - I love it.

@alanmcruickshank
Could you share how to upgrade caravel to superset.
db upgrade does not take effect. the table structure not changed from caravel's to superset's


SeriesData bins should be ArrayLike, not SparseArray

Rejected. SeriesData.merge relies on being aware of the "default" value of the SparseArray, and the SparseArray structure is meant to expose information about how sparse the bins are (via density()).
So it's important that it's a SparseArray.


a new strategy to triple the write speed of LevelDB

LevelDB 's write throughput is very poor when the DB grew large(suppose 500GB). It hurts me in my write heavy application.
In a 7 level db, to save a record should take 1 write to log, 1write to level0, about 11 writes to move up a level, that is 66 writes for it to reach level6(about 90% records is in this level).
I have tried another level strategy to arrange data. It will only need about 15 writes. Is anyone interested?
I test it on an aliyun ECS, here is the write result:
mergedb: 853.131 micros/op;    4.6 MB/s
leveldb: 1913.242 micros/op;    2.0 MB/s

In my strategy, size of n+1 level's data is only about twice the size of n level's data. And every compaction will compact all adjacent non-empty levels from level0 to a single level, which resulting less writes.
Suppose a LevelDB with 7 levels, there will be about 20 levels in my new db. The idea write amplification will be 1 write to log, 1 write to level0, and about 10 write to move up to level20, that is about 12 writes compared to 68 writes in original db.
I don't want to stop the write operation, so I will not compact data in the above way, then resulting several extra writes. In all, the new writes will be about 15.

How about the "read performance" in your approach?
I haven't think much about your approach, but obviously more levels will lead to "slower read".

more levels need some more bits for bloom filter to reduce miss-hit read.
I discuss my strategy on rocksdb, and found that they have an similar strategy called "universal compaction"

@yedf, any update with such issue ? will your strategy not only for improvement of write to be merged into leveldb ?  since it's a long time since leveldb not release new version .

@cmumford, any update with such issue ?

@cmumford, any update with such issue ?


App is mot moving to slot selection

Hi, App is mot moving to slot selection even after there i an availability of slots

@mrinalkapur  Please see https://github.com/yashwanthm/cowin-vaccine-booking/issues/42#issuecomment-844367428
Closing this as it's a duplicate.


linux 64 bits gnu/musl add clone_args for clone3 syscall.

r? @Amanieu
(rust-highfive has picked a reviewer for you, use r? to override)

@bors r+

:hourglass: Testing commit 6681953e0e7dcddd8025cbd06586f2e197479ac5 with merge 184712a15a0ae42349019535949612adae736a52...

:broken_heart: Test failed - checks-actions

@bors r+

:pushpin: Commit ff11bdc0c9248ebb0ba7d2c291b454224ffe998a has been approved by JohnTitor

:hourglass: Testing commit ff11bdc0c9248ebb0ba7d2c291b454224ffe998a with merge 99756b1a974d293a3c61d13f5cf4affe6d0998fc...

:sunny: Test successful - checks-actions, checks-cirrus-freebsd-12, checks-cirrus-freebsd-13
Approved by: JohnTitor
Pushing 99756b1a974d293a3c61d13f5cf4affe6d0998fc to master...


Pulsar Admin API client should support retries on operations

Is your enhancement request related to a problem? Please describe.
When autoscaling brokers up and down the namespace bundles move about.
I would like for the Admin Client to handle the retries when a namespace bundle moves and I am trying to either create a new topic or delete a topic.
As best I can tell the Pulsar Client Admin code does not support retries .
Compare Admin Client https://github.com/apache/pulsar/blob/master/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/BaseResource.java
to Pulsar Client
https://github.com/apache/pulsar/blob/04aa9e8e51869d1621a7e25402a656084eebfc09/pulsar-client/src/main/java/org/apache/pulsar/client/util/RetryUtil.java
Describe alternatives you've considered
Leave things as they are but this means that the Pulsar Client and the Admin Client are inconsistent and
then I have to write the retry code myself which I am finding complex as certain exceptions are retryable e.g. 503
but others are not e.g. Topic already exists
CC: @sijie

Bump @sijie ?

Bump

@frankjkelly Based on some discussions, the recommended approach here would be to handle the retries on the client side by leveraging the pulsar-admin's response codes.

@martijngonlag Sorry I took so long to get back to you.
The problem then becomes our code is littered with the following since some of the exceptions are not retryable and others are. And then we have to catch this further up and (using Spring) do the retries. It's an awful lot of boilerplate and particularly problematic for customers like us that create and delete a lot of topics. Especially with ZK being such a non-performant beast (and involved on the create/delete topic flow).
    private void createTopic(final URI topic) throws StreamManagerAdminException {
        try {
            val start = System.currentTimeMillis();
            pulsarAdmin.topics().createNonPartitionedTopic(topic.toString());
            val durationMs = System.currentTimeMillis() - start;
            recordDurationMetric("createTopic", durationMs);
        } catch (final PulsarAdminException e) {
            castAndRethrowException(e);
        }
    }

    private void castAndRethrowException(final PulsarAdminException e) throws StreamManagerAdminException {
        //Need to be able to distinguish between exceptions to be able to know which are retryable or not
        // However we do not want to expose Pulsar Internals beyond this layer
        if (e.getStatusCode() == 503) {
            throw new StreamManagerServiceUnavailableException(e);
        } else if (didPulsarAdminExhaustRetries(e)) {
            throw new StreamManagerRetryException(e);
        } else {
            throw new StreamManagerAdminException(e);
        }
    }

I think that it makes sense to retry some (most) of the admin operations.
We do retries while producing messages why can't we do them in the admin API ?

@eolivelli Thank you - yes that was my thinking / hope too - consistency is awesome as it reduces cognitive load and anything that helps "pave a golden path" will help with adoption 🙏 😄


Multiple Input Load Model Problem- Missing Input

Hello,
I builded two inputs and one output model. After some epoch, i saved model with below code.
model.save('weightedModel.h5').
Then, i  try to load model with below code.
model = load_model("weightedModel.h5", custom_objects={'loss': w_categorical_crossentropy(w_array) })
This is my generator:
def generate_data_generator(generator, X1, X2, Y, batch_size):
    genX1 = generator.flow(X1,Y ,seed=7, batch_size = batch_size)
    while True:
      Xi = genX1.next()
      yield [Xi[0],X2[:Xi[0].shape[0],:]], Xi[1]

After calling load_model, above generate_Data_generator functions gives error :

"ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 array(s), but instead got the following list of 2 arrays:"

When i changed to  last lines of generate_data_generator function to yield [Xi[0],Xi[1]], it doesn't give any error. Problem is that  my defined model has two inputs, but after loading model, it works only one input. What is the reason of this?
Thanks in advance.

@e181978 Did you manage to fix the problem?


Add Memory Game by @matiasvillaverde

[x] Project URL: https://github.com/matiasvillaverde/mobile-ios-vipergame
[x] Update contents.json instead of README
[x] One project per pull request
[x] Avoid iOS or open-source in description as it is assumed
[x] Use this commit title format if applicable: Add app-name by @github-username
[x] Use approved format for your entry

@matiasvillaverde thank you!


Upgrade mocha to 1.21.5

Mocha has fixed a bunch of important bugs since 1.18 so upgrade it.

An NPM release after merging this would be great too! :)

Thanks! Just keep in mind that grunt-mocha only uses Mocha for reporters.

published


Dockerfile improvement suggestion (multistage build) + docker-compose

Hello,
I'd like to suggest improvements for the Dockerfile.
 Note that I have added nano, $EDITOR and $LANG environment so the mirrorbits edit command works well (vim can be used instead, but is bigger).
But an even smaller image can be produced by using Alpine Linux base.
FROM golang:alpine as build

LABEL<EMAIL_ADDRESS>
ADD . /go/mirrorbits

RUN apk --update add \
    git pkgconf build-base zlib-dev protobuf libprotoc

RUN go get -u github.com/maxmind/geoipupdate2/cmd/geoipupdate && \
    go install -ldflags "-X main.defaultConfigFile=/etc/GeoIP.conf -X main.defaultDatabaseDirectory=/usr/share/GeoIP" github.com/maxmind/geoipupdate2/cmd/geoipupdate

RUN cd /go/mirrorbits && \
    make install PREFIX=/usr

FROM alpine:latest

# Set your own GeoIP settings
ENV GEOIP_ACCOUNT_ID 000000
ENV GEOIP_LICENSE_KEY<PHONE_NUMBER>000000

RUN mkdir /usr/share/mirrorbits /var/log/mirrorbits /srv/repo

COPY --from=build /usr/bin/mirrorbits /usr/bin/mirrorbits
COPY --from=build /usr/share/mirrorbits/base.html /usr/share/mirrorbits/base.html
COPY --from=build /usr/share/mirrorbits/mirrorlist.html /usr/share/mirrorbits/mirrorlist.html
COPY --from=build /usr/share/mirrorbits/mirrorstats.html /usr/share/mirrorbits/mirrorstats.html
COPY --from=build /go/bin/geoipupdate /usr/bin/geoipupdate
COPY --from=build /go/mirrorbits/contrib/docker/mirrorbits.conf /etc/mirrorbits.conf

RUN apk --update --no-cache add \
    rsync ca-certificates tzdata

RUN echo -e "AccountID ${GEOIP_ACCOUNT_ID}\nLicenseKey ${GEOIP_LICENSE_KEY}\nEditionIDs GeoLite2-City GeoLite2-Country GeoLite2-ASN" > /etc/GeoIP.conf && \
    mkdir /usr/share/GeoIP && \
    /usr/bin/geoipupdate

ENV EDITOR=/usr/bin/vi

ENTRYPOINT /usr/bin/mirrorbits daemon -config /etc/mirrorbits.conf

EXPOSE 8080

The resulting image size is 108.1MB
I'm also using a very simple docker-compose.yml file
version: '2'

services:
  redis:
    image: 'redis:alpine'
    restart: always
    volumes:
      - mirrorbits_db/:/data/
  app:
    restart: always
    build: 'mirrorbits'
    depends_on:
      - redis
    ports:1
      - 8080:8080
    volumes:
      - /var/lib/openmandriva/abf-downloads/:/srv/repo/
      - /root/mirrors/mirrorbits/mirrorbits.conf:/etc/mirrorbits.conf
volumes:
  mirrorbits_db: {}

Then with
docker exec -it mirrorbits_app_1 ash
(or bash instead of ash if using debian base)
we can use all mirrobits commands (with "mirrorbits edit" command working)
edit: added tzdata needed by ftp to work correctly.

If it can help, we build and maintain a mirrorbits docker image on the Jenkins infrastructure project here, and our image is available on DockerHub
We manage the GeoIP database on the helm chart here

If it can help, we build and maintain a mirrorbits docker image on the Jenkins infrastructure project here, and our image is available on DockerHub
We manage the GeoIP database on the helm chart here

It's great @olblak , thanks for the info. Are you the official maintainer of Mirrorbits now?

It's great @olblak , thanks for the info. Are you the official maintainer of Mirrorbits now?

If it can help, we build and maintain a mirrorbits docker image on the Jenkins infrastructure project here, and our image is available on DockerHub
We manage the GeoIP database on the helm chart here

Absolutely not, I made the effort several months ago to move to mirrorbits on the Jenkins project and since it's an open infrastructure project, everything is public. So I have examples to share.
I am also behind  https://hub.docker.com/orgs/mirrorbits but we DockerHub policy change, it's not really useful anymore
IMHO not installing the GeoIP database in the docker image is better since it needs to be updated on a regular basis

If it can help, we build and maintain a mirrorbits docker image on the Jenkins infrastructure project here, and our image is available on DockerHub
We manage the GeoIP database on the helm chart here

Absolutely not, I made the effort several months ago to move to mirrorbits on the Jenkins project and since it's an open infrastructure project, everything is public. So I have examples to share.
I am also behind  https://hub.docker.com/orgs/mirrorbits but we DockerHub policy change, it's not really useful anymore
IMHO not installing the GeoIP database in the docker image is better since it needs to be updated on a regular basis

Thanks for the informations, I agree with the fact geoip should be updated regularly, and having it in the docker image, unless some trick with a volume and a scheduler, is not a good idea.
What DockerHub policy change was a problem?

Thanks for the informations, I agree with the fact geoip should be updated regularly, and having it in the docker image, unless some trick with a volume and a scheduler, is not a good idea.
What DockerHub policy change was a problem?

DockerHub put in place strong limitations on its free tier. So either you pay for it or you apply to the OSS program as explained here

DockerHub put in place strong limitations on its free tier. So either you pay for it or you apply to the OSS program as explained here

Please share patches :D


[Enh]: Support for anti-join

We would like to learn about your use case. For example, if this feature is needed  to adopt Narwhals in an open source project, could you please enter the link to it below?
No response
Please describe the purpose of the new feature or describe the problem to solve.
Hands down one of my favorite feature 😂
The fact that polars has native support is really convenient
Suggest a solution if possible.
It should look something like the following:
(df
.merge(other, how='outer', indicator = True)
.loc[lambda t: t["_merge"]=="left_only"]
.drop("_merge", axis=1)
)

We can decide to name the indicator column with another non-default value
If you have tried alternatives, please describe them below.
No response
Additional information that may help us understand your needs.
No response

yeesssss nice one


Get form list

Allow users to customise get_form_list by ensuring it is used everywhere.
This cherry picks the commit from this PR (to preserve attribution)
https://github.com/jazzband/django-formtools/pull/62
And adds a regression test.

Thanks a lot for finishing this!!


Poc extensions api product registration

Fixes #8311

adding new procedure/api for extensions product registration

This one needs some TLC. Currently targeted at a released branch release-2.7.2

Let's close this, since this is purely a POC exercise and record will be kept for future analysis


Time zone drop down is too narrow to use

Expected behavior and actual behavior:
I would expect that the time zone dropdown would be wide enough to actually read the values in order to pick one. As it is, the dropdown is so narrow in some situations that you can't read anything past "GMT" meaning you have no idea which timezone you're picking.
Screenshots:

At the moment the dropdown value displayed automatically resize, and the dropdown content is this size + X. When the displayed value is too small we end up in that case where content can't be read.
Maybe we should add a property allowing the content to adjust its width automatically based on what's inside? Happy to hear about design suggestions

Yeah, I figured it was based on the display value, and likely a quirk of px-dropdown rather than px-rangepicker specifically. I would think we would do width auto, based on the contents, but maybe specify a max width so it doesn't run off the screen or something.

Closing, as this is more an issue with dropdown than rangepicker.


Dfam Browse download

Additional context
Dear developers,
I'm working with the gff3 of the C. elegans annotation in worm base parasite,
I got the rows of repetitive elements that look like this:
I       RepeatMasker    repeat_region   1622    1744    431     +       .       Target=LONGPAL1 136 261 +
I would like to get the annotation of these repeats at the class or family level using the "target" information.
I noticed that most of the targets have a hit in Dfam Browse however my gff3 has 116933 rows, Is it possible to download the table of Dfam browse?

I noticed that most of the targets have a hit in Dfam Browse however my gff3 has 116933 rows, Is it possible to download the table of Dfam browse?

It is! If you choose the Caenorhabditis elegans taxon and check both "Ancestors" and "Descendants", you will see the families RepeatMasker would have used in that search: https://www.dfam.org/browse?clade=6239&clade_ancestors=true&clade_descendants=true.
It sounds like you are mainly interested in the classification, which is included in the HMM and EMBL files. At this time we do not have an option to download in a tabular format like TSV, so I formatted and attached one here using RepeatMasker's class/subclass nomenclature:
celegans_classes.txt
For future reference, we also provide some other options for accessing Dfam data, including the HTTP API and the https://github.com/Dfam-consortium/FamDB/ tool and data files (Dfam.h5).

Closing due to inactivity. Please feel free to re-open or open a new issue if you have more questions!


dnsminder won't work on ubuntu 24.04

I can't get dnsminder to run on Ubuntu 24.04, nor can I uninstall it.  From what I can tell it appears to be a change in the Ubuntu 24.04 DNS resolver.

Update it appears the problems are related to resolvconf not being a service on Ubuntu 24.04.

@prascher Thank you for the report. UbuntuCE's repos and packages are currently only established for Ubuntu 22.04. Once Ubuntu 24.04 hits it's first point release we will begin the process of creating the new iso and repos for the next major release of UbuntuCE.
Thanks again


⚠ Spryr has degraded performance

In 6b6383f, Spryr (https://spryr.com) experienced degraded performance:

HTTP code: 200
Response time: 6743 ms

Resolved: Spryr performance has improved in b4ba353 after 7 minutes.


Media files marked as uncompressed cannot be played

We recently discovered this issue, when we started using a new packager to create EPUB3 books. As we know, all EPUB3 books are ZIP files that contain a number of resources, including media files. Up to this point, media files were "compressed" into the ZIP file, even though the "compression" did not result in any gain, because the vast majority of media formats (like MPEG-4, H.264, etc...) are not only highly compressed, but also take advantage of the specific media to compress it even further.
Because of that, recently we adopted a packager that would mark those resources as "uncompressed", because as we just said, it does not make sense to "compress" media files. And then we discovered the following problem. For uncompressed entries, the libzip code follows a different code path, which currently has a bug when seek is done. That bug is such that the libzip code loses track of the right position in the file. As a result, media files marked as "uncompressed" cannot be played by the Readium SDK, through the Launcher.

So, I spent quite some trying to make the latest libzip code work with the Readium SDK. A brief explanation about this: libzip is an Open Source project. More than two years ago, Jim Dovey took a snapshot of the libzip code as it was at the time, and put it into the Readium SDK itself, which started using it to deal with the EPUB3 files (since they are basically ZIP files). Since them, the libzip Open Source project kept evolving, and recently it actually reached version 1.0. The problem, as I find out, is that some of the features that are being used by the Readium SDK are no longer available in libzip. They have been replaced by more sophisticated features, that would require quite some code change in the Readium SDK to make them work. I can testify that it is not easy: I spent quite some time over that and still more changes were needed before I could get anything to work at all.
Unfortunately, I do not have the needed time to make the new version of libzip work. On the other hand, we must fix the uncompressed media playback issue. Fortunately, I had made a fix beforehand in the libzip code in Readium SDK, that takes care of this issue. I made that fix in a separate feature branch, and I created a pull request to have that feature branch integrated into develop.
We can take this fix for now and, later, somebody that has the necessary time can try to update the libzip code in the Readium SDK.
The pull request is at:
https://github.com/readium/readium-sdk/pull/179

I commented in the PR (#179) about @DennisPopovDn 's feature branch (updated zip lib with support for large files). Let's try to figure out what the current status of that work is (what version of the zip lib is used, and whether this fork includes significant or trivial changes compared to the original).

I took a look at the changes that Dennis Popov made. He reached the same problem that I did, i.e., that _bytes_left is no longer accessible for us to use as a way to keep track of the read position in a given file. He then tried a different solution: he wrote external code to keep track of the cursor itself. He makes a call to zip_source_stat() when the ZIP file is opened, to find out the total size of the file. Then, at each read operation, he diminishes the value of its own bytes_left, to keep track of the cursor.
I see two issues with that approach. The first is just my fear (which it could be unfounded) that somehow our code would lose sync with the libzip code, so that the cursor that our code is keeping itself would lose track with the actual position kept by libzip code. That is something that we can verify, though.
The biggest issue is the second one. Dennis gave up on trying to make his solution work with Seek. He #ifdef'ed that piece of code out and added the following comment:
#if 0   // this functionality is disabled due to macro incompatibility with the latest libzip
The problem is that we need the Seek functionality: the media playback needs it. Therefore, unless we modify Dennis' change to also work with Seek, we cannot take his change.
I also found out that I cannot compile Dennis' change in Xcode. I wonder if he was looking at the issue with libzip solely from a Windows point of view.
The bottom line is that there is still quite work to be done before we can take Dennis' change into the Readium SDK.

Thank you Nelson, this helps a lot.

@nleme , @danielweck , I've started fixing 'Seek' method support

@DennisPopovDn, this sounds great!

@DennisPopovDn - do you have any idea how you are fixing seek?
@danielweck - I found another way around this. I got the latest version of libzip, and I discovered this: libzip actually works perfect with seek and tell if you remove the CRC computation when reading a file. This is not a big deal because in the current code we are not using CRC anyway, therefore we are not worse than before. On the other hand, we can take the latest code for libzip and use seek and tell fine. I already write the code for that, and so far it is working great. :-)
I can talk about this more on tomorrow's Readium SDK meeting. It is a very complex issue, that probably will work better if I talk to you guys. I'm writing an e-mail to Ric to let him know that tomorrow's (06/24/2015) meeting may be long.

@nleme, this is great news that you found another way around this!  It seems like this is a kind of combination of the old zip_fseek.c file with the new zip_source_seek.c code? And in this case your approach may be  better than my more complicated one...
My idea was to replace byte_stream:844 zip_fseek(....) call with the new one
zip_source_seek.c:39
ZIP_EXTERN int zip_source_seek(zip_source_t *src, zip_int64_t offset, int whence)

because the latest libzip sources appear to be not having  the old function
zip_fseek.c:58
ZIP_EXTERN int zip_fseek(struct zip_file *zf, long pos, int whence)

and there is no zip_fseek.c file at all in the latest libzip source directory (at least I didn't find that in my latest sources)
So the idea was to adapt the zip_source_seek function for calling from  byte_stream:844 line instead the old one.
Also I'm taking into account your point about not to duplicate bytes_left variable if possible.
Please let me know if I missed something here...Thanks

@DennisPopovDn - unfortunately, it turned out to be too good to be true. I had started my tests with uncompressed content, since that is where the issue that triggered this whole investigation started (just see the title of this issue). However, once I moved forward to test content that was compressed, I found out that my solution would not work. :-( Let me explain why.
Let me refresh this for the people who were not at the last Readium SDK meeting. The current version of libzip is based upon the concept of "layered sources". You can interpret that as being a way to implement the "chain of responsibility" design pattern in pure C, without object orientation support. In it, each "layer" would behave like an object in the chain of responsibility, and each "layer" has a function associated to it that does the work that corresponds to each layer's purpose.
So, you start first with a raw file layer, which reads the raw file. Then, on top of that layer you add the "window" layer, which just keeps track of which ZIP entry in the ZIP file is currently being dealt with. This layer supports Seek and Tell, and things stop here when your content is not compressed. That corresponds to what I was seeing, and why this was working so fine.
However, things get messed up once you deal with compressed content. When that happens, libzip adds another layer on top of the "window" layer": it is the "deflate" layer. That layer will deal with decompressing the content. You can see it in the file zip_source_deflate.c file. It will use the deflate_decompress() function to deal with all the possible operations on the file. But, it explicitly rejects the Seek and Tell operations.
The end result is that, if you try to perform a Seek or Tell operation, it will simply return an error code.
Because of that, you cannot call zip_source_seek() on content that has been compressed, because it will just return an error code.
Currently, I'm stuck. The creators of libzip have gone out of their way to not support Seek and Tell. I already wrote to the libzip distribution list to see what is their take on this (I'll put here their answer as soon as I get it).
I read that you were trying to modify the code in the ZipFileByteStream to use zip_source_seek(). Based on what I saw in my own tests, that may not work on compressed content (if you find out differently, please let me know!). Do you have any other idea about how we can make Seek work with the latest version of libzip? Thanks for any help on this, and let me know if you need any help working on this.

@nleme, the libzip discussion forum is still down and NIH hasn’t responded to our support request to fix it.  Maybe someone can suggest another way to get in touch with these guys?  In the meantime @DennisPopovDn can work on a prototype where we’ll modify the code in ZipFileByteStream to use zip_source_seek() per his comment above.  The prototype needs to support compressed and uncompressed content.  Let me know if that doesn’t make sense, otherwise we’ll start working on it this week.

DRM inside had the same problem during implementing the LCP on early this year.
The problem was in that the fseek_bytes function (in the zip_fseek.c) could not calculate the zf->fpos and zf_cbytes_left values properly for the uncompressed resource.
We have fixed this matter with simply modifying inside of  the _zip_fseek_bytes() function.

Modified code is following;
/* seeking by raw byte amounts - no compression/decompression to handle /
int _zip_fseek_bytes(struct zip_file zf, off_t abspos, off_t flen)
{
/* can't set a negative offset /
if (abspos < 0) {
_zip_error_set(&zf->error, ZIP_ER_INVAL, 0);
return -1;
}
/ CAN set offset past EOF: keeps offset, sets EOF */
else if (abspos >= flen) {
zf->flags |= ZIP_ZF_EOF;
zf->bytes_left = 0;
    /* added by DRM inside, C.H. Yu on 2015-04-13 */
    // Without following added codes, uncompressed EPUB media content would not be properly random accessed,
    // such as bad index access error
    zf->fpos = _zip_file_get_offset_safe(zf->za, zf->file_index);
    zf->fpos += flen;
    zf->cbytes_left = 0;
    /* adding end */
}
/* not at or past EOF? ensure EOF is unset and update bytes_left */
else {
    zf->flags &= ~ZIP_ZF_EOF;
    zf->bytes_left = flen - abspos;

    /* added by DRM inside, C.H. Yu on 2015-04-13 */
    // Without following added codes, uncompressed EPUB media content would not be properly random accessed,
    // such as bad index access error
    zf->fpos += (abspos - zf->file_fpos);
    zf->cbytes_left = zf->bytes_left;
    /* adding end */
}
zf->file_fpos = abspos;
return 0;

}

How does this compare to the solution in PR 179?  I think we should merge 179 if everyone agrees it is the right solution. I believe Bluefire has tested in thoroughly? This merge would resolve the issue but won't address the issue for files larger than 4 GB, but would be a good short-term solution, IMO.

Based on our latest discussion about this issue, currently I have been investigating how to bring the Seek functionality back into the latest version of libzip.
Just today we had a small piece of good news. Thomas Klausner, one of the authors of libzip, actually wrote back on a inquiry I had made some time ago. Originally, I had written about how to disable the CRC layer, given that that layer was returning that Seek was an unsupported operation. He answered back saying that we could pass a flag to disable that layer.
However, as I found out later, that is not the only obstacle. The deflate layer, the one that deals with compressed content, also blocks the Seek operation. Thomas made a small remark about that in this answer. Quote:
Currently, you can only seek in stored data (i.e., uncompressed data).
The deflate level does not support seeking.
I replied back to him thanking for answering our previous question, and then asked to him exactly about that. I.e., why compressed content does not support seeking and how hard it would be to bring it back. As soon as he answers back, I'll put his answer in this thread.

Good news: Thomas Klausner, from libzip, just answered back. He was quite nice. He confirmed what we suspected: Jim Dovey wrote the code for Seek in the old version of libzip.
Bad news: he stated that Seek was never supposed to be part of libzip in the first place, it was just added by Jim Dovey. As such, he said that they do not have plans to add that functionality back to libzip, and suggested that I ask Jim to add that functionality back to the current version of libzip.

What I meant above is that Thomas stated that it will be up to us to implement Seek in libzip for compressed content.

Another bit of information. I received this morning an e-mail from Dieter Baron, the other developer who works at libzip. Let me paste the important part of his e-mail below:

We will try to add the Seek and Tell functionalities into the deflate layer.

Please don’t. Rather, write a seeking layer and put that on top if the underlying layer doesn’t support seeking directly.
However, be aware that seeking in a compressed stream is extremely inefficient: you can only read it from the beginning. So to seek back, you have to start over from the beginning and discard everything until you get to where you want.
If the files aren’t too big, it is much better to read them into a buffer and do random access on that.
(the phrase with the ">" was part of my original mail to the libzip authors)
In a nutshell, Dieter is saying to not add the Seek functionality to the deflate layer, but instead create a brand new layer (following their implementation of "chain of responsibility" in C) which would do the Seek.
He also goes back to a point that I've seen them doing before, i.e., suggesting that we do not do Seek at all, and instead just decompress the whole thing in a buffer.
Simply decompressing the whole file in a buffer is not something that is recommendable in mobile platforms: those files could be bigger than the available memory, especially in Android devices. Then, there is the possibility of writing the decompressed file to disk. The problem, then, is that we may be dealing with encrypted files, and you would not want to save in disk a decrypted file. Therefore, to be able to do this, we would have to do the following:
Decrypt ZIP entry -> Decompress ZIP entry -> Encrypt ZIP entry again -> Write ZIP entry to disk
Followed by:
Read ZIP entry from disk -> Decrypt ZIP entry -> Serve bits to Cocoa HTTP Server
That would be quite a change at this point.

@nleme I agree with your assessment.  It sounds like the LibZip guys don't really understand the workflows we need to support.

@DennisPopovDn - how is your work on putting Seek back into libzip going? And how is that looking with regards to what Dieter said in his e-mail? I'm not demanding anything, especially because I'm aware that this is not an easy job, I'm just curious to know how the stuff you are doing is going to match with what Dieter said.

be aware that seeking in a compressed stream is extremely inefficient: you can only read it from the beginning. So to seek back, you have to start over from the beginning and discard everything until you get to where you want.

Correct. That's how I implemented byte stream "seek-ability" in SDKLauncher-OSX / ObjectiveC (a long time ago, before this feature was available natively in libzip), and that's how Jim Dovey implemented it as well (see _zip_fseek_to_start + _zip_fseek_by_reading):
https://github.com/readium/readium-sdk/blob/master/ePub3/ThirdParty/libzip/zip_fseek.c#L124

Regarding the performance implication of seeking from the start of the byte stream: with both my old ObjectiveC implementation and Jim's current C implementation, seeking from the current position is an optimization that works in most streaming scenarios (e.g. video playing within an ebook page from start to end, or sequential audio Media Overlays playback), see:
https://github.com/readium/readium-sdk/blob/master/ePub3/ThirdParty/libzip/zip_fseek.c#L133
(note that _zip_fseek_to_start is not invoked when the cursor is positioned ahead)
else if (abspos > zf->file_fpos) {
        // read & decompress bytes until we reach the right position
        return _zip_fseek_by_reading(zf, abspos-zf->file_fpos);
    }

To further comment on what Daniel said. We can put back the code for doing a Seek into libzip. There is the question, though, of how to do it.
If we modify the libzip code, then we can add a new layer as Dieter suggested. This way, it minimizes the changes on the existing files of libzip, which it would make easier to merge changes from libzip into our own version of libzip.
On the other hand, adding Seek into the deflate layer itself would be a larger change, and more difficult to merge with later changes into libzip.
Then, we can also add Seek as a wrapper around libzip. @danielweck, if I understood right, that is what you did way back in the day, right? Do you think that was a good approach?

@nleme, I've been meditating on the libzip (how to add it into deflate layer with caching the data, w/o caching the data etc), and agreed with Dieter that implementing it as the separated layer seem to be the better idea... in progress
BTW I'm remembering some discussion at Readium about LCP decryption and byterange request (like a Seek feature), and it had been resolved with adding a cache buffer right to the some readiumsdk class.. it seems to me the buffering resolution has already been made for encrypted files.. May be it makes sense to implement that work around for compressed files as well?

@nleme:

Then, we can also add Seek as a wrapper around libzip. @danielweck, if I understood right, that is what you did way back in the day, right? Do you think that was a good approach?

The performance loss in my OSX-ObjectiveC implementation was probably due to creating / deleting ByteStream instances, which were needed in order to reset the zip file position to zero (equivalent to _zip_fseek_to_start  in the zip lib logic, see https://github.com/readium/readium-sdk/issues/177#issuecomment-120085546 ) when HTTP-requesting a byte range starting before the current stream cursor. Perhaps also because of the maintenance of intermediary byte buffers (I cannot remember exactly, but I'm pretty sure some "middle man" buffering mechanism was indeed needed).
Anyway, I guess here the proposal would be to implement the seek/tell functionality as "close" as possible to the native libzip code (very thin wrapper, no overhead, and crucially: no need for JNI byte buffer conversions in Android).

@DennisPopovDn - you are correct: we fixed one issue that we had around non-byte-range requests in the Readium SDK by making a cache. That fixed some big problems that we had. However, I don't think we can apply the same solution to the byte-range request case because of the mobile platforms. In that case, the problem is that you may have a really big media file that, if loaded in a buffer in memory, could possibly completely fill out the memory of the device (or not even fit) especially in some of the less powerful Android devices.
@danielweck - I see what you are saying. Very thin wrapper outside of the libzip code, just to keep track of the reading position (Tell) and to perform the required read operations (Seek). That is something that we can try.

Is this issue now fixed? See already-merged PR from DRM-Inside:
https://github.com/readium/readium-sdk/pull/219/files#diff-e8c489d80c21c04a0f36514e70875fd3

@danielweck - With merged codes from DRM inside's PR, I think this issue is fixed.
Because the reason why DRM inside modified the zip_fseek.c was that we had the same problem with what  @nleme mentioned in the issue #177.
We had verified the validity of the modified code with Readium Launcher and several uncompressed media resources in the zip file.

I think this issue can be closed, as well as @nleme 's associated PR (superseded by @drminside 's merged PR).
See:
https://github.com/readium/readium-sdk/pull/179
https://github.com/readium/readium-sdk/pull/219/files#diff-e8c489d80c21c04a0f36514e70875fd3


Improve spot HA by utilising ASG capacity rebalance

Capacity rebalance helps by being proactive in trying to replace Spot Instances before they are interrupted. Full docs - https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-capacity-rebalancing.html

Current Behaviour
When a spot instance receives its 2-minute interruption warning nothing happens, the instance is terminated after 2 minutes then a new instance is started after the original is terminated.
New Behaviour
When a spot instance receives its 2-minute interruption warning the ASG immediately provisions a new instance which hopefully will boot and move the floating EIP before the original is terminated. With this approach, there is minimal downtime when spot instances are terminated.

You can test this out using AWS Fault Injection, if you go to the EC2 management console then click Spot Requests in the sidebar. You have the option to select a spot request, click actions then Initiate Interruption

Should this reduce service interruptions? Sometimes my server stops working while the fck-nat instance is being replaced.

The only thing I would change is the ha_additional_instance_types variable, would it be better to use an empty array by default? Otherwise, after an update, users would have an ASG with a behavior and an instance that they did not choose.
This pull request should fix the problems I'm having while using fck-nat. My spot instance often fails to start because there is no spot capacity available. For example these are the errors that often happen to me when using the fck-nat instance:

@GabrieleOlmi

The only thing I would change is the ha_additional_instance_types variable, would it be better to use an empty array by default? Otherwise, after an update, users would have an ASG with a behavior and an instance that they did not choose.

It's been a while since I last looked at this but IIRC capacity rebalance requires a mixed instance policy and within a mixed instance policy you need to define a minimum of 2 instance types. Adding the ha_additional_instance_types and defaulting it to the next cheapest instance type was the only sensible approach I could think of.
Defaulting ha_additional_instance_types to an empty array would cause an error if end users set capacity_rebalance = true without explicitly setting ha_additional_instance_types to their preferred failover instance.
Perhaps just some documentation to clear up this behaviour in the README would be enough.
@RaJiska it would be good to hear your thoughts on this.


YETUS-277. full report in csv format

This is a somewhat trivial implementation of a csv-style report.  I have no idea if it will work with what @busbey had in mind though...

Looks useful.

I think I'll go ahead and merge this one in and we can play with it a bit before fulling documenting it. Thanks @ndimiduk


One-click node DNS LookupSRV error

Hello! I believe I was running the one-click node solution successfully for two weeks, based on this guide.
However, Yesterday I ran out of RAM, which caused a whole slew of errors. I figured, okay I'll just stop my node, and get it back up and running again. However, I am now faced with the following error, and the node cannot sync:
{"file":"bootstrap.go","function":"github.com/algorand/go-algorand/tools/network.readFromSRV","level":"info","line":43,"msg":"ReadFromBootstrap: DNS LookupSRV failed when using system resolver: DNS error: NXDOMAIN","time":"2023-09-14T13:10:18.458582Z"}
{"file":"bootstrap.go","function":"github.com/algorand/go-algorand/tools/network.readFromSRV","level":"info","line":43,"msg":"ReadFromBootstrap: DNS LookupSRV failed when using system resolver: no signature in DNS response for _algobootstrap._tcp.mainnet.algorand.network","time":"2023-09-14T13:20:17.841736Z"}
I've read that I can change the DNS security config flag ("DNSSecurityFlags": 0) in ./data/config.json of the container, however if I change that JSON, it just gets overwritten next time I execute algorun start.
I've tried updating algorun, since they say it should be v0.1, but it looks like I might be on a different beta version right now:
PS C:\Users\Eric> algorun --version
algorun, version 0.0.1b4
PS C:\Users\Eric> pipx upgrade algorun
⚠  Overwriting file C:\Users\Eric.local\bin\algorun.exe with C:\Users\Eric.local\pipx\venvs\algorun\Scripts\algorun.exe
algorun is already at latest version 0.0.1b4 (location: C:\Users\Eric.local\pipx\venvs\algorun)
I'm really not sure what to do. I've tried uninstalling docker, and algorun, and reinstalling both again. Anyone have any ideas, or ways to debug?

I changed to a clean directory, and attempted an algorun -v start again, and that message ("Docker definition is out of date; please run algorun localnet reset") has seem to have gone away.
PS D:\Docker\algorun> algorun -v start
 found in cache C:\Users\Eric\AppData\Local\algorun\last-version-check
Could not determine latest version
Running 'docker compose version --format json' in 'D:\Docker\algorun'
docker: {"version":"v2.21.0-desktop.1"}
Running 'docker version' in 'D:\Docker\algorun'
docker: Client:
docker: Cloud integration: v1.0.35+desktop.4
docker: Version:           24.0.6
docker: API version:       1.43
docker: Go version:        go1.20.7
docker: Git commit:        ed223bc
docker: Built:             Mon Sep  4 12:32:48 2023
docker: OS/Arch:           windows/amd64
docker: Context:           default
docker:
docker: Server: Docker Desktop 4.23.0 (120376)
docker: Engine:
docker: Version:          24.0.6
docker: API version:      1.43 (minimum version 1.12)
docker: Go version:       go1.20.7
docker: Git commit:       1a79695
docker: Built:            Mon Sep  4 12:32:16 2023
docker: OS/Arch:          linux/amd64
docker: Experimental:     false
docker: containerd:
docker: Version:          1.6.22
docker: GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
docker: runc:
docker: Version:          1.1.8
docker: GitCommit:        v1.1.8-0-g82f18fe
docker: docker-init:
docker: Version:          0.19.0
docker: GitCommit:        de40ad0
Docker compose file does not exist yet; writing it out for the first time
Starting Algorand mainnet node now...
Running 'docker compose up -d' in 'D:\Docker\algorun'
docker: time="2023-09-14T10:25:19-04:00" level=warning msg="The \"PWD\" variable is not set. Defaulting to a blank string."
docker: time="2023-09-14T10:25:19-04:00" level=warning msg="The \"PWD\" variable is not set. Defaulting to a blank string."
docker: Container mainnet-container  Creating
docker: Container mainnet-container  Created
docker: Container mainnet-container  Starting
docker: Container mainnet-container  Started
Started; the node is now catching up to the latest ledger state

I used a VPN, and now the error has gone away. It looks like it is a Rogers ISP issue - are they blocking this DNS lookup?


Complement of integer literal

In slalpha4, I type:
~0

in the shell, it prints
0

which is of course wrong.  If I type
~(0)

it correctly print
-1

Happens outside the shell too:
import ballerina/io;

public function main() {
   io:println("~0 is ", ~0);
}

prints
~0 is 0

The following program  should give an error as operator '!' not defined for 'int' (currently this is compile fine)
public function main() {
    int a = !5;
}


staging to main (maintainance)

This is a maintenance release that:

Updates env vars in test runs to not be dependent on secrets
Updates DWD harvester to v1.2.3
Updates OSM harvester to v1.4.0

@raphael-arce @dnsos This is a maintenance release. Are you fine with that?


ggrastr & viridis R Packages Not Found

Hi,
We are running a cellxgeneVIP instance on AWS, and I am having a couple issues when trying to generate volcano plots in VIP DEG and trying to generate density scatter plots in VIP Density Scatter.
In DEG, the error returned:
ERROR @server: Traceback (most recent call last):
  File "/home/*****/anaconda3/envs/VIP/lib/python3.8/site-packages/server/app/VIPInterface.py", line 57, in route
    taskRes = distributeTask(data["method"])(data)
  File "/home/*****/anaconda3/envs/VIP/lib/python3.8/site-packages/server/app/VIPInterface.py", line 731, in DEG
    raise SyntaxError("in volcano.R: "+res.stderr.decode('utf-8'))
  File "<string>", line None
SyntaxError: in volcano.R: Error in library(ggrastr) : there is no package called ‘ggrastr’
Execution halted

In Density Scatter, the error returned:
ERROR @server: Traceback (most recent call last):
  File "/home/*****/anaconda3/envs/VIP/lib/python3.8/site-packages/server/app/VIPInterface.py", line 57, in route
    taskRes = distributeTask(data["method"])(data)
  File "/home/*****/anaconda3/envs/VIP/lib/python3.8/site-packages/server/app/VIPInterface.py", line 1193, in DENS2D
    raise SyntaxError("in R: "+res.stderr.decode('utf-8'))
  File "<string>", line None
SyntaxError: in R: Error in loadNamespace(x) : there is no package called ‘viridis’
Calls: loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart
Execution halted

And a screenshot of the DEG issue:

It looks like a similar issue mentioned here, but there's no information about a solution there. Any suggestions on how to install/identify the packages needed? Thanks for any help!
Cheers,
Trent

Please follow this,
https://github.com/interactivereport/cellxgene_VIP#4-install-r-packages
If you don't have admin right, please use VIP_conda_R.yml at step 2.

Got it, I was using someone else's existing VIP install but didn't realize I needed to configure my own R environment on my end. Packages were installed using Step 4 and all is good, thank you for the quick response!


Merge HOTP and TOTP algorithms into this library?

Would be a neat addition. I have made several.
For people using TOTP in C#, it would go a lot more smoothly than importing yet another library. It is small and simple and I could do it.
Thoughts?

Hi @Hydroque ,
thanks for you offer and I really appreciate it. But before doing work which has already been done, have a look at the OneTimePassword payload generator:
https://github.com/codebude/QRCoder/blob/master/QRCoder/PayloadGenerator.cs#L1472
There is already a HOTP/TOTP implementation. If you idea/implementation is different, feel free to contribute. :-)

Yes, I understand what you have. Apart from URLs, you have the actual data.
https://github.com/Hydroque/LuaOTP

Ok, you meant not only the payload's structure, but the contents. So you meant to deliver a HOTP/TOTP generator. Feel free to write one and send a merge request.


/session on negative time displays the wrong time.

Repro steps

Start new session ( 3 minute long so far )
Take a break that's longer than the session has been active ( 10 minutes )
/session

Expected Result
Active session on timby for 0 hours and -7 minutes
Actual Result
Active session on timby for 0 hours and 53 minutes
System Info

Info: the negative time is displayed correctly in the frontend overview, just not in the /session

implementation for negative session times has been implemented for /session


[SECURITY] Releases are published insecurely

CWE-829: Inclusion of Functionality from Untrusted Control Sphere
CWE-494: Download of Code Without Integrity Check
The build files indicate that this project is uploading artifacts over HTTP instead of HTTPS. Any of these artifacts could have been MITM to maliciously compromise them and infect the build artifacts that were produced. Additionally, if any of these JARs or other dependencies were compromised, any developers using these could continue to be infected past updating to fix this.
This vulnerability has a CVSS v3.0 Base Score of 8.1/10
https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?vector=AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H
This isn't just theoretical
POC code has existed since 2014 to maliciously compromise a JAR file inflight.
See:

https://max.computer/blog/how-to-take-over-the-computer-of-any-java-or-clojure-or-scala-developer/
https://github.com/mveytsman/dilettante

MITM Attacks Increasingly Common
See:

https://serverfault.com/a/153065
https://security.stackexchange.com/a/12050
Comcast continues to inject its own code into websites you visit (over HTTP)

These Vulnerabilities are Considered "Out Of Scope" by the PayPal Security Team
I originally reported this vulnerability privately to the PayPal bug bounty program but it was determined to be "Out Of Scope" by the HackerOne team that pre-filters these reports.
If this is considered a security vulnerability, I'd advise the PayPal team to communicate this to the HackerOne team.
Here is the link to my original report:
https://hackerone.com/reports/504119
Source Locations
Insecure Download
https://github.com/paypal/NNAnalytics/blob/4701020d582e002fe6982483adf86f5d99298d65/build.gradle#L136-L138
https://github.com/paypal/gimel/blob/33192730edd6741ef459fba98b84e20d4f3ff980/pom.xml#L42-L58
https://github.com/paypal/spring-social-security-paypal-openidconnect-quickstart/blob/6a57453c67e404652f966f8414943008ec2da74c/pom.xml#L203-L224
https://github.com/paypal/aurora/blob/a2021e6829fb6024eac45f4afe54e2fbc70b847c/grails-app/conf/BuildConfig.groovy#L45
WARNING! If any of these builds are using a shared or re-used ~/.gradle or ~/.m2 cache between builds and any of these downloads were maliciously compromised, the compromised jar may remain inside of cache directory and continue to be used in the future.
Insecure Upload
Passwords in this upload are being sent in plaintext and should be considered compromised!
https://github.com/paypal/digraph-parser/blob/3a947dd99e521c23da657bd56cb85f46bca56fa6/pom.xml#L21-L30
https://github.com/paypal/Gibberish-Detector-Java/blob/034bbbcce7bf5d4099e02a64362b31bddbbb29de/pom.xml#L40-L43
https://github.com/paypal/SeLion/blob/248163ce623b3d8a6c0444c677a2cedce2aec823/pom.xml#L70-L73
Fix and Public Disclosure
At a minimum, all of these code locations where artifacts are uploaded insecurely needs to be fixed. Previous releases should be rebuilt with the fix applied. The checksum of the released artifacts and artifacts built in a trusted environment should be made. If the checksums match, you can be certain that they weren't compromised.
If the checksums don't match, indicating a compromised artifact, CVE numbers need to be issued for the potentially malicious artifacts.
The ability to check if checksums match assume that these projects have reproducible builds.

Thank you for you bug report! I want to be sure that I understand - the problem in our code is that the upload url is http instead of https, right?

Correct

Fixed in this commit.

Have you also retired the credentials used to publish artifact?

I've changed my Sonatype password, the one that was used for the deployment. Is that sufficient?

It should be. You also need to perform the checksum comparisons to ensure that none of the artifacts published were compromised inflight.

Is there any plans to audit the previously published artifacts for compromise?
If not, a CVE should be issued for previously released artifacts.
Here's a similar CVE for Eclipse hawkBit that was requested by Wayne Beaton of the Eclipse Security Team due to my discovering this vulnerability in several of the Eclipse repositories.
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-10240

Ping!


Remove FLAC++-specific audio source.

FLAC support lives on via libsndfile.
Can anyone else test this?

Seems legit


V2 throws error in repo without serviceName

Describe the bug
After upgrading to V2 runs are throwing an error about serviceName being blank.

@iansu This still happening, are you planning to fix that or this action has no more support?


ModuleNotFoundError: No module named 'azure.iot' ?

Env: Ubuntu with  python 3.8
$ pip3 list  | grep -i azure
azure (4.0.0)
Symptom:
++ python3 ./simple_send_message.py
Traceback (most recent call last):
File "./simple_send_message.py", line 3, in
from azure.iot.device.aio import IoTHubDeviceClient
ModuleNotFoundError: No module named 'azure.iot'

resolved user error :)
My brain is completely scrambled with the plethora of pip and python version tried, but this seems to work
$ pip3 --version
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)
$ python -V
Python 3.8.0
$ pip3 install azure-iot-device
$ python  ./simple_send_message.py

@MikeWilcoxMicrosoft, thank you for your contribution to our open-sourced project! Please help us improve by filling out this 2-minute customer satisfaction survey

whew

from azure.iot.device.aio import IoTHubDeviceClient
ModuleNotFoundError: No module named 'azure.iot'; 'azure' is not a package


Group repository constituents are not shown.

/#/group does not show constituents for neither of the groups.

Under the current implementation, there should be a label just after Constituents saying "(mouse over to view)". When you roll the mouse over that, it should display the constituents as a tooltip. Are you saying the mouse-over doesn't work?
Looking at this further, it may not be the most intuitive way to present the information. I'll have to look into better ways to present this in a compact way (maybe an expandable panel of some sort).

Yes, hover does not show any tooltip.

I moved this to a subpanel on the listing page that will display if you hit the '+' link next to "Constituent(s)". And on the hosted repo listing page, I moved from using tooltips to help with the capabilities to using a legend in the control panel.


runOutput Data Bricks Data Factory

The runOutput examples for data bricks and data factory communication did not work for me in Azure Data Factory V2. The runOutput of the Data Bricks Notebook sometimes needs to be converted to a string before usage.
These changes appear to do the job.

@Quisl : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@ktoliver this is in conflict. Is there anyone who can resolve the conflicts from your team?

@JasonWHowell The merge conflict is resolved. Thanks.

#sign-off

@JasonWHowell: I'm sorry - only the author of this article, @nabhishek, can sign off on your changes. But we do have an exception process - if you are on the Microsoft content or product team for this product area, you can ask the PR review team to review and merge it by sending mail to the techdocprs alias.


37 implement rqspline using distrax

Implementation is done.
The compilation time of the part seems to be pretty significant (One minute for training loop and one minute for sampling). Need to knock that down

The RQspline NF model can have runaway behavior after long training, which probably has to do with points going outside the RQ spline region. I think some normalization layers should fix that.

Smaller learning rate seems to alleviate the behaviour, tho we should still take a close look to the issue

@marylou-gabrie I have merged the branch, but I will leave this PR open so you can have a look. Once you have finish checking the branch, please delete the branch.


In Chat API, setting stream to true gives a different json structure

When I implemented Chat, I didn't test what settings stream to true would do. Well, stream is very nice! But it returns data in a different way and this SDK currently doesn't support that.

@arthurgarzajr Can you provide an example of the response?

I'm facing the same issue. I'm not sure what's the response type when setting stream to true, but here's an example from both response with Postman

Dropdown Opened
Dropdown Closed

It looks like the Content-Type property changes to text-event-stream when stream is set to true. Do you guys know a way to receive this data type in Swift by any chance?

stream set to true
stream set to false

@jaysack I think we would need to add Event Source support to this framework in order to handle this use case. Would need to do more research before taking a stab at an implementation. Considering removing the stream parameter in the interim.

I've implemented streaming successfully using vanilla UrlSession async/await on a personal project. I've never contributed to open source before but I'd be happy to see if I can contribute.

@cgaaf More than happy for you to take a stab at it!

@ronaldmannak https://github.com/dylanshine/openai-kit/pull/36 adds stream support, curious if I could get your thoughts before merging

@dylanshine oh wow, with 19 changed files it looks like a bigger change than I thought. Let me use the branch and see how it works.

@ronaldmannak A lot of it was replacing HTTPClient.Body in favor of Data to be more flexible

PR merged...have fun


Fixing wrong link

Why did nobody notice that?

Idk, probably because not many people see the readme.


Update qos_policy_config_test.go

cisco-dp-1.2

Pull Request Functional Test Report for #1686 / f889f9cdbdd391516ccc319020e54206dda14448
Virtual Devices

Device
Test
Test Documentation
Job

Arista cEOS

DP-1.2: QoS policy feature config

Cisco 8000E

DP-1.2: QoS policy feature config

Cisco XRd

DP-1.2: QoS policy feature config

Juniper cPTX

DP-1.2: QoS policy feature config

Nokia SR Linux

DP-1.2: QoS policy feature config

Can you please add more details in the description?

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
23 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.05%) to 47.896%

Files with Coverage Reduction
New Missed Lines
%

topologies/binding/options.go
23
41.82%

Totals

Change from base Build<PHONE_NUMBER>:
-0.05%

Covered Lines:
1320

Relevant Lines:
2756

💛  - Coveralls

b/285610043 has the internal test results.


fix: Fix false positives related to generators

Description:

Postponed due to https://github.com/dudykr/stc/pull/624


Assessment export property lists not complete and not in the right case

All properties on the main page and each specific assessment export page with the API calls are shown in the wrong case (upper vs. lowercase). The JSON response returns properties like "configurationCategory" and "osPlatform" rather than "ConfigurationCategory" and "OSPlatform". The file exports; however, do appear to use the case shown (starting with uppercase letter).
The list under the "Export secure configurations assessment" "Properties (JSON response)" section does not include "osVersion" and "rbacGroupId"; however, they appear in the real output of the API call. This is true for both the main assessment page and the specific page for "Export secure configuration assessment" (https://docs.microsoft.com/en-us/microsoft-365/security/defender-endpoint/get-assessment-secure-config?view=o365-worldwide).
The list under the "Export software inventory assessment" "Properties (JSON response)" section does not include "rbacGroupId"; however, it appears in the real output of the API call. This is true for both the main assessment page and the specific page for "Export software inventory assessment" (https://docs.microsoft.com/en-us/microsoft-365/security/defender-endpoint/get-assessment-software-inventory?view=o365-worldwide).
The list under the "Export software vulnerabilities assessment" "Properties (JSON response)" and "Properties (delta export JSON response)" sections do not include "rbacGroupId"; however, it appears in the real output of the API calls. This is true for both the main assessment page and the specific page for "Export software vulnerabilities assessment" (https://docs.microsoft.com/en-us/microsoft-365/security/defender-endpoint/get-assessment-software-vulnerabilities?view=o365-worldwide).

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 57b8d46d-5e90-4b8b-c9c7-4beb77bb93ca
Version Independent ID: 821c862e-673d-2b67-d055-d4aa86f4a1fb
Content: Export assessment methods and properties per device
Content Source: microsoft-365/security/defender-endpoint/get-assessment-methods-properties.md
Product: m365-security
Technology: mde
GitHub Login: @jweston-1
Microsoft Alias: v-jweston

An additional item, in the "Export software inventory assessment" properties, it shows "id" as a valid field. However, this field does not appear to be in the actual output of the API call.

@MastaDisasta Thank you for your feedback. We have made changes to the document via PR#https://github.com/MicrosoftDocs/microsoft-365-docs/pull/6035
Once the author approves, the changes will be live.
Thanks
Sri

@msbemba Thank you very much for the contribution and updating the documentation with PR. @MastaDisasta Hope this update is helpful for you. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


Add pineapple seed to --farming

Recommendation/bug maybe idk
Pineapple seed has never been on the b --farming tab though it is a farmable fruit tree seed.
Both osb and bso

Done in next update.


Upgrade linux RBE tests to bazel 0.22

Upgrade bazel version used for linux RBE tests.

Known failures:
https://github.com/grpc/grpc/issues/18608
https://github.com/grpc/grpc/issues/18111


DASHBUILDE-98: Static data set definition does not open in data set editor

Fix + test
@romartin Can you review & merge please?

+1

Closed: 16c7d30395357b5ee9e5decb5e430b8a541f184e


Added the ability to crawl content links

Hi, I've added a deep web crawler that extracts links as an extension to this article.

withoutSelector crawls internal external links without a selector.
attributeWhitelist allows you to optimize performance by whitelisting HTML attributes to be preserved.
isContentLink determines whether to enable crawling internal external links.

I hope you find this PR useful. If you have any questions, please feel free to reach out to me.

Preparing review...

PR Analysis

🎯 Main theme: Added the ability to crawl content links
📝 PR summary: This PR adds functionality to crawl content links as an extension to the existing article. It introduces new options to crawl internal and external links, optimize performance by whitelisting HTML attributes, and determine whether to enable crawling internal and external links.
📌 Type of PR: Enhancement
🧪 Relevant tests added: False
⏱ Estimated effort to review [1-5]: 3, because the PR introduces new functionality and requires reviewing the added code and logic.
🔒 Security concerns: No security concerns found

PR Feedback

💡 General suggestions: - It would be helpful to provide more context in the PR description about the purpose and use cases of the new options and how they can be configured.

Consider adding some inline comments to explain the purpose and functionality of the new code.

Ensure that the code follows consistent formatting and indentation throughout the file.

 🤖 Code feedback:

relevant file: src/main.ts
suggestion: Consider extracting the logic for handling external links into a separate function for better modularity and readability. [important]
relevant line: async function handleExternalLinks(

relevant file: src/main.ts
suggestion: Add error handling for any potential exceptions that may occur during the processing of external links. [medium]
relevant line: } catch (error) {

relevant file: src/main.ts
suggestion: Consider using a more descriptive variable name instead of 'jsonData' to improve code readability. [medium]
relevant line: const jsonData = {

relevant file: src/main.ts
suggestion: Add input validation and error handling for the new options to ensure that they are properly configured. [medium]
relevant line: withoutSelector: string;

Preparing review...

thanks for contributing @994AK!
my main question - how is this different than the current enqueueLinks code we have?
Aka the crawler already finds links and crawls them, so I'm interested in understanding what you find to be missing and what required you to create (from what I can tell) an additional implementation of what already existed
Perhaps sharing your use case would help contextualize this?

I'm glad to see your reply, @steve8708.
Feature Introduction
My feature is primarily designed for "when scraping this content, if there are external links within the content, it allows the crawler to further access these external links, generate a JSON file to store them, and expand the content."

isContentLink - Whether to enable it. Access external links within the content and add additional information.
withoutSelector - Like external links. Choose the area that is not fixed, typically in terms of specifications, the main content is the most important.
attributeWhitelist - Preserve some HTML attributes. I didn't remove the HTML structure because when scraping, I want GPT to understand the structure for expanding queries and reading comprehension.

Use Case Example
export const config: Config = {
  url: "https://www.builder.io/c/docs/developers",
  match: "https://www.builder.io/c/docs/**",
  selector: `.docs-builder-container`,
  isContentLink: true,
  withoutSelector: `main`,
  attributeWhitelist: ["href", "title"],
  maxPagesToCrawl: 50,
  outputFileName: "output.json",
};

When isContentLink is enabled, it first accesses https://www.builder.io/c/docs/developers to retrieve the initial page and fetch the links within it.
The content within the withoutSelector selector is scraped and saved.
This material can then be saved for users as extended information.
I am currently out of town and unable to use a computer, so I apologize for any inconvenience.
What do you think of this feature idea? Looking forward to your response.

I think I can refactor the code to make it easier to understand and make the functions more complete. @steve8708
Thank you for the suggestions, enqueueLinks does indeed seem to work much better than our custom implementation.
Regarding preserving HTML, I understand your point, and it could be useful for users to experiment with. Additionally, your idea of removing empty non-semantic tags is a good one, as it can help save unnecessary bytes.
Considering your suggestions, I have thought about making some adjustments to the configuration, as follows:
  url: "https://www.builder.io/c/docs/developers",
  match: "https://www.builder.io/c/docs/**",
  selector: `.docs-builder-container`,
  crawlExternalPages: false,
  includeHtml: true, // Set includeHtml to true to preserve HTML
  htmlAttributeWhitelist: ['title', 'href'], // Only needed if includeHtml is true
  maxPagesToCrawl: 50,
  outputFileName: "output.json",
};

withoutSelector
As for 'withoutSelector,' this option is used to specify an HTML selector, and if a page contains a tag that matches this selector, it won't be crawled. For example, if you set 'withoutSelector' to '.no-crawl,' any page containing a tag with the class 'no-crawl' will be excluded from crawling based on the conditions you provide.
I plan to refactor the code during my vacation and resubmit it. Thanks again for your suggestions and feedback, and I hope we can improve this project together!


save always updates with server-side state

The save code says:

// After a successful server-side save, the client is (optionally)
// updated with the server-side state.

From the code that follows, I was assuming this was controlled using options.parse, but it seems i'm wrong. How do you make sure the model does not update itself using the server state?
FYI, I used the following to test this:
  // Succeeds
  test("`save` updates model with server response", function (t) {
        t.plan(2);
        var model = new Backbone.Model({props: {testing: 'string'}});
        model.sync = function (method, model, options) { options.success({testing: 'bar'}); };
        model.save({testing: 'foo'}, { parse: true });
        t.equal(model.get('testing'), 'bar');
        t.ok(true);
   });

  // Fails
  test("`save` does not update model with server response", function (t) {
        t.plan(2);
        var model = new Backbone.Model({props: {testing: 'string'}});
        model.sync = function (method, model, options) { options.success({testing: 'bar'}); };
        model.save({testing: 'foo'}, { parse: false });
        t.equal(model.get('testing'), 'foo');
        t.ok(true);
   });

{ parse: false } will only prevent piping data through model's parse function.

How do you make sure the model does not update itself using the server state?
Why would you exactly want to do that?

My initial problem was after doing a second change when the first change wasn't confirmed yet; in that case, the response to the first change sets the state to an outdated change. I guess the problem resolves itself eventually when the response to the second change comes in.
In the end, I found out my issue was more fundamental because i'm using a multi-threaded server, so I had to solve it in a different way than avoiding to update state on the client side (which was something i didn't need).


threshold 1

When I input 1 in the threshold field, trying to create a 1/2 Multisig wallet.  The interface says 1 is an invalid number.

We limit threshold between 2 and the number of members, same as polkadot apps

This is debatable I think.  In some scenarios, 1/m is required.  It means either member of the group or either one device of the list can represent others.  What do you think? @HackFisher

@AlexChien  https://polkadot.js.org/docs/substrate/errors/#minimumthreshold  Threshold must be 2 or greater.

OK, understood.


add deny purge property to stream spec

Added possibility to set "deny_purge" property on a stream

Hey @bruth @samuelattwood,
any chance to get this merged in the near future? Is there anything open from my side?

I think we should be good to merge this.
@bruth any objection?

Any updates on this? @bruth @Jarema

I think the PR is fine.
@bruth any objections, considering you asked for clarifications?

Hi @Jarema @bruth, anything we can do to advance the merge ? We'd really like to implement nack but need this feature :D

Hey, is there a timeline when I can expect this change to show up in an actual release?
AFAICT the latest (0.15.1) release is one week old and I am kind of surprised that it does not contain changes that have been merged to main for a month prior to the release

Hi @Jarema,
I saw that there is a new release tag v0.15.2, which still does not include changes since Jul 02.
Just like v0.15.1, github claims that the tagged commit does not belong to any branch of nats-io/nack and the chain of parents of the commits is 7b8b67f (v0.15.2) -> 31a8e48 (v0.15.1) -> 2dd3825 (v0.15.0) with 2dd3825 being the last commit known to github in the repo.
In any case the releases leave out this change and a number of dependabot updates (basically any change on main since Jul 02), so I am curious as to the rationale of these releases.
Either the tags are on an accidentially unpublished branch or this may be intentional. If it is the latter could you share some insight on when th expect the changes to apeear in an release?

Hey. Those are automated patch releases that just bump dependencies.
I will work on release tomorrow.


Parquet Writer Ignores "max statistics size" specification in WriterProperties

Describe the bug
WriterProperties::max_statistics_size is ignored in the writer
https://docs.rs/parquet/17.0.0/parquet/file/properties/struct.WriterProperties.html#method.max_statistics_size
To Reproduce
Set the stats size to 1 (byte) and notice that statistics are still happily created
Expected behavior
The statistics size limit should be respected (and documented more carefully -- like is it the total size of all statistics? If the limit is exceeded will partial statistics have been written, etc). Looking at the java or C++ parquet writer for inspiration is likely a good idea.
Additional context
In https://github.com/apache/arrow-rs/pull/2022 @tustvold  fixed the writer to respect the "do/don't compute stats", however the "max size of computed statistics" setting is still ignored

With the addition of statistics truncation, is this property even needed? It remains unused.

I think it probably can be removed


Mass vs me

In your code, you have me AND Mass
I don't understand

This was done to toggle the mass of the particle as some function of the electron mass. Obviously, it's not a variable that can be toggled by users explicitly right now.

Ah!
Thanks. Makes more sense

Isaac

On Feb 13, 2018 10:22 AM, "James McElveen"<EMAIL_ADDRESS>wrote:

This was done to toggle the mass of the particle as some function of the
electron mass. Obviously, it's not a variable that can be toggled by users
explicitly right now.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/jmcelve2/cheshire/issues/3#issuecomment-365298942,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABmk8u_ClH3rLMwbq8BEmDVu9XjrjDrwks5tUaisgaJpZM4SDnnX
.


Update pt-BR.coffee

Coverage decreased (-0.1%) to 61.963% when pulling 8d8a923a4cf3a106bb8464ca593bdd0fe55e6fb6 on traducaoenlix:patch-10 into 65641ed6f4c61668696e7f538807ed1ded522da0 on codecombat:master.

Thanks. Merged.


Fix locale bug with neato integration

Breaking Change:
Nothing breaks

Description:
The neato integration with the default docker image is currently broken because a locale is missing. This PR installs the locale en_US.utf-8 into the docker image. I know that this image will be migrated to an alpine-based image soon, but it would help people with this problem until then.
It can also be a reminder for the people who migrate the Docker images to include this locale in the alpine image as well. Please tell me when I can help with the migration.
Related issue (if applicable): fixes #25773
Pull request with documentation for home-assistant.io (if applicable): home-assistant/home-assistant.io#<home-assistant.io PR number goes here>
Example entry for configuration.yaml (if applicable):
neato:
  username: !secret email
  password: !secret neato

Checklist:

[x] The code change is tested and works locally.
[x] Local tests pass with tox. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.
[x] I have followed the development checklist

If it is not possible to merge this PR before 0.98.0, please close it. However, please tell me how I can test if this issue still exists with the alpine-based image.

To prevent questions: The Neato API does not accept any other locale than en_US.UTF-8 at the moment.
https://github.com/stianaske/pybotvac/pull/42

Update: this issue doesn't affect the alpine builds


Make root calendar div an application to use correct navigation mode …

…for nvda
Description
NVDA has browse and focus modes. By wrapping the calendar in role="application" the focus mode is chosen by default which allows arrow-keys navigation.
How has this been tested?
Manual testing with Chrome/Firefox+NVDA.
Documentation changes
[Do the changes include any API documentation changes?]

[ ] Yes, this change contains documentation changes.
[x] No.

Related Links
[Attach any related links/pull request for this change]

Review checklist
The following items are to be evaluated by the author(s) and the reviewer(s).
Correctness

[ ] Changes are backward-compatible if not indicated, see CONTRIBUTING.md.
[ ] Changes do not include unsupported browser features, see CONTRIBUTING.md.
[ ] Changes were manually tested for accessibility, see accessibility guidelines.

Security

[ ] If the code handles URLs: all URLs are validated through the checkSafeUrl function.

Testing

[ ] Changes are covered with new/existing unit tests?
[ ] Changes are covered with new/existing integration tests?

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

As you commented, this change is necessary for now for the change of https://github.com/cloudscape-design/components/pull/291. We can do it like this now and change to table markup or alternatives later 😄 .


Something went wrong while linking. Error: ENOENT: no such file or directory

Environment (I am using react-native 0.57.0):
Output from react-native info:
    System:
      OS: Windows 10
      CPU: (8) x64 Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
      Memory: 1.91 GB / 7.87 GB
    Binaries:
      Yarn: 1.9.4 - C:\Users\yossi\AppData\Roaming\npm\yarn.CMD
      npm: 5.6.0 - C:\Program Files\nodejs\npm.CMD
    IDEs:
      Android Studio: Version  <IP_ADDRESS> AI-173.4720617

Description

My project was up and running
I added the package react-native-check-app-install (using yarn)
When I tried to run react-native link I got the following error:

rnpm-install info Linking react-native-check-app-install android dependency
rnpm-install ERR! Something went wrong while linking. Error: ENOENT: no such file or directory, open 'C:\esites-PI\mobile-ui\PriceInsight_app\android\app\src\main\java\eu\<appname>\mobileapp\MainApplication.java'
Please file an issue here: https://github.com/facebook/react-native/issues

The problem seems to be that my MainApplication.java file is in android\app\src\main\java\trade\<appname>\mobileapp and android\app\src\main\java\eu does not exist

Solved it by below mentioned steps:-

deleted the project dir
cloned the project
npm i
react-native link


Improve performance

gimli is currently about twice as slow as rust-dwarf. Need to figure out why and see if gimli can be improved to the same or better speed.
I'm currently tracking down a performance bug in rust-dwarf (due to making endian generic). It's looking like a codegen bug in rustc, so hopefully what I learn from that will be applicable to gimli too.

Yikes.
Twice as slow doing what? DIE tree traversal? Are you measuring using the benches or something else?
If we profile and find which operations that dominate the results, it would be nice to add micro benches for those operations.

DIE tree traversal. I added a benchmark in rust-dwarf that uses gimli for comparison purposes.

perf shows that 40% of the time is spent in bench_parsing_debug_info. There is some inlining occurring, but a lot of it seems to be that all the ParseResult handling optimizes poorly. For example, we call cursor.current, and the result is copied around on the stack at least 3 times in quick succession. It doesn't help that the result is a DebuggingInformationEntry value, rather than a reference, so there's more to copy.
I've also spent a bit more time on the rust-dwarf regression, but the problem there is affected by MIR, so it might be worth waiting for MIR to stabilize before spending too much time looking at performance. I did find another problem there that I had to use unsafe to avoid, and the root cause of it seemed to be Result handling too.

perf reports parse_unsigned_leb at about 6.5%. Copying leb128::read::unsigned into parse_unsigned_leb (and fixing the error return values) changes that to about 2.5%. Hand coding the read_exact further reduces that to 1.5%. Note that leb128::read::unsigned was already being inlined. Looking at the assembly, the main difference is the amount of error handling code, and some of it is showing as hot spots. I think that having different error types for gimli and leb128 is bad (and similar for the io error type for read_exact).

Wow, that's unexpected and unfortunate. I guess we should just manually inline it all and stop using a separate crate for LEB128 parsing?

Yes we may need to do that, which I agree is unfortunate. If may be worth investigating if rustc can be changed to optimize this better.

Just FWIW -- leb parsing was pretty much the only bit of DWARF reading in gdb that benefited from optimization.  It's important to inline it.

Dumb low hanging fruit: c61b35f04a4b3278814ade668a2db9c34c1fc5e1

Is leb128::read::unsigned still being inlined now that parse_unsigned_leb is inlined? (I would check myself but I probably won't get to it for a few days.)

Funky. It looks like parse_unsigned_leb isn't always (ever?) getting inlined. Maybe those speedups were just from inlining parse_u8 and friends? Those ones are indeed getting inlined.
It appears that leb128::read::unsigned is still getting inlined into parse_unsigned_leb, but we should probably add #[inline] to leb128s exported functions to be sure and express our intent to the compiler. In fact, I am actually surprised that it gets inlined at all across crates without that attribute.

In fact, I am actually surprised that it gets inlined at all across crates without that attribute.

I just realized that leb128::read::unsigned is generic, so it has to appear in serialized form in the compiled crate to allow new specializations, and this ends up also allowing inlining.

I notice that constants.rs is using u64 for constants that will never be that large. This probably has some cost.

I notice that constants.rs is using u64 for constants that will never be that large. This probably has some cost.

The reason I initially switched them all to u64 was because we read them from leb128 which returns u64 and it also lets us precisely represent unknown variants.
Switching to a smaller representation will need to start range checking whenever we read these types.

By the way, gimli performance is now comparable to rust-dwarf, so I'll use this issue just to record random things I notice until I get back to looking at performance properly. One of those random things is that 9286858c improved the debug_info performance, but I can't explain why.
2ae61e361eb85d7684b11a092bd2ba7384859de5 test bench_parsing_debug_info                  ... bench:   2,993,722 ns/iter (+/- 62,450)
9286858c7db26c6f78da07bbf981409599662285 test bench_parsing_debug_info                  ... bench:   2,789,972 ns/iter (+/- 68,726)
I measured this a couple of times to check.

Not sure this issue contains anything of value now. Performance improvements are ongoing. The main concerns I still have are poor optimization of moves by the compiler, but that's being improved in the compiler, and is not something we can fix here.


Improve channel performance by serializing telemetry directly to stream to be used in Transmission

The JsonSerializer used to serialize Telemetry currently uses a MemoryStream to write serialized data to. This is then converted to a byte[], and stored inside Transmission.
https://github.com/Microsoft/ApplicationInsights-dotnet/blob/develop/src/Microsoft.ApplicationInsights/Extensibility/Implementation/JsonSerializer.cs#L64
This byte[] is then converted again to a new MemoryStream, for writing to network.
https://github.com/Microsoft/ApplicationInsights-dotnet/blob/develop/src/Microsoft.ApplicationInsights/Channel/Transmission.cs#L165
Proposing to avoid this conversions back and forth - modify JsonSerializer to write to just a MemoryStream and modify Transmission to use the same stream to write to network. This should avoid unwanted memory allocations/GC  etc and boost performance.

@cijothomas , based on the milestones mentioned above, can we assume that v2.14 already fixes that? Thanks

The milestone for this is "Future", means this is not done.

This also leads to GC pressure when used in server applications. We are currently going through a process of optimising memory allocations to reduces numbers of GCs and are now at the point that all our own memory allocations are dwarfed by Application Insights allocations. Some screen shots from JetBrains dotMemory. This is after just writing like 100 events.
System.Byte[] allocations:

System.Char[] allocations:

Looks like links are broken, but the usage of MemoryStream is still there.
https://github.com/microsoft/ApplicationInsights-dotnet/blob/19a3ecf77bebd7bbcffb6ca553b920016a412ed4/BASE/src/Microsoft.ApplicationInsights/Extensibility/Implementation/JsonSerializer.cs#L53
https://github.com/microsoft/ApplicationInsights-dotnet/blob/19a3ecf77bebd7bbcffb6ca553b920016a412ed4/BASE/src/Microsoft.ApplicationInsights/Channel/Transmission.cs#L175
Should I try to fix this ?


How to simple writing？？

	query := map[string]interface{}{ 		"query": map[string]interface{}{ 			"bool": map[string]interface{}{ 				"must": []interface{}{ 					map[string]interface{}{ 						"term": map[string]interface{}{ 							"user_status": map[string]interface{}{ 								"value": 1, 							}, 						}, 					}, 					map[string]interface{}{ 						"term": map[string]interface{}{ 							"user_id": map[string]interface{}{ 								"value":<PHONE_NUMBER>53854208, 							}, 						}, 					}, 				}, 			}}, 	}

Hello, please provide more information about what are you trying to achieve, what is not working, and so on.

I need to query multiple conditions, but this writing method is very difficult
eg:
I want to query through three conditions: username, userstatus and registrationtime. There will be many map [string] interface {} {}. It is very inflexible
sorry ,my english is very Poor. Can you understand what I mean ？

I understand what you're trying to achieve. The package accepts an io.Reader as the query definition, so it's up to you to provide the query — it could be a serialization of map[string]interface{} —, or it could be simply a strings.Reader. Have a look into _examples/encoding and _examples/xkcdsearch for samples.
In Elasticsearch terms, you probably want the Bool query — if you look into the documentation, and switch the language to Go, you'll see an example.