Fixed :meth:~DecimalNumber.string_to_mob for opengl and added tests

Overview: What does this pull request change?

Motivation and Explanation: Why and how do your changes improve the library?

This is another batch of tests reused from cairo with the opengl renderer. These are grouped together as they were all failing due to string_to_mob_map holding objects from the cairo renderer that were being reused by opengl. I created a separate map for the opengl renderer to use so the tests can pass now.
Links to added or changed documentation pages

Further Information and Comments

Reviewer Checklist

[ ] The PR title is descriptive enough for the changelog, and the PR is labeled correctly
[ ] If applicable: newly added non-private functions and classes have a docstring including a short summary and a PARAMETERS section
[ ] If applicable: newly added functions and classes are tested

Was the old string_to_mob_map implementation causing errors just for the tests or was it for opengl generally?

Was the old string_to_mob_map implementation causing errors just for the tests or was it for opengl generally?

Only for tests that I am aware of, think to create a runtime issue you might have to change renderer during a single run or something

Can we resolve the issues on this? Else i would suggest closing it

Closing due to inactivity.


Soft validation of Taxon name / page range of original source

The Soft validation of the Taxon name is unaware of multiple page ranges of the original source.

The validation apparently only takes into account the first range (pp. 28–65). Page 144 is within the second (pp. 134–167). Old references often come in several installments, which usually are treated as a single reference in case they were included in an issue published in the same year.

Yes, the pages in the Source is a free text, there are many different formats, including roman numbers. Citation could be comming from a plate, insteade of the page number. There are many different cases when this validation may fail. It is still very useful for exact ranges.

Closing as explained.


Impossible To Research Seaglide

I believe that this seaglide is impossible to research. Spent 4 or 5 hours trying to find it to no avail.
Looking at the spoiler log, it seems there are only 3 fragments that got generated, but 5 are required to build it. 3 is all I found in-game.
spoilerlog.txt
alternateStarts.csv
biomeSlots.csv
recipeInformation.csv
wreckInformation.csv

The spoiler log shows you that the fragment gets generated in three different biomes. How many fragments actually spawn is up to the game. I'm getting the same issue as you though, so I'm increasing spawn rates for all three of those biomes.

I did end up finding the remaining parts - I found six in total (of the five needed). For some reason they were incredibly slanted towards the area around the aurora.


Release 0.3.1

Aggregates issues related to release 0.3.1

0.3.1 and up:

finish refactoring TextToText encoder-decoder (secondary)
EarlyStopping
nn.DataParallel for the models
Ensure all problems have the fastest implementation of __getitem__ and collate_fn (if not done in 0.2)
Continuous integration: have automated unit tests in every problems & models, set up jenkins and GitHub hook
Have the install scripts working from scratch on several machines types
Introduce tutorials

0.4:

Timing utility (if not done with StatisticsAggregator)
bAby QA (if not done previously)
Version 2.0 for data descriptor, not as static (if not done with continuous integration in 0.3)
Refactor utils/GenerateFeatureMaps, make it more generic

0.5 ?
1.0:

Full PyTorch 1.0 support
Documentation accessible on the web (readthedocs.io ?)


Update CONTRIBUTORS.yaml

@PratikDJagtap Any specific reason to have usegalaxyp in the CONTRIBUTOR file? Maybe better on the home page: https://github.com/galaxyproject/training-material/blob/master/_layouts/home.html#L140

I think I misunderstood. I will close the issue. I will have the logo placed on the homepage later.


fromString needs exception :: EXCEPTION effect

This action throws an exception:
fromString "0xe40b5" Hex

Thanks for the report. Now that we're switching to Effect, the type includes the possibility of throwing exceptions, so I think this can be closed.


Client app [Tauri]

Fix + Improve Logs :
Client App [Notification]
Client App Fix Issue [Bug] 'export' button does not work https://github.com/Yidadaa/ChatGPT-Next-Web/issues/2884

preview looks better


envoy: Configure internal_address_config to avoid warning log

This was missed in the previous PR #35090
Relates: #34968, #35090

/test


I can't seem to get Nativewind running in Expo 49 without Expo router

Subject: Issue with Upgrading to Expo Router Version 3.0.0-next.34 and NativeWind Integration
Description:
I'm encountering an issue while upgrading an existing project to Expo Router version 3.0.0-next.34 and integrating NativeWind. Previously, I had been using version 2.0.11, and my teammate's new project with version 3.0.0-next.34 is working well with the new features. I've followed the updated documentation on the Expo website but have been unable to successfully implement the feature in my older project.
Issue:
When attempting to run the project, I'm encountering the following error related to Android bundling:
Android Bundling failed 1955ms
error: node_modules/expo/AppEntry.js: [BABEL] /home/idevosm/idevosm/osmaxin/scorebanter/score-banter-app/node_modules/expo/AppEntry.js: NativeWind preset was not included (While processing: "/home/idevosm/idevosm/osmaxin/scorebanter/score-banter-app/node_modules/nativewind/babel.js")

Attempts Made:

I've included the necessary configurations in my metro.config.js file as follows:
const metroConfig = require('@expo/metro-config');
const withNativewind = require('nativewind/metro');

const config = withNativewind(metroConfig.getDefaultConfig(__dirname));

module.exports = config;

In my babel.config.js file, I've made the following adjustments:
module.exports = function (api) {
  api.cache(true);
  return {
    presets: ['babel-preset-expo'],
    plugins: ['react-native-reanimated/plugin', 'nativewind/babel'],
  };
};

I've also tried using the recommended metro.config.js from the Expo website:
// metro.config.js
const { getDefaultConfig } = require("expo/metro-config");

const config = getDefaultConfig(__dirname, {
  isCSSEnabled: true,
});

module.exports = config;

To resolve the issue, I even moved the "nativewind/babel" plugin from the plugins array to the presets array in the babel.config.js file as my teammate's project is working fine with this configuration.

Despite these attempts, I am still encountering the same error. I also tried removing the entire metro file, but the error persists.
Any guidance or suggestions to resolve this issue would be greatly appreciated. Thank you.

I am having the same issues, could you perhaps post the package.json so I can get the same versions of expo and nativewind as you?

You need to follow all the required steps to use V3
Update your tailwind.config.js and it'll work

you can find newer instruction how to setup for expo here https://next.nativewind.dev/quick-starts/expo
works like a charm for
"expo": "~49.0.7",

v2 has some compatibility issues with Expo Router.
NativeWind v4 has full support https://www.nativewind.dev/v4/getting-started/expo-router

v2 has some compatibility issues with Expo Router.
NativeWind v4 has full support https://www.nativewind.dev/v4/getting-started/expo-router

Thanks for this update, I will like to know is the v4 stable for production, when should we be expecting it to be production ready. Again, I really appreciate the good work you have been doing on this project


Nested non-required object validation does not work

Describe the bug
When an non-required object inside an object has required fields, the object widgets validation ignores the non-required attribute of the object and highlights the required sub fields of the non-required object as required.
To Reproduce

Use configuration below
Create New Something
Only fill title string

Expected behavior
Entry saved correctly because some-metadata-property is not required.
Current behavior
Entry is not saved and property-specific-string is shown as required.
Screenshots

Applicable Versions:

Netlify CMS version: 2.4.2

CMS configuration
backend:
  name: test-repo
media_folder: assets
collections: # A list of collections the CMS should be able to edit
  - name: something
    label: 'Something'
    folder: nested
    create: true
    fields:
      - name: title
        widget: string
      - name: metadata
        widget: object
        fields:
          - name: some-metadata-property
            widget: object
            required: false
            fields:
              - name: property-specific-string
                widget: string

@selaux could you please edit your config file like so:
fields:
  - name: some-metadata-property
    widget: object
    fields:
      - name: property-specific-string
        widget: string
        required: false

@selaux could you please edit your config file like so:
fields:
  - name: some-metadata-property
    widget: object
    fields:
      - name: property-specific-string
        widget: string
        required: false

Hi, yes that might work in this specific case (when there is one property). But my usecase is that either some-metadata-property is not set or it is an object with specific properties. Making all these properties optional will allow the user to set a partial object, which is not my goal.

I am running into this issue as well. On some collection pages, I have an optional primary field that has several sub-fields—some required and some optional—and I cannot save a new page because of the required sub-fields (only relevant if someone is going to include that primary field, which they don't have to), even though the primary field is optional.

@selaux  @maxgardner   technically  the widget option required doesn't work on object and list widgets but on the actually input fields widgets.

@barthc Is this intended or does it have historical reasons?
One way I could think of to improve this would be instead of setting empty values such as "", [] or {} for fields, we could just remove the field value altogether when it is empty. This way non-required objects would just not be set. I dont know if there would be other implications though.
Another way would be to reflect requiredness in the ui and allow to add and remove property values to/from objects manually (by some kind of +/- buttons to the object editor ui).

Is this intended or does it have historical reasons?

I think it makes sense this way cause there is a  widget option pattern that works with the required  option https://www.netlifycms.org/docs/widgets/#common-widget-options  where you can define the regex pattern to validate against and the custom error message to display, again the pattern option technically doesn't work for list and object widget.

I'm honestly not certain what the original intent was, but the use cases mentioned here are sensible and should be supported. That said, and as @barthc points out, field validation may not be the right tool for the job.
List widget
On a list widget, what you really want is a set minimum number of items, with individual field validation being handled by the individual fields, including required validation.
Object widget
On an object widget, the UX of required validation would be confusing. All fields of the object are present and visible, and you can save the entry without filling them in, but if you fill in one of those fields, now the rest are required. There's no simple way to explain that to the user. What you really want is conditional visibility of a set of fields based on the value of another field, which again allows validation to be handled at the individual field level. This is covered under https://github.com/netlify/netlify-cms/issues/1267.
Thoughts?
Sent with GitHawk

@barthc That makes sense; thanks for explaining. The field validation definitely seems like it's working as expected in that case, and this particular problem is out of its scope.
@erquhart I think the conditional display of certain objects based on the user selecting to include that object would work well as a solution for my scenario; so long as the validation of any required sub-fields also was contingent upon this display status. Seems the issue you linked addresses that pretty clearly. Thank you!

Is there any timeline on allowing object widget fields have the option "required: false"? This is a feature/bug problem I have run into inadvertently.

I'm facing this issue right now. I'm not sure if the UX problem @erquhart explained is common, because I'm another user expecting that a nested required field shouldn't throw a validation error if no fields are filled in. The current UX makes me expect this.
I'll explore solving this using #1267

Is there any progress on this? If not, I'd be willing to submit a PR that adds an "omit value" checkbox to the object widget, that collapses it and treats all fields inside as not required.

@bigblind this seems like a workaround. We are looking for objects to force their required: false to its child widgets. That should be done in the config, not having a UX component that allows that.

Marking the object as optional would still be done via the configuration.
The checkbox or toggle would only show up for object fields configured to
be optional, and checking it indicates you want to omit the object
entirely, This then means that a required field in an object is still
required if the object is included in the data.
I personally don’t think that is a hacky workaround, it just adds Ui to
produce any output that’s allowed by the schema.
On September 24, 2024, Franco Bugnano @.***> wrote:
@bigblind https://github.com/bigblind this seems like a workaround. We
are looking for objects to force their required: false to its child
widgets. That should be done in the config, not having a UX component that
allows that.
—
Reply to this email directly, view it on GitHub
https://github.com/decaporg/decap-cms/issues/2103#issuecomment-2370458039,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFUBVPXZPR4D5FTZCMR3QLZYEKT3AVCNFSM6AAAAABOPOHRDOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZQGQ2TQMBTHE
.
You are receiving this because you were mentioned.Message ID:
@.***>

The original request is: don't validate the object if all fields are empty, but do validate if at least one field is not empty.
To achieve this, we don't need a new UI toggle, but we have to update the object validation to allow that.

Ok. I interpreted it as a more general request to allow omitting the values in optional object fields, didn't know the specific UX of just leaving fields blank was set in stone.


SLow loadin satellite data

Describe the bug
Loading one example fo the satellite is slow
To Reproduce
from ocf_datapipes.load import OpenSatellite
import time

t0 = time.time()
satellite_datapipe = OpenSatellite(
    zarr_path="gs://solar-pv-nowcasting-data/satellite/EUMETSAT/SEVIRI_RSS/v3/eumetsat_seviri_uk.zarr"
)

satellite_xr = next(iter(satellite_datapipe))

selected = satellite_xr.isel(
    {
        "time_utc": slice(10, 45),
        "x_geostationary": slice(100, 150),
        "y_geostationary": slice(100, 150),
    }
)
print(time.time() - t0)
selected_sum = selected.sum()  # this takes 0.01 seconds to load
print(time.time() - t0)
selected = selected.values  # this takes 5 seconds to load
print(selected)
print(time.time() - t0)
print(f"{selected.nbytes/10**6} MB")

Expected behavior
Should happen in < 1 seconds
Additional context
Chunk size is
(33, 33, 33, ..., ..., ..., 33, 33), (11,), (298,), (615,))

Suggestion from @jacobbieker is to satellite_xr.rechunk({'time': 12, 'variable': 11, 'x': 298, 'y': 615}) and resave it

What if you try that on Leonardo? Or Donatello? Does it still take that much longer to load than for sum?

whats the location on Leonardo?

 /mnt/storage_ssd_8tb/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/satellite/EUMETSAT/SEVIRI_RSS/zarr/v3/eumetsat_seviri_uk.zarr

/mnt/storage_ssd_4tb/metnet_train/eumetsat_seviri_uk.zarr on Donatello, mnt/storage_ssd_8tb/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/satellite/EUMETSAT/SEVIRI_RSS/zarr/v3/eumetsat_seviri_uk.zarr on leonardo

tried just using xr.open_zarr(zarr_path="gs://solar-pv-nowcasting-data/satellite/EUMETSAT/SEVIRI_RSS/v3/eumetsat_seviri_uk.zarr") to isolate the problem

What if you try that on Leonardo? Or Donatello? Does it still take that much longer to load than for sum?

It also took 5 seconds on Leonardo for OpenSatellite

tried just using xr.open_zarr(zarr_path="gs://solar-pv-nowcasting-data/satellite/EUMETSAT/SEVIRI_RSS/v3/eumetsat_seviri_uk.zarr") to isolate the problem, this gets it down to more like 1 second

It will probably worth checking what slows down OpenSatelliteIterDataPipe, but 1 second still feels way too slow.

tried just using xr.open_zarr(zarr_path="gs://solar-pv-nowcasting-data/satellite/EUMETSAT/SEVIRI_RSS/v3/eumetsat_seviri_uk.zarr") to isolate the problem, this gets it down to more like 1 second

It will probably worth checking what slows down OpenSatelliteIterDataPipe, but 1 second still feels way too slow.

0.7 seconds, still seems slow, but better than 5

Yea, 0.7 seconds is very good for loading one example, as one batch for example will be 32, and that'll add up quickly, to 22.4 seconds per batch

There seems to be a difference with
dataset = xr.open_zarr(store=zarr_path)

and
dataset = xr.open_dataset(leonardo_path, engine="zarr", chunks="auto")

The first is much quicker

I'll pay a PR, as this feels a big enough different and is slowly down training. Further speed ups im sure can be made

I've been creating various test zarrs, all of 3 days, or 864 timesteps, on leonardo under /mnt/storage_c/ going through options of chunking along time, size along spatial chunks, type of compression, number of channels included, and more. So we should have a better idea soon which one works best for us. They are all being created from the JPEG-XL compressed single timesteps saved on leonardo, not the raw files, to make it quicker to create them right now.

That sounds great, yea we should should see what size they are, and maybe have a tiny bit of code, to see how long one example takes to load.  Thanks for doing this @jacobbieker

Yeah, I am getting the sizes now, and was planning on copying the code above with a few changes to see how it affects loading times. But just to compare it to the current ones.

For sizes so far, compressing HRV files, the clear winner is JPEG-XL, its beating out bz2, zstd, and zlib by quite a lot, without needing to make the data into ints. It still is being compared to BitRound, Quantize, and ZFPY compressions too though, which are all somewhat lossy as well, like JPEG-XL. But, compared to the losses ones, for 3 days (864 timesteps) of data, the JPEG-XL takes 818MB of space, while BZ2 takes 39GB, so its only 2.1% the size of the worst compression. That would put 1 year of HRV data at ~100GB with JPEG-XL vs 4.75TB, and the whole 2014-2022 dataset at around 38TB for BZ2 vs 800GB for JPEG-XL. Still need to see how the 11 other non-HRV channels differ, and more chunk sizes, but size-wise it is a lot smaller. The best compression in size other than JPEG-XL so far is bz2 with the data as int8s, and 12 timesteps per chunk, at 2 GB for the 3 days of data, or twice the size of JPEG-XL.

Compressions
Size
Type
Speed

Current
TODO
TODO
TODO

JPEG-XL
0.8 GB
TODO
TODO

BZ2
2 GB
int8
TODO

Worth filling this out @jacobbieker

Compressions
Size
Type
Speed

Current
TODO
TODO
TODO

JPEG-XL
0.8 GB
float32
TODO

BZ2
2 GB
int8
TODO

Worth filling this out @jacobbieker

Thanks, I'll fill it out as I go

In looking at https://github.com/observingClouds/xbitinfo for the amount of "real information" in the satellite imagery. For keeping 99% of the information, we can BitRound down to 7 bits, or 99.9%/99.99% by going up to 11 bits.  I believe 99% is probably good enough, and I'll try that out

The fastest 10 loading times from leonardo onto Donatello for 378 combinations of HRV data.  If Bitround is 0.0 it means there is no rounding done. Precision of 8 means int 8, while 16 is in half-precision and 32 is full prevision.
For the tradeoff between space and size, the two winners seem to be JPEG-XL, or ZFP with timestep chunks of 4. ZFP is easier to use and doesn't have the issues that JPEG-XL is slightly harder to install support for (i.e. can't seem to install the library on Leonardo, as its not in the Ubuntu or Debian repos) vs ZFP. ZFP does require 4x4x4 chunks though, so is very inefficient for chunks that are not multiples of 4, and if the chunks are not multiples of 4 it pads each chunk to be a multiple of 4.
 ZSTD is what we are using for the NWP data, and doesn't need another install like ZFP, and is not as particular about the 4x4x4 blocks as ZFP is. I would go with bitrounding to 10 bits, using fp16, and timestep chunks of 4 (~20 minutes each), with these larger spatial chunks. Smaller chunking in x and y was also tested, but these larger chunks allow for greater compression and apparently easier to load.

thanks for doing all this, really useful to have it written down. Is it right we should go for the first one or the third one? First one is the fastest, but the third is the smallest? (Amazing difference in size btw). Im not quite sure on the trade-offs of ~15% slower to load, but reduce size by 98%.

I think the 3rd one because its so much smaller, it'll just have to transfer so much more data over a network with the larger one.

For the 8 years of satellite data, this corresponds to about 1.9TB for the first one, and 40GB for the third one

So actually not that big for the larger one, might try just making both? 40GB is incredibly small, and 2TB is pretty small too, relatively

Downside for the first one as well is that its int8, so theoretically is losing more information than the other ones

40GB sounds ideal - good for sharing too / running local stuff

Yeah, I'll run some more tests and do some plotting of it, to make sure its not just all rounded down to 0s or something, but then start on that

Hmmm, I just remade the best running ones according to this, and they are much larger, 12GB instead of 51MB. I'm thinking this is because to speed up testing a bit, for most of these, I created one 3 day Zarr with JPEG-XL, then created these other options based off that Zarr. Which means it was compressed once with JPEG-XL, then compressed again with the bitrounding, etc. But still need to look into it some more.

The fastest ones still all have the same spatial chunking, and either 4 or 12 timesteps, so just trying it again. But still on track to make the int8 version, which is the fastest anyway, and whose size makes more sense, as thats been more constant since the beginning.

Ah, it seems like the very small ones are nearly all NaNs, which explains how small they get, I'm assuming just from the multiple lossy compressions. But the int8 one should be still fine at least, just no 40GB file

Also now trying Blosc2, in this little wrapper:
from numcodecs.registry import register_codec
from numcodecs.abc import Codec
from numcodecs.compat import ensure_contiguous_ndarray
import blosc2

class Blosc2(Codec):
    """Codec providing compression using the Blosc meta-compressor.
    Parameters
    ----------
    cname : string, optional
        A string naming one of the compression algorithms available within blosc, e.g.,
        'zstd', 'blosclz', 'lz4', 'lz4hc', 'zlib' or 'snappy'.
    clevel : integer, optional
        An integer between 0 and 9 specifying the compression level.
    See Also
    --------
    numcodecs.zstd.Zstd, numcodecs.lz4.LZ4
    """

    codec_id = 'blosc2'
    max_buffer_size = 2**31 - 1

    def __init__(self, cname='blosc2', clevel=5):
        self.cname = cname
        if cname == "zstd":
            self._codec = blosc2.Codec.ZSTD
        elif cname == "blosc2":
            self._codec = blosc2.Codec.BLOSCLZ
        self.clevel = clevel

    def encode(self, buf):
        buf = ensure_contiguous_ndarray(buf, self.max_buffer_size)
        return blosc2.compress(buf, codec=self._codec, clevel=self.clevel)

    def decode(self, buf, out=None):
        buf = ensure_contiguous_ndarray(buf, self.max_buffer_size)
        return blosc2.decompress(buf, out)

    def __repr__(self):
        r = '%s(cname=%r, clevel=%r)' % \
            (type(self).__name__,
             self.cname,
             self.clevel,)
        return r

register_codec(Blosc2)

With Blosc2, the best option of the two up there is Zstd
It gives, for 1 day of data, 3.9GB file with FP16, which gives around 10TB for HRV data for the last 8 years.
If saved as int8, then its 2TBish for the last 8 years of HRV.
These are some of the best and so far fastest loading ones. They are all still larger than the JPEG-XL version, but are easier to use and quicker to decode and encode.

I think the way forward for this is to create 3 versions of each of HRV and non HRV for now. One would be the int8 version that is smaller and faster to load, one that is JPEG-XL for keeping as much of the original data as possible and easier to share size-wise, and the FP16 one for greater precision, while still being faster to load than JPEG-XL, although much larger. Sound good @peterdudfield @devsjc? The FP16 one might use up a lot of storage_c, but yeah.
The ones other than the JPEG-XL one would chunk in 12 timesteps at a time in time, and in the 1392x1044 spatial chunks, nonHRV would be also chunked all 11 channels together. This should cut down a lot on the number of requests our models need to read from disk.

Although for some reason, the compression seems to do worse if chunking all 11 channels together, instead of separately.

I think the way forward for this is to create 3 versions of each of HRV and non HRV for now. One would be the int8 version that is smaller and faster to load, one that is JPEG-XL for keeping as much of the original data as possible and easier to share size-wise, and the FP16 one for greater precision, while still being faster to load than JPEG-XL, although much larger. Sound good @peterdudfield @devsjc? The FP16 one might use up a lot of storage_c, but yeah.
The ones other than the JPEG-XL one would chunk in 12 timesteps at a time in time, and in the 1392x1044 spatial chunks, nonHRV would be also chunked all 11 channels together. This should cut down a lot on the number of requests our models need to read from disk.

Could you give the size and load times like you did above, for these different methods.

Although for some reason, the compression seems to do worse if chunking all 11 channels together, instead of separately.

how does this affect size and loading speed?

Although for some reason, the compression seems to do worse if chunking all 11 channels together, instead of separately.

how does this affect size and loading speed?

For size, with the blosc2 zstd fp32, its 19.2GB for the 11 channels chunked, and 13.9GB for individual channels. 10.5GB vs 7.5GB after bitrounding down to 13 bits, and and 4.2GB vs 2.8GB for uint8, so between 30 and 50% larger. As for loading speed, I'll add it in with the other ones in a bit, going to do stuff.

The non-HRV is also being created again on Donatello, as the lossless data is quite large, so the nonHRV values might change a bit (probably a bit larger and slower to access going by the differences between the ones created from lossess HRV and the lossy one)

Im strugglering a bit with lots of numbers here. Could we get something that summarizes shows the difference in speed and size with the diffenent methods:
Something like this for me would be very useful

Compressions
Size
Type
Speed

Current
TODO
TODO
TODO

JPEG-XL
0.8 GB
TODO
TODO

BZ2
2 GB
int8
TODO

Yep, can do that

feel free to add other rows if needed

Here it is for the HRV ones, seems like I might need to try ZFP a bit more

Compression
Speed
Size
Type
Bitround
Timestep Chunk
Channel Chunk

JPEG-XL
3.28152
1.4GB
FP32
7
1
1

JPEG-XL
3.42203
2.9GB
FP32
11
1
1

JPEG-XL
3.77315
4.0GB
FP32
13
1
1

Blosc2 ZSTD
0.07168
4GB
Uint16
None
12
1

Blosc2 ZSTD
0.07943
3.9GB
FP16
None
12
1

ZSTD
0.10940
6.5GB
FP32
13
12
1

Blosc2 ZSTD
0.11301
6.1GB
FP32
13
4
1

ZSTD
0.13513
2.8GB
FP32
7
4
1

Blosc2
0.143705
8.9GB
FP32
13
4
1

Blosc2
0.154134
6.1GB
FP32
13
12
1

Blosc2 ZSTD
0.167381
4.0GB
Uint16
None
4
1

Blosc2 ZSTD
0.186322
4.0GB
FP16
None
4
1

ZFP Tolerance=2
0.223568
354MB
FP32
11
4
1

The ZFP is the most lossy, so would need to check more how much error there is, but these ones did all return non-NaN results, so actually did return data

ZFP actually seems to be making everything the same value, not really sure why, so disregarding that now then. In lossless mode, ZFP creates huge (>30GB) files for this single day, so spacewise isn't really feasible

More ones for just Blosc2 ZSTD compression

Compression
Speed
Size
Type
Bitround
Timestep Chunk
clevel

Blosc Zstd
0.072362
825MB
Uint8
None
12
5

Blosc Zstd
0.18395
13GB
FP32
None
4
5

Blosc Zstd
0.273139
12GB
FP32
None
4
9

Blosc Zstd
0.336419
5.9GB
FP32
13
4
9

Blosc Zstd
0.398397
4.0GB
FP16
None
12
5

Blosc Zstd
0.59300
827MB
Uint8
None
4
5

Blosc Zstd
0.81063
4.1GB
Uint16
None
12
9

Blosc Zstd
1.2086
6.2GB
FP32
13
4
5

Blosc Zstd
1.240698
6.2GB
FP32
13
12
5

Blosc Zstd
1.69135
4.1GB
Uint16
None
4
9

Blosc Zstd
1.86772
12GB
FP32
None
12
9

Blosc Zstd
1.970765
4.0GB
FP16
None
4
5

Blosc Zstd
2.0007
5.9GB
FP32
13
12
9

Blosc Zstd
2.53900
13GB
FP32
None
12
5

Another set of outputs from Blosc2 ZSTD testing, this time removing any results where all the values are the same (indicating that the compression is compressing too much and making the data useless) or any NaNs are in the data (as the data loaded shouldn't have any NaNs)

Compression
Speed
Size
Type
Bitround
Timestep Chunk
clevel

Blosc Zstd
0.079500
4.0GB
FP16
None
12
5

Blosc Zstd
0.087661
3.9GB
Uint16
None
12
9

Blosc Zstd
0.091952
3.7GB
FP16
None
12
9

Blosc Zstd
0.13554
5.0GB
FP32
11
4
5

Blosc Zstd
0.136107
4.8GB
FP32
11
4
8

Blosc Zstd
0.145195
13GB
FP32
None
4
5

Blosc Zstd
0.146209
12GB
FP32
None
4
9

Blosc Zstd
0.160394
4.8GB
FP32
11
12
8

Blosc Zstd
0.167418
4.5GB
FP32
11
12
9

Blosc Zstd
0.16927
4.0GB
FP16
None
4
5

Blosc Zstd
0.18170
6.2GB
FP32
13
12
5

Blosc Zstd
0.20448
12GB
FP32
None
12
9

Blosc Zstd
0.20723
5.0GB
FP32
11
12
5

Blosc Zstd
0.21482
3.7GB
FP16
None
4
9

I think the best option from this is the FP16, 12 timestep chunk compression.  clevel=5 would result in a 10.5TB HRV 8-year dataset, while clevel=9 would give a 9.75TB dataset. clevel=5 is a lot faster to write and read, so the small tradeoff in size is probably worth it. The space difference might be more pronounced in the non-HRV data though, so do need to check that when can..

The HRV and non-HRV Zarrs are being created now, using Donatello to create them on Leonardo. They are being built using this script: https://github.com/openclimatefix/Satip/blob/main/scripts/read_and_combine_satellite.py saving them out as FP16, clevel 5, 12 timesteps per chunk. Each chunk is between 70-140MB on disk for the non-HRV. Each Zarr will cover a single year of data.
Because there are missing timesteps, the most likely next step after creating these ones is to download and append the missing timesteps to the zarrs. This might mean, though, that the 12 timestep chunks contain timesteps that are not near each other, and so might require one more creation pass where, after the missing data is added and the timesteps sorted in order, a new zarr is created.

Oh, and the one caveat for the HRV data is that, as the HRV images are sometimes slightly different values in the X dimension (between seemingly 5560-5580), they are being cropped down to 5548, so up to around 30 pixels will be missing. But it simplifies the creation a lot, and I would assume those 30 pixels are not too important being on the edge.

Great work Jacob!
Yeah, I think your plan sound good. Trading an increase in file size for an increase in read performance sounds good to me!
What exactly does the "speed" column mean in the results in this thread? Is it something like the number of seconds to read one chunk?

Its the amount of seconds to read a 35 timestep, 50x50 pixel set of data in the middle of the image. So, for example, the current compression (the one on GCP public datasets) takes 4.39 seconds to read that amount of data off my laptop's SSD.

The timesteps aren't aligned to the chunks, they have all been tested with this data being selected
selected = satellite_xr.isel(
    {
        "time_utc": slice(10, 45),
        "x_geostationary": slice(1000, 1050),
        "y_geostationary": slice(1000, 1050),
    }
)

As an update: All the HRV yearly zarrs are complete from 2014-2022. There might be some missing timesteps, which I am working to fill by getting the steps from EUMETSAT. Non-HRV is complete from 2018-2022, and is working on 2017 and earlier. The completed ones are being uploaded to GCP, and hopefully then onboarded onto Google Public Datasets.

Thanks @jacobbieker . This is for the full Europe extent right? It would be great to just extract the UK ready for ML modelling. Perhaps this is a task @devsjc could help with

Yes, the full geographical extant. And yeah, happy to have @devsjc do that bit!

Sure thing @peterdudfield , @jacobbieker; I've created an issue in Satip to track work on that, I'll raise any queries there.
https://github.com/openclimatefix/Satip/issues/150.

I think speeding it up is done for now. The splitting it out and such is tracked in other ones.

One issue that has come up, is since the xarr was created by appending to the time coordinate when making them, there are thousands of files making up the time dimension. Rewriting it bring it down to between 1 and 4 files. This should mean speeding up the opening  by quite a lot


0-fs can't reach the hub.grif.tf:9900 if a port forward for port 9900 exist on the node

let me explain the setup:
zos on bare metal -> zos VM -> 0-robot container
this is the nft ruleset of the zero on bare metal:
table ip nat {
	chain pre {
		type nat hook prerouting priority 0; policy accept;
		iif != "core0" iifname "zt*" tcp dport 6600 dnat to <IP_ADDRESS>:6600
		iif != "core0" tcp dport 9902 dnat to <IP_ADDRESS>:9900
		iif != "core0" tcp dport 9910 dnat to <IP_ADDRESS>:9900
		iif != "core0" tcp dport 9909 dnat to <IP_ADDRESS>:9900
		iif != "core0" tcp dport 9907 dnat to <IP_ADDRESS>:9900
		iif != "core0" tcp dport 9906 dnat to <IP_ADDRESS>:9900
		iif != "core0" tcp dport 9904 dnat to <IP_ADDRESS>:9900
		iif != "core0" tcp dport 9908 dnat to <IP_ADDRESS>:9900
		iif != "core0" tcp dport 9901 dnat to <IP_ADDRESS>:9900
		iif != "core0" tcp dport 9903 dnat to <IP_ADDRESS>:9900
		iif != "core0" tcp dport 9900 dnat to <IP_ADDRESS>:9900
	}

	chain post {
		type nat hook postrouting priority 0; policy accept;
		ip saddr <IP_ADDRESS>/16 masquerade
		ip saddr <IP_ADDRESS>/16 masquerade
	}
}
table inet filter {
	chain pre {
		type filter hook prerouting priority 0; policy accept;
	}

	chain forward {
		type filter hook forward priority 0; policy accept;
	}

	chain output {
		type filter hook output priority 0; policy accept;
	}

	chain input {
		type filter hook input priority 0; policy drop;
		ct state { established, related } accept
		iifname "lo" accept
		iifname "vxbackend" accept
		ip protocol 1 accept
		iif "core0" udp dport { domain, bootps, bootpc } accept
		iifname "zt*" tcp dport 6379 accept
		iif "kvm0" udp dport { domain, bootps, bootpc } accept
	}
}

the problem is that in the container, 0-fs tries to access hub.grid.tf:9900
but because of the nft rule, 9900 is forwarder to a local zerodb that we happen to also have listening on that port 9900 on the local machine which makes everything fails since of course the file we are looking for are not in the local zerodb.

I have to clarify that it's the 0-fs on the guest ZOS that has this issue, 0-fs on the host (for host containers) or even the guest vm are working fine

I have to clarify that it's the 0-fs on the guest ZOS that has this issue, 0-fs on the host (for host containers) or even the guest vm are working fine

indeed

More input from the latest issues we faced
Setup
---------------
    Minio    |
      vm     |
--------------
     | vm bridge
-------------------------------------------------------
- containers-bridge
  - 9901:ZDB0
  - 9902:ZDB1
  - 9903:ZDB2
  - ...
--------------------------------------------------------

We have a Minio ZOS vm running on top of host ZOS vm. The host vm also hosts local zdb with "proper" port forwards.
The rules applies to all traffic that is intended to dest port 9901, 9902, etc as shown in the graph, except for traffic coming from the container bridge.
Main problem happens is that a traffice that is NOT directed to host zos node (traffic that should be routed instead) also gets redirected to the zdb. for example if the any of VMs (minio zos in this case) is trying to reach out for hub.grid.tf:9901 it will be redirected to the locally hosted zdb instead!

Problem is that the port forward rule should be explicit to only redirect traffic that is intended to the host itself (like an input traffic), the problem is we don't know all the host ips and we can't change that dynamically
One suggestion is to use OpenVSwitch to do so, but that's require a rewrite for a huge part of zos networking and can't be done fast enough (I am still not sure if this can fix our problem in the first place)
Another faster solution is to only enable port forward for traffic that is intended to the IP of the eth0, the bond backend interface, and the main zt ip.

This issue is also blocking when deploying minio. Cause since we can have zerodb running on every node, when minio is trying to reach a zerodb on a another node but that port of zerodb is also forwarded on the local node, the same bug appears.

fixed by: https://github.com/threefoldtech/0-core/commit/d7e76cd5b488fa2f8cf7cc3f0b98334d096fb37d


About saving the vote_index.mat

Hello! I noticed that before training the network, I should generate a vote_index.mat. However, when I tried to generate and save the vote index for a 256*256 image, there came up an error:
ht_map = hough_transform(256, 256, 3, 1)
vote_index = ht_map.astype(np.float32)
sio.savemat('vote_index256.mat', {'vote_index': ht_map})
OverflowError: Python int too large to convert to C long

It seems that the matrix is too big to restore. Is there any way to restore this matrix efficiently?
By the way, if I want to use hough transform in each decoder stage of my network to get multiscale line features, should I generate multiple "vote_index"s for every resolution?
Looking forward to your kind reply!

Please take a look at this project, where a CUDA implementation is available for high-resolution HT.
The vote_indices are saved in a .zip file.
It has been tested on 256x256 feature maps.

Please take a look at this project, where a CUDA implementation is available for high-resolution HT. It has been tested on 256x256 feature maps. Links: https://github.com/yanconglin/VanishingPoint_HoughTransform_GaussianSphere/blob/7292b2e0a458770072d69e9c08d651a5edac9a27/parameterization.py#L78
https://github.com/yanconglin/VanishingPoint_HoughTransform_GaussianSphere/blob/7292b2e0a458770072d69e9c08d651a5edac9a27/vpd/models/vanishing_net.py#L24

Wow, thank you for your rapid reply! I will take a look at this project and modify my code. Thank you again!

Adding a hough transform layer same as the VanishingPoint_HoughTransform_GaussianSphere repository causes a huge computation burden. I added a HT_CONV layer to make ht and iht transformation on an 864256*256 tensor in my network, the training time cost of one epoch increases from 18min to 35min, is it caused by the huge matrices multiplication? Are there any efficient implementations?

Resolution increased x4 times. it will definitely lead to longer latency. So far I do not see other options. Perhaps downscaling the feature maps would the the first thing to try, although I do know exactly what you try to achieve.

Thank you! I will try it later.
---- Replied Message ----
| From | @.> |
| Date | 11/18/2024 16:01 |
| To | @.> |
| Cc | Zhang<EMAIL_ADDRESS>|
| Subject | Re: [yanconglin/Deep-Hough-Transform-Line-Priors] About saving the vote_index.mat (Issue #18) |
Resolution increased x4 times. it will definitely lead to longer latency. So far I do not see other options. Perhaps downscaling the feature maps would the the first thing to try, although I do know exactly what you try to achieve.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

In case you need high-resolution HT, it is better to optimize the CUDA code using shared memory stuff. It will perhaps solve your problem. My implementation is a minimal, workable version.

I was using the CUDA implementation of ht and iht in VanishingPoint_HoughTransform_GaussianSphere, I think maybe I should use the hough transform in a deeper stage which has lower resolution. Thank you for your suggestion!


Dosen't rotate ip and dont response on ip ban.

When my ip get rate-limited it dosent rotate ip & dosen't respond whereas works after restarting.
is there any way to make it work without restarting.
Version info
Client used: <IP_ADDRESS>
Output of java -version:
openjdk version "14" 2020-03-17
OpenJDK Runtime Environment (build 14+36-1461)
OpenJDK 64-Bit Server VM (build 14+36-1461, mixed mode, sharing)

it dosent respond after ip ban, it just searches and then nothing happens. After restarting works fine.

Please post logs and your ratelimit config


push a multi-arch image

This will allow the ip-masq-agent DaemonSet to work on any arch out of the box (in a follow-up PR) by using a multi-arch image (manifest list).
We're planning to use the agent for http://github.com/kubernetes-sigs/kind and nominally support ADM64, ARM64 and PPCLE64 for testing, this change will make using the agent simpler.
An example is at: gcr.io/bentheelder-kind-dev/ip-masq-agent:v2.3.0-1-g51408b3, produced with make all-push REGISTRY=gcr.io/bentheelder-kind-dev.
/cc @MrHohn

This pattern is borrowed from other Kubernetes Images: https://cs.k8s.io/?q=DOCKER_CLI_EXPERIMENTAL&i=nope&files=&repos=
In this case I referenced https://github.com/kubernetes/dns/blob/master/images/dnsmasq/Makefile which has a very similar Makefile to this repo.
cc @aojea

If this change is accepted, we can follow-up with s/-amd64// in ip-masq-agent.yaml once a new image is pushed.


scrollEnabled don't remove ScrollViews component

Hi, i've FlashList inside ScrollViews component, i've set scrollEnabled=false in order to remove scroll component but i've this error:
VirtualizedLists should never be nested inside plain ScrollViews with the same orientation because it can break windowing and other functionality - use another VirtualizedList-backed container instead.
I've to use FlashList into a lot of parts of my app but for one i've to put it into ScrollViews component and i want disable scroll for remove error, is possible?
Thanks

You should never put a FlashList inside a ScrollView, unless the direction is different.
So having a horizontal FlashList inside a vertical ScrollView is fine, but not if both are in the same direction.
Furthermore, you should not use ScrollView at all.
Use a FlashList and render another FlashList in that case. But as well, only when they are not in the same scroll direction.
Elements inside ScrollView render all at once and not lazy at all.


Submit Workflow: Emoji Wine

Preflight

[X] The Alfred team has invited me to submit this workflow.

The submitted workflow:

[X] Was authored by me. ⓘ
[X] Uses Workflow Configuration or has no user-facing settings. ⓘ
[X] Does not have an auto-updater. ⓘ
[X] Does not contain unsigned and non-notarised binaries. ⓘ
[X] Does not download extra software. ⓘ
[X] Uses keywords with more than one character. ⓘ
[X] Has a descriptive About section. ⓘ
[X] Has an icon of at least 256x256 px. ⓘ

Homepage
https://github.com/BenjaminOddou/alfred-emoji-wine
Alfred Forum page
https://www.alfredforum.com/topic/20333-emoji-wine/
Gallery page content
Usage
Search emojis with the wine keyword.

↩: Copy the emoji to the clipboard.
⌘↩: Paste the emoji to frontmost application.
⇧↩: Open the emojipedia page.
The harvest keyword is used to refresh the API and manage tags. Tags are used to search a group of emojis with custom keywords or sentences.

Tags
emoji, api, emoji-picker, emojipedia
Test data
/
Homebrew dependencies
No Homebrew dependency.
Bundled libraries
Python Pillow library (shipped in a lib directory from an universal binaries to suit intel and arm64 chips).

Example of tags (must be named tags-<en/fr...>.json) to be placed in ~/Library/Application Support/Alfred/Workflow Data/com.benjamino.emoji_wine.
[
    {
        "id": "7df02fe8d440",
        "title": "French Touch",
        "emojis": [
            "🥐",
            "🧀",
            "🍾",
            "🇫🇷",
            "🥖",
            "🍷"
        ]
    },
    {
        "id": "078f7f569d84",
        "title": "Unicode 15.0",
        "emojis": [
            "🫸🏽",
            "🪈",
            "🪭",
            "🫷🏽",
            "🫨",
            "🫸🏼",
            "🫏",
            "🪼",
            "🫸🏻",
            "🐦⬛",
            "🫷🏼",
            "🩵",
            "🫚",
            "🫸",
            "🩶",
            "🫷🏾",
            "🪇",
            "🫛",
            "🫎",
            "🪮",
            "🪻",
            "🫷🏻",
            "🫸🏾",
            "🩷",
            "🫷🏿",
            "🫸🏿",
            "🫷",
            "🛜",
            "🪽",
            "🪿",
            "🪯"
        ]
    }
]

Unfortunately, the Pillow library is making this impossible to run. Those binaries, the .so files, are not signed and thus are stopped by macOS. Sometimes those aren’t actually crucial and can be deleted without repercussions, but in this case that doesn’t work.
However, I looked around and pillow is available as a Homebrew formula so it may be possible for you to shift to use that instead? Then pillow would be listed as a dependency of the workflow and installable via the dependencies manager.
This is tangential, but it looks like you have installed the library with a Homebrewed Python. They should always be installed with the Python which ships with macOS (/usr/bin/python3) to ensure they work on most people’s installations. That doesn’t matter for this case though, as it’s not the source of the issue.

@vitorgalvao, indeed I didn't realized that I was shipping unsigned binaries. Normally I am putting all my python dependencies along with the workflow, that's why !
I can remove the lib folder inside the workflow and make pillow in the list of homebrew package (I am gonna change the docs also) then it should be good to go ? Is there other improvements I should do ?
Many thanks in advance.

@vitorgalvao, I updated the workflow and removed the embedded pillow library. Now it's required as a Homebrew dependency.

Thank you for the submission and fast turnaround. It will list not only pillow as a dependency, but python3 as well since it needs a Homebrew version. That required an extra bit of consideration, but everything looks fine. Should be live at https://alfred.app/workflows/benjaminoddou/emoji-wine/ during the day tomorrow.


Order Return: Change RMA to Return Number in sorting menu

Environment Details
Spartacus: epic/order-return-cancel branch
Browser: Chrome
Device: mac
Happens on: http://dev-com-10.accdemo.b2c.ydev.hybris.com:4200/electronics-spa/en/USD
Steps to Reproduce

Have a return request
Go to Order History -> Returns
Click on the Sorting menu

Actual Result
The menu has Date and RMA.
Maybe should change RMA to Return Number or similar for translation later.

Tested on dev10.


[enhancement] Bulk Download Frontend Implementation

What type of PR is this? (check all applicable)

[ ] Refactor
[X] Feature
[ ] Bug Fix
[ ] Optimization
[ ] Documentation Update
[ ] Community Node Submission

Have you discussed this change with the InvokeAI team?

[X] Yes
[ ] No, because:

Have you updated all relevant documentation?

[ ] Yes
[X] No

Description
Front end implementation for the bulk download feature, extends  #5159. Establishes the socket connection, listeners, and download when the packaging of the bulk download is complete.
Related Tickets & Documents

Related Issue #3899
Closes #

QA Instructions, Screenshots, Recordings
https://github.com/invoke-ai/InvokeAI/assets/22108632/3294ee7a-1cc8-49cd-9b62-0ca32d59889b

Merge Plan

This PR should be rebased and merged after #5159
Added/updated tests?

[ ] Yes
[X] No : front end changes

Hot damn! Thanks for the video.


HADOOP-18417. Upgrade to M7 of surefire plugin

This addresses an issue where the plugin's default classpath for executing tests fails to include org.junit.platform.launcher.core.LauncherFactory.  This commit was originally part of MAPREDUCE-7386, but has been pulled out so that it can be tracked independently.  It has already been tested as part of that pull request.

[X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 47s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
40m  1s

trunk passed

+1 :green_heart:
compile
0m 30s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
0m 31s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
mvnsite
0m 37s

trunk passed

+1 :green_heart:
javadoc
0m 40s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 29s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
shadedclient
63m 31s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 19s

the patch passed

+1 :green_heart:
compile
0m 18s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javac
0m 18s

the patch passed

+1 :green_heart:
compile
0m 19s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
0m 19s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
mvnsite
0m 22s

the patch passed

+1 :green_heart:
javadoc
0m 21s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 23s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

-1 :x:
shadedclient
3m 30s

patch has errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
0m 20s

hadoop-project in the patch passed.

+1 :green_heart:
asflicense
0m 38s

The patch does not generate ASF License warnings.

71m 34s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4795/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4795

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint

uname
Linux c58eeba1aab5 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / a75bbb35b6e130f3166b1ae471ee082cc3bc8101

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4795/1/testReport/

Max. process+thread count
693 (vs. ulimit of 5500)

modules
C: hadoop-project U: hadoop-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4795/1/console

versions
git=2.25.1 maven=3.6.3

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.


Replace winston logging with bunyan

Codecov Report

Merging #326 into master will decrease coverage by 0.29%.
The diff coverage is 76%.

@@            Coverage Diff            @@
##           master     #326     +/-   ##
=========================================
- Coverage   76.14%   75.84%   -0.3%
=========================================
  Files          21       22      +1
  Lines        1467     1569    +102
  Branches      255      278     +23
=========================================
+ Hits         1117     1190     +73
- Misses        350      379     +29

Impacted Files
Coverage Δ

lib/index.js
10.1% <0%> (ø)
:arrow_up:

lib/helpers/versions.js
100% <100%> (ø)
:arrow_up:

lib/helpers/yarn.js
100% <100%> (ø)
:arrow_up:

lib/helpers/npm.js
100% <100%> (ø)
:arrow_up:

lib/helpers/package-json.js
100% <100%> (ø)
:arrow_up:

lib/api/gitlab.js
5.76% <100%> (ø)
:arrow_up:

lib/config/file.js
100% <100%> (ø)
:arrow_up:

lib/config/index.js
100% <100%> (ø)
:arrow_up:

lib/worker.js
64.91% <100%> (ø)
:arrow_up:

lib/workers/branch.js
100% <100%> (ø)
:arrow_up:

... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update be5e4db...d28710b. Read the comment docs.


aws-cpp-sdk-transfer fails to upload file to s3 when target bucket name contains a dot

Hi,
I tried to upload a file using aws-cpp-sdk-transfer project, and when the bucket name contains a dot, transfer fails. generating error:
Unable to connect to endpoint
it this by intention? this is a bug or a feature of the sdk which does not dot at bucket name?
I tried a bucket named 'test_11.this'. problem reason is that the url is being built as virtual host name.
https://test_11.this.s3.amazonaws.com/a.txt and not using the path-like url i.e. https://s3.amazonaws.com/test_11.this/a.txt

Hi, there are some restrictions and limitations on bucket naming, the name should be DNS compliant, see here

You don't have to abide by the DNS compliance rules if you turn off virtual addressing (one of the parameters in the S3Client constructors). Virtual addressing is turned on by default.

Hi,
Thanks for the answers.
From the link you gave me, http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html I understand bucket name can contain a single period. "Bucket names must be a series of one or more labels. Adjacent labels are separated by a single period (.)"
I tried turning off the 'virtual addressing' at s3 client constructor, but it did not help, and I continued getting an error "Unable to connect to endpoint"
I tried also a bucket name with period with no numbers at all (i.e. only lowercase letters), but it did not work too

Sorry, I just read your original comment. Underscores are not allowed in bucket names.


[Suggestion] CraftTweaker support?

add CraftTweaker support
even editing repices dont work at all
or adding a Ore piece to a OreDict wont work
or rename any items or do anything with CraftTweaker at all

Could you please rephrase this in clear detailed manner using complete sentences.

I am closing this pending a more complete description. Most of the things you are requesting don't sound like things that would be done on Ex Nihilo's side.


Extension new permission: read and change your data on github.com

Select your issue type: (check at least one)

[x] Bug
[x ] Question
[ ] Suggestion
[ ] Other (please describe):

Describe your issue:
What is the reason for requiring such permission?

Cross origin permission to get remote patches to disable adblock detection
https://github.com/lap00zza/9anime-Companion/blob/82879507bf6246846552f08052a05e6d134b32f1/src/dashboard.ts#L331-L341

I tried without having to resort to the permission but wasnt able to find a solution. If I find one I will remove that perm. If you have any idea please do share.

Why not use pastebin.com instead of Github? Github's access is worrisome.

So that I can keep all the related code together. But yes I otherwise agree, I should probably remove the github permission.

I understand the code being together part. You can even have a small python script to update the pastebin from the repository json file.


Update README.md

Does not hide items

Does not hide items

Does if you remove "#" Ln. 270-273

thanks


Print a line from lily

This pull request includes a minor change to the MyConsoleApp/Program.cs file. The change adds a new line of code that prints "Hello from Lily!" to the console.

Can you sign your commit?


Shutter Support (MQTT Shutter Value)

Hi,
Theo just implement the Shutter Code from StefanBode's fork in the develop Branch.
However: when you SetOption80 to 1 (at least you should have two relays) you activate Shutter support. Here you got special MQTT Data what the Sonoff Adapter don't expect (or accept).
stat/RolloEZ/SHUTTER1 = "value" is set on every Shutter hold and very needfull for smothless Shutter support. The code Support up to 4 Shutter. So a implementation in the Sonof ioBroker-Adapter would be great.
Here full Tasmota Log:
22:13:12 MQT: stat/RolloEZ/RESULT = {"POWER1":"ON"}
22:13:12 MQT: stat/RolloEZ/POWER1 = ON
22:13:12 SHT: Shutter 0: Real Pos: 1900, Target 46000, source: Shutter, start-pos: 4 %, direction: 1, rtcshutter: 0.0 [s]
22:13:13 SHT: Shutter 0: Real Pos: 3600, Target 46000, source: Shutter, start-pos: 4 %, direction: 1, rtcshutter: 0.9 [s]
22:13:14 MQT: stat/RolloEZ/RESULT = {"POWER1":"OFF"}
22:13:14 MQT: stat/RolloEZ/POWER1 = OFF
22:13:14 MQT: stat/RolloEZ/SHUTTER1 = 90
22:13:14 MQT: tele/RolloEZ/RESULT = {"Shutter1":{"Position":90,"direction":0}}
and by the way:
Thanks for your great work!

OK just solved it by myself.
Just add in /lib/datapoints.js
SHUTTER1:       {type: 'number', role: 'state',                   read: true, write: false},
restart the adapter
have fun

Ticket kann geschlossen werden.
Aktuell in der 2.3.3 vorhanden.


Code passes tests

Challenge Profile Lookup has an issue.
User Agent is: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0.
Please describe how to reproduce this issue, and include links to screenshots if possible.
My code:

//Setup
var contacts = [
    {
        "firstName": "Akira",
        "lastName": "Laine",
        "number": "0543236543",
        "likes": ["Pizza", "Coding", "Brownie Points"]
    },
    {
        "firstName": "Harry",
        "lastName": "Potter",
        "number": "0994372684",
        "likes": ["Hogwarts", "Magic", "Hagrid"]
    },
    {
        "firstName": "Sherlock",
        "lastName": "Holmes",
        "number": "0487345643",
        "likes": ["Intriguing Cases", "Violin"]
    },
    {
        "firstName": "Kristian",
        "lastName": "Vos",
        "number": "unknown",
        "likes": ["Javascript", "Gaming", "Foxes"]
    }
];

function lookUpp(firstName, prop){
// Only change code below this line
for (var p = 0; p < contacts.length; p++) {

  if(contacts[p].firstName == firstName) {
  if (contacts[p].hasOwnProperty(prop)===true){
    return contacts[p][prop];
  }
  if (contacts[p].hasOwnProperty(prop)===false){
    return "No such property";
  }
  }

}
 return "No such contact";
// Only change code above this line
}
//lookUpp("Kristian", "lastName");
//lookUpp("Bob", "Number");
// lookUpp("Akira", "address");
lookUpp("Sherlock", "likes");
//lookUpp("Harry","likes");
// Change these values to test your function

@mikedhare please don't change the function names given in the challenge.

A lookUpProfile function that takes firstName and a property (prop) as arguments has been pre-written for you.

If you get stuck or have questions with a challenge in the future, please first ask questions to the Help Room. Happy coding!


Adapting to Lastet Near Lake Framework

Drop CI python version to 3.11 (near lake framework doesn't work with 3.12)
Load .env in Makefile
[Minor] Fix enumeration in Readme
🔑 Adapt main file to use latest near-lake-framework
Use >= on requirements file (also version bump near-lake-framework to 0.0.8)
Add AWS env details to readme (closes #7)

Thanks for your quick review. I made a few updated to the near-lake today as well, will circle back here tomorrow to add a few more improvements.


s390x: /x86/sse2/mm_loadl_epi64/emul/cpp ERROR

[ 51%] Building C object CMakeFiles/simde-test-emul.dir/x86/sse2.c.o
/usr/bin/gcc -DSIMDE_BUILD_CPP_TESTS -DSIMDE_NO_NATIVE -I/<<PKGBUILDDIR>>/test/..  -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -DSIMDE_ENABLE_OPENMP -fopenmp-simd   -std=gnu99 -o CMakeFiles/simde-test-emul.dir/x86/sse2.c.o   -c /<<PKGBUILDDIR>>/test/x86/sse2.c
[...]
[ 74%] Building CXX object CMakeFiles/simde-test-emul.dir/x86/sse2.cpp.o
/usr/bin/g++  -DSIMDE_BUILD_CPP_TESTS -DSIMDE_NO_NATIVE -I/<<PKGBUILDDIR>>/test/..  -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2   -o CMakeFiles/simde-test-emul.dir/x86/sse2.cpp.o -c /<<PKGBUILDDIR>>/gcc_test/x86/sse2.cpp
[...]
/x86/sse2/mm_loadl_epi64/emul/c      [ OK    ] [ 0.00000280 / 0.00000271 CPU ]
[...]
/x86/sse2/mm_loadl_epi64/emul/cpp    [ ERROR ]
Error: /<<PKGBUILDDIR>>/gcc_test/x86/sse2.cpp:4290: assertion failed: r.i64[0] == test_vec[i].r.i64[0]<PHONE_NUMBER>640 == -7993810106397570817)
Error: child killed by signal 6 (Aborted)

Full log at https://buildd.debian.org/status/fetch.php?pkg=simde&arch=s390x&ver=0.0.0.git.20200412-1&stamp=1586770086&raw=0

@nemequ Can we add some -O2 instances to the test matrices?

@nemequ Can we add some -O2 instances to the test matrices?

Yes, I think that's a good idea.  I've resisted that in the past because it messes up coverage analysis, but since we don't have that anymore (☹) I don't see any reason not to add at least -O2 to the builds.

I haven't been able to reproduce it outside of the test suite, so I haven't been able to produce a decent test case to send to the GCC folks.
96a4ea309ff707432517e873bc5268b78dedb3e8 should fix the issue, though.


Compatibility with SDK v3.

Hi all,
Is this driver compatible with the most recent SDK?
Best regards,
Diogo Martins

Yes, the interface is compatible with the SDK 3.0. We haven't updated the tests yet, but the API didn't change.

Perfect, thanks!


[DO NOT MERGE] Adds a new subtype for non-boolean types

Right now, a oclif flag type can only be boolean or option. This limits the information that you could learn about the command programmatically using the --json flag. Rather than adding other flag types for type, we add a new subType value that specifies the actual flag type
https://github.com/forcedotcom/cli/issues/279
Testing

Pull branch
yarn build
yarn link
Go to plugin-command
yarn link "@oclif/core"
Run bin/dev commands --json
Note that the new subType values are correct for this command's flags

@iowillhoit I'm not convinced yet that this is something we can actually support. While we can put a subType on oclif's native flag types, it doesn't solve the problem of custom flags created by other people. How do ensure that everyone adds a subType to their custom flags? We could make it a required property but then you're looking a major version bump.
Instead you have to look at ways for custom to pull that information out for every single flag, which I'm stumped on. The type of flag is really the type it returns from its parse - for instance, Flags.string is a string and Flags.duration is a Duration. Those types are the first type parameter given to custom but we don't have that type information available to us at runtime. So I'm not seeing at the moment how we can actually provide this information universally for any flag

That makes sense to me, going to close the PR and Issue
cc @mshanemc


invalid agricraft parents for botania:white_flower_plant

The mutations for the botania white plant is invalid. It should reference vanilla:wheat_plant and vanilla:melon_plant. The vanilla prefix is missing
mutation file:
https://github.com/Filostorm/Multiblock-Madness/blob/master/config/agricraft/json/defaults/mod_botania/mutations/botania_white_mutation.json
see corrected file in AgriPlants repo:
https://github.com/AgriCraft/AgriPlants/blob/1.12.2/mod_botania/mutations/botania_white_mutation.json
log file:
[00:13:12] [Client thread/INFO]: Invalid Mutation: Invalid Parent 1: "wheat_plant"!
Mutation:
- Parents:
- Parent 1: wheat_plant
- Parent 2: melon_plant
- Child: botania:white_flower_plant
- Chance: 0.5
[00:13:12] [Client thread/INFO]: Invalid Mutation: Invalid Parent 2: "wheat_plant"!
Mutation:
- Parents:
- Parent 1: botania:white_flower_plant
- Parent 2: wheat_plant
- Child: botania:yellow_flower_plant
- Chance: 0.5

It also appears you've accidentally gotten a copy of botania's mutations and soils mixed in with mystical world folder, duplicating them
https://github.com/Filostorm/Multiblock-Madness/tree/master/config/agricraft/json/defaults/mod_mysticalworld/mutations

It's honestly weird how much I've had to do to get agricraft working

Botania white petal seeds now have their own custom recipe in the TC cauldron, so it should be fixed now.


Create a Basic Loading Splash Screen

User Story
As a user, I want to see a nice loading screen until the program is ready to use so that I don't try to use the program while it is loading and encounter unexpected errors.
Description
Create a splash screen that gives the user some feedback that Platform.Bible is loading until the program is loaded and ready to use.

https://medium.com/red-buffer/how-to-create-a-splash-screen-for-electron-app-602b4da406d

There was a joke that the wait time could be used for a devotional time - but why not...? :-)

A splash screen is from an era when PCs were slow. So using one could make our application feel older than it needs to be. There are many other ways to handle this now, e.g. skeletons, loading spinner, etc. How about we determine our criteria for how quickly the app needs to open for us not to need a splash screen and make sure we are faster than that? How might we obviate the need for a splash screen by having a fast-loading interface?

The reason for a splash screen would not be any loading inside the app (where skeletons etc could be used), but the time until Electron at all shows anything. In my case using released Paratext-10-Studio-0.1.0.exe it takes 28 seconds without anything on the screen, see screen recording - and at least theoretically I have a fast machine.
Also @lyonsil as we talked about this last week.

Thanks, Sebastian. I believe you, but I think we'll want to dig into what might be causing the slow startup. I just packaged up Platform on my dev box and ran it, and it took about 2 seconds to open the UI. I think maybe Ira's point and my reluctance is that we would be better served by speeding up the app on open than adding stuff to hide it. We might need the splash screen, but hopefully we don't.

Note that running the portable executable (when you run the .exe with no supporting files around it including the builds from GitHub Actions. Running the installer installs a non-portable version) means the application has to dump hundreds of megabytes of data into temp folders before actually starting the app. I guess this portable delay is due to electron-builder. https://www.electron.build/nsis.html#portable I don't see much on that site in the way of configuring how the portable app works.
Anyway, regular users will probably install the application, so they won't experience this portable delay. In my experience, starting the not-portable app is fast like a couple seconds or less.
@Sebastian-ubs , are you running portable when you see this 28-second delay?
There's also the win-unpacked folder that gets created when you run a build. That folder is basically the same as what gets installed on the user's machine and probably the same as what gets dumped into temp folders. Running the exe in there starts up quickly.

@tjcouch-sil Just FYI, the 2 second startup I mentioned is the portable (i.e., single executable file) version.

I tried a couple of things on the Mac Mini that IT setup for us.

When we install the arm64 version of Platform.Bible, it opens very quickly. Less than a second.
When we install the x64 version of Platform.Bible, it opens very slowly. Something like 10 seconds. Apple allows running x64 versions of binaries on their arm64 silicon through a translation layer, but it's much slower than the native, arm64 version.

In both cases, the macOS installer is a full installation, not a portable version.


Bridge for Katla to Holesky (BUG)

Describe the bug
Information for reflection:
One transaction is delayed. It has been 11 days.
Not claimed - "An invalid proof was provided for this action"
IT'S A BAG.!!!
Explorer - https://explorer.katla.taiko.xyz/tx/0x544ee36142e07c7f7d0d79c66932cf486525c2e8ec36604da3a52ec487182617

Steps to reproduce
Transfer ETH from Katla to Holesky at bridge.katla.taiko.xyz

Your transaction has been claimed already and you have received your funds on Holesky. It just doesn't display the correct status in the UI yet, but that will come with a fix very soon.

facing this issue.. bridge stuck and swapping getting too much time

wallet address :- 0xD9C53DeE28dF57BB75BB18e31DcaB5f16650C1Da

taking too much time for bridge and lastly getting error while bridging

@Harshal9900 Can you add more about the transaction where you're facing this issue

Your transaction has been claimed already and you have received your funds on Holesky. It just doesn't display the correct status in the UI yet, but that will come with a fix very soon.

No, it's not true! The ETH test tokens never arrived on Holesky! Yes, the transaction was confirmed on the Katla chain, but no tokens were received on the Holesky chain. That's why I'm saying it's a BUG!!! I don't really care about this issue. 81 ETH more or less, it doesn't matter! I was just sharing that I found a BAG ;)

@cyberhorsey in case you want to follow up here

@jemy9900 Which transaction "is stuck"? Are you getting the same "Invalid proof" error message? If so, for which one

stale

1 month
https://explorer.katla.taiko.xyz/tx/0x544ee36142e07c7f7d0d79c66932cf486525c2e8ec36604da3a52ec487182617
5 days
https://explorer.katla.taiko.xyz/tx/0x7f75d2a4f4f9f5319d22064724c4efa02d19be8c0fa9f5bf188a01f112de524f


ABCOpts: Bail out if we don't know how-to hoist an array operation

Description: Array bounds checks assumed to see certain forms of SIL and only asserted them to be expected. This causes crashes when this assumption is broken.
Instead of asserting the forms it can handle check and bail if the expected forms are not encountered.
Scope: This issue was hit on one project so far.
Risk: Low, we bail out if the IR is not of the expected form.
Tested: SIL regression test added.
Radar: rdar://problem/28204253

@eeckstein Can you review?

@swift-ci Please test.

LGTM

Build failed
Jenkins build - Swift Test OS X Platform
Git Commit - 1aebe29f0bbec3f0444d28f93e486a9221b75b46
Test requested by - @aschwaighofer


Move preparfors pass

As title

Do not merge. PR for CI testing.

This demonstrates that at least for Numba, the NoPythonRewritesPass does not need to be sandwiched between PreParforsPass and ParforsPass.


Add azuresql driver bind

When accessing azure managed sql server via Azure AD for authentication we need to use azuresql driver
So the behavior of azuresql is the same of sqlserver.
This PR added it to default bind with AT

Hi,
When I use azuresql as driver. The sql will executed with error like
mssql: Incorrect syntax near '?'
This PR can fix it.

This would also fix my issue #928, thanks


VibrationalStructureResult does not print excited state energies

Information

Qiskit Nature version: cb05d69661a9d467aed9d3f208ea5630f7dc9fbf
Python version: 3.9.7
Operating system: Linux

What is the current behavior?
When printing a VibrationalStructureResult, the excited state energies are not being printed:
=== GROUND STATE ENERGY ===

* Vibrational ground state energy (cm^-1): 2534.621084885807
The number of occupied modals is
- Mode 0: [0.9999999999999999, 0.9999999999999999, 0.9999999999999999, 0.9999999999999999]
- Mode 1: [0.9999999999999989, 0.9999999999999989, 0.9999999999999989, 0.9999999999999989]
- Mode 2: [0.999999999999999, 0.999999999999999, 0.999999999999999, 0.999999999999999]
- Mode 3: [1.0000000000000002, 1.0000000000000002, 1.0000000000000002, 1.0000000000000002]
- Mode 4: [0.9999999999999997, 0.9999999999999997, 0.9999999999999997, 0.9999999999999997]
- Mode 5: [0.9999999999999981, 0.9999999999999981, 0.9999999999999981, 0.9999999999999981]
- Mode 6: [0.9999999999999997, 0.9999999999999997, 0.9999999999999997, 0.9999999999999997]
- Mode 7: [0.9999999999999999, 0.9999999999999999, 0.9999999999999999, 0.9999999999999999]
- Mode 8: [0.9999999999999996, 0.9999999999999996, 0.9999999999999996, 0.9999999999999996]
- Mode 9: [0.9999999999999992, 0.9999999999999992, 0.9999999999999992, 0.9999999999999992]
- Mode 10: [0.9999999999999997, 0.9999999999999997, 0.9999999999999997, 0.9999999999999997]
- Mode 11: [1.0, 1.0, 1.0, 1.0]
- Mode 12: [0.9999999999999987, 0.9999999999999987, 0.9999999999999987, 0.9999999999999987]
- Mode 13: [0.9999999999999994, 0.9999999999999994, 0.9999999999999994, 0.9999999999999994]
- Mode 14: [0.9999999999999993, 0.9999999999999993, 0.9999999999999993, 0.9999999999999993]
- Mode 15: [0.9999999999999982, 0.9999999999999982, 0.9999999999999982, 0.9999999999999982]

Steps to reproduce the problem
from qiskit_nature.algorithms.excited_states_solvers import ExcitedStatesEigensolver
from qiskit_nature.algorithms.excited_states_solvers.eigensolver_factories import (
    NumPyEigensolverFactory
)
from qiskit_nature.converters.second_quantization import QubitConverter
from qiskit_nature.drivers.second_quantization.gaussiand import GaussianForcesDriver
from qiskit_nature.mappers.second_quantization import DirectMapper
from qiskit_nature.problems.second_quantization.vibrational import VibrationalStructureProblem

driver = GaussianForcesDriver(
    logfile="test/problems/second_quantization/vibrational/resources/CO2_freq_B3LYP_ccpVDZ.log"
)

vib_problem = VibrationalStructureProblem(driver, num_modals=2, truncation_order=2)

vib_converter = QubitConverter(DirectMapper())

vib_numpy_solver = ExcitedStatesEigensolver(
    vib_converter, NumPyEigensolverFactory(use_default_filter_criterion=True)
)
vib_result = vib_numpy_solver.solve(vib_problem)
print(vib_result)

What is the expected behavior?
I would expect a section like in the ElectronicStructureResult case:
=== EXCITED STATE ENERGIES ===
...

Suggested solutions
Check the length of computed_vibrational_energies and print excited states (if present) just like the ElectronicStructureResult does it here: https://github.com/Qiskit/qiskit-nature/blob/3df2e2a/qiskit_nature/results/electronic_structure_result.py#L315-L328

@mrossinek You can assign me to this issue.


 Do you think we could centralize the logic?

That’s a valid point. I’ll look into refactoring to centralize the feature flag management.

sd


Fix h2 database errors when filtering users by multi attributes

Purpose

When filtering users by multi attributes by making embedded H2 as the default user store, some query executions are failing.

Related to https://github.com/wso2/product-is/issues/13879

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/2458825609

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/2458825609
Status: success


docker-entrypoint.sh issue

Any idea why this is happening after I do docker run:
/etc/dispatch/docker-entrypoint.sh: line 12: syntax error near unexpected token `start'
I set the chmod u+x, but I do not see the problem in the script.

@tgrzinic try updating your local master and see if you still get this error. I'm hoping my PR addressed it.

Yeah, double quotes have saved my day.
I find some other problems. There is a problem with .env location in config.py, it should be copied to the container and have a DISPATCH_HELP_EMAIL defined.
Also when I try to access the app this happens, and crashes the container:
No metric providers specified metrics will not be sent.
Usage: dispatch server start [OPTIONS] APP
Try "dispatch server start --help" for help.

Error: Missing argument "APP".

I am closing this because the entrypoint issue is fixed. Thanks!


Copy Revision - Homepage - Trainings

On the homepage, there is a reference to trainings/calendar that needs to be adjusted. (see screenshot for context)

We are concerned that a user might think they have to attend a training in order to use the site - which is not the case. Therefore, change
"New to MAPLE? Check out Our Calendar to see and join an upcoming training session!"
to
"New to MAPLE? For extra support, check Our Calendar and join an upcoming training session"

Hello World!


Update dependencies, remove peer dependencies.

Thank you but we do not want to remove the peerDependency, they serve a purpose of indicating the versions of webpack that this is compatible with


Hint component can render block-level elements as valid HTML

What
Switch the hint component wrapping element from span to div
Why
Spans are inline elements. Meaning they shouldn’t have block elements in them, like paragraphs etc.
For most hint text uses that’s fine. But there are many examples where hint text is a couple of lines. Not one.
Also see: https://github.com/alphagov/govuk-frontend/pull/1855

This will be released as part of 4.0.0

Released


Samtools (htslib)  integration testing

Need to add SAM/BAM/VCF round trip integration tests (htslib) to complement existing htsjdk based unit tests.   As motivated by #1104

Wanted to see if anyone has suggestions on three aspects of this before I get going on it:
1) How should we inform the test suite of where it can find samtools executable?
We could  ask that a "SAMTOOLS_HOME" env variable be set and if test suite finds a working samtools there then those tests run, otherwise skipped with a warning output?   Or - the test itself could download a samtools docker container, but then docker is needed.   Method we use needs to work with Jenkins.

Where should the intregration test code should go?  One idea: test/scala/org/bdgenomics/adam/rdd/read/SamtoolsSuite.scala for the sam/bam

Is it an ok  approach to extend ADAMFunSuite for these samtools intergration tests just as we do for all the unit test suites?

Going out of order:

Is it an ok approach to extend ADAMFunSuite for these samtools intergration tests just as we do for all the unit test suites?

That sounds good to me!

How should we inform the test suite of where it can find samtools executable?

We could ask that a "SAMTOOLS_HOME" env variable be set and if test suite finds a working samtools there then those tests run, otherwise skipped with a warning output? Or - the test itself could download a samtools docker container, but then docker is needed. Method we use needs to work with Jenkins.

We'll want to tag all of these tests with some sort of an integration test tag. That said, I think that requiring Docker for the integration tests is A-OK. We'll be the main consumers of the integration test suite.
We'll also need to create a repo with integration test files, because these should be real life sized inputs.

Where should the intregration test code should go? One idea: test/scala/org/bdgenomics/adam/rdd/read/SamtoolsSuite.scala for the sam/bam

I would probably put them in an org.bdgenomics.adam.integration package. This one would probably fit best in org.bdgenomics.adam.cli.integration, actually.

Closing this in favor of #1407.


Facebook Friends email

Hi,
This is a question rather than issue.
I would like to access Friend's email address via Facebook. Assuming that, my application is still in beta I would like to find out:
a. Whether, if someone approves access to (email, user_profile and user_friends) by accessing application - does it cover user's email scope?
b. Does python-social-auth allow retrieval of Friend's email address? If yes, what scope needs to be used?
c. If above requires extended permission, does it require application review process?
With best,
Rajan

I think you cannot access friend's email address.
But, if possible, only access a friends installed your apps. You can get friend's name and id using graph API. And then you can get email address using friend's id.
Good luck!

Thanks for the reply as I am searching for correct answer since last several days.
So, suppose at least couple friends have installed app and if I can get their id, name via user_friends scope. How can I use friend's id to access their email address using Graph API?


About pretrained model

Hi, I notice the Pretrained IR-SE50 model which is used for ID loss during training is trained based on face dataset. If I wanna train the model for car dataset whose ID feature shows big difference from FaceID's. Is it also work fine for the ID loss? Or is it better to replace the pretrained model with the one trained based on car datset. Thank you!

Closing for now, feel free to open if more clarifications are needed!


[17.06] Re-vendor SwarmKit to 4b872cfac8ffc0cc7fff434902cc05dbc7612da5

Related to docker/for-linux#2
/cc @mavenugo @andrewhsu @aaronlehmann
Includes:

docker/swarmkit#2203
docker/swarmkit#2210
docker/swarmkit#2212

LGTM
Note that there is another swarmkit PR that is proposed for 17.06: https://github.com/docker/swarmkit/pull/2211

Actually apologies, I'd like to also propose https://github.com/docker/swarmkit/pull/2213 although it's less show-stopping.

This PR brings in git commit https://github.com/docker/swarmkit/commit/f7a002c5f1d497158c211285813c3b10d6d0f5c5 into the engine's vendor dir to address issue https://github.com/docker/swarmkit/pull/2203
I've confirmed the vendor.conf change in this PR is only pulling in the one desired commit without the tests (because vndr would have stripped the test out of the vendor dir):
https://github.com/docker/swarmkit/compare/1a3e510517be82d18ac04380b5f71eddf06c2fc0...f7a002c5f1d497158c211285813c3b10d6d0f5c5

i rebased

LGTM
integration tests and unit tests are green


Fix/564

Closes #564

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@2b9bc44). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #566   +/-   ##
=======================================
  Coverage        ?   63.90%
=======================================
  Files           ?       50
  Lines           ?     1892
  Branches        ?        0
=======================================
  Hits            ?     1209
  Misses          ?      683
  Partials        ?        0

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


hiscore: Bounty Hunter emblem indicators cause hiscore lookups to use wrong name

Searched for this issue, didn't seem to find one for this. Feel free to close if this is a duplicate.
Describe the bug
In the new version of Bounty Hunter, players who have emblems have their name strings modified to have the tier of emblem they are currently carrying appended to their username. This causes the 'lookup' option on players to use the wrong name and fail.
To Reproduce

Go to a Bounty Hunter world
Right click someone with an emblem skull and click 'lookup'

Expected behavior
The correct player name should be used in the lookup.
Screenshots

In this instance, the player's actual name is 'lite tur', but the lookup option sends 'lite tur4' to the hiscore tab.
Environment (please complete the following information):
Linux Mint 19, Java 8
RL 1.6.13
Launcher 1.6.1

https://github.com/runelite/runelite/blob/baf4f58768ef98e180457846c35b74135bb229ff/runelite-client/src/main/java/net/runelite/client/plugins/hiscore/HiscorePlugin.java#L186
It looks like the actual symbol for the emblem is not a text character, so it's omitted from the text returned by event.getMenuTarget(). A hack might be needed to handle this in the HS plugin itself by doing some text removal logic there based on if someone has a Bounty Hunter emblem skull. I don't know if there's something in the API yet to check for that specific type of skull.

This is because we strip images from names (such as when players have BH emblems or clan chat icons appearing in their menu options) but apparently the number appearing after BH emblems is a plain 4 afterward.


ice3 antd-pro-scaffold template esm error

ice3 antd-pro-scaffold template。
使用pnpm install，接入gpt-vis后报错。

see https://github.com/facebook/create-react-app/issues/11865

gpt-vis 是 esm pakages，webpack 5 resolve  规则不一致，需要配置一下
https://github.com/webpack/webpack/issues/11467#issuecomment-691873586

https://stackoverflow.com/questions/70964723/webpack-5-in-ceate-react-app-cant-resolve-not-fully-specified-routes/75109686#75109686


Improve SplitContainer behavior, keeping asked split position whenever possible

Fixes https://github.com/godotengine/godot/issues/43749. (edited with the correct issue number)

This fixes #43749 not #64273
While the latter is very similar, it's also about Import dock being too wide.

split_offset seems to be calculated relative to the container. The origin is no longer dynamic, unless both expands are enabled.

split_offset seems to be calculated relative to the container. The origin is no longer dynamic, unless both expands are enabled.

Yeah, that's what solve the issue. I think it's the most expected behavior, but indeed, it kind of breaks compatibility.


Switching between contracts in taker UI has no effect

I think the PR: #2679 broke the dropdown in the taker UI.
Switching between contract symbols will not update the rate. I believe this is because we do not receive the necessary event and do not reconnect to the stream to retrieve old offers.
The taker UI will only update if a new offer was received.
I tried to explain this in this comment: https://github.com/itchysats/itchysats/issues/2673#issuecomment-1212732598
Available offers:

Note: this was the reason why I introduced the path parameter in the first place.
I propose timeboxing the bug investigation to 1h and rather rollback the PR as it has worked before.

I've verified it. Pre this PR it was working.

@bonomat I don't think this is a ui-bug, but an issue with how we get the latest events from the daemon. With #2679 we combined ETHUSD and BTCUSD offers into a single event. Now if we request the latest event we either get ETHUSD or BTCUSD. This happens because we using the channel as state to republish the latest sent event. The reason why it works with different API endpoints is because we keep separate channels for ETHUSD and BTCUSD.
However, I don't think that's the best design, because it makes our API very inflexible and introduces an additional connection. Instead I propose that we separate the state from the channel and manage it independently - thus allowing us to be more flexible on what we sent upon a subscription request.

@holzeis : to paraphrase what you are saying:
we should not get the latest state when connecting to the feed but have an extra API which allows us to get the latest state explicitly (e.g. the latest offer for a particular contract) and when connecting to the feed we will only get new events.
I think this is a fair alternative to our current approach. I have seen something similar to what we are using in the wild though. For certain events BitMEX sends out a full image of the current state when connecting to the websocket and then only sends out diffs whenever something changes.
I personally prefer what you are proposing because this makes things more explicit.

@bonomat well we could split it into different APIs, but personally I like that on subscription I get the latest state, respectively this could also be an argument on the API call if I want to get the latest state or only new events. I would vote for keeping the whole feed api bundled into one endpoint and manage filtering (behaviour) through e.g. query parameters. That way the client can decide what he is interested in - how many feeds he would like to use, etc. I guess that flexible design would also be helpful on the long run, when we considering that solutions will be built around itchysats. (Also note, that we already have different clients with different needs for the feed api - see auto taker / maker)


[Linked Account] NFT sometimes is not showing up

This bug already solved in Android prod version r2.3.2


Lack of Foobar version in foobar_save.s1

Environment

Elona Foobar verision: v0.2.5

Details
The version of Elona Foobar is not serailized/deserialized! We can't write any migration code now!
I'll post a new issue about compatibility of save data or something else.
Specification
Read
Read a version from save file, and convert the save if needed. Then, overwrite it by the latest one.
Write
Always write the latest version.

As usual, we can know version of save data from save data itself, in this case however, the version is not stored in save data.
Therefore...

Rename foobar_save.s1 to foobar_data.s1.

If foobar_save.s1 exists in save folder, the save is old version.

When loading old version save, delete foobar_save.s1 and run some migration code. Then add foobar_data.s1(with foobar version) and save.
After v0.2.6, we can read foobar version from save data(foobar_data.s1).


Fix HyperLink click event not being generated

Add Sense::click to the HyperLink widget

I think this will be solved by https://github.com/emilk/egui/pull/855 - please test!

@emilk Thanks!


Please add a licence

Hi
This is a very nice work, Kudos! l would like to use your code for my research. Could you please specify the licence? Without a licence, the default rule says that I legaly cannot use your code for my project.

Done. Added MIT License for this and all other diffusion repos.


Why are these new changes not a different project?

The nice point of this package was that it was an extension of System.IO.  This latest release breaks everybody's projects.  It really should be its own nuget package, and you should restore the old one as to not break anybody's code

Sorry about that, I've relisted all packages now!


Helm upgrade with --force fails with possibly incorrect error message

Output of helm version:
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}

Output of kubectl version:
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.0", GitCommit:"0ed33881dc4355495f623c6f22e7dd0b7632b7c0", GitTreeState:"clean", BuildDate:"2018-09-28T15:20:58Z", GoVersion:"go1.11", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.5", GitCommit:"9b635efce81582e1da13b35a7aa539c0ccb32987", GitTreeState:"clean", BuildDate:"2018-08-02T23:42:40Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.): GKE
I'm trying to upgrade a helm release using the --force flag as there's resources which I know already exist (not part of the original release) but I want helm to delete and recreate them so they become part of the new release. Is this how the force flag works?
Either way, the upgrade fails but the error message is strange. The first time we ran it we got:
Error: UPGRADE FAILED: no Service with the name "notifications" found

Obviously I went to our cluster to check if the service existed and as expected it, it did exist.
We re-ran the upgrade and then we got a different message:
Error: UPGRADE FAILED: no Secret with the name "notifications" found

This time for the secret instead of the service and again we checked and the secret definitely exists.
Is the actual problem that --force doesn't work as I expected and the release is failing because the resources DO exist rather than don't exist but helm showing us the wrong error message?

I have a couple questions:

Can you share the chart that's causing the problem?
You say there are assets not part of the original release? How were they added for helm to notice them?
Is the name in the chart template for these the exact same as what you see?

Usually the name is set to something related to the release so that multiple versions of a chart can be installed. This is often the fullname...
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "fullname" -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}

@mattfarina the chart templates are custom charts but I'm fairly confident the issue is simply an incorrect error message. As an example, if I create a secret named blah (outside of helm using kubectl or something) and then create a chart which includes a secret for the same namespace with with the same name i.e. blah, when I try to install the chart it should fail saying that it can't create the secret because a secret of the same name already exists.
Instead we're seeing this:
Error: UPGRADE FAILED: no Secret with the name "blah" found

But the secret does exist i.e. we created it outside of helm and it's still there now. And the fix is to remove the existing secret so the secret in the helm chart doesn't have a name collision.
Let me get an example together which I can post here

It was a bit more difficult to reproduce than I first thought but here's how to reproduce the problem.
Clone the example repo:
git clone https://github.com/antonosmond/helm-example.git
cd helm-example

The chart has 2 secrets. The first is purely so we can deploy an initial
release successfully. The bug only appears when there has been a successful
release previously. The second secret is initially commented out and will
be used to demonstrate the bug after the initial release.
Install the chart which should be successful:
helm upgrade --install example . --force

Create a secret outside of helm named example e.g.
kubectl create secret generic example --from-literal=password=password

Now uncomment the second secret in the chart at templates/secret-2.yaml
At this point we have created a secret named example outside of helm so
if we try to run helm upgrade (now that we've uncommented the second secret)
it should try and create a secret named example in the new release but
should fail telling us it can't create the secret because a secret with
the same name already exists:
helm upgrade --install example . --force

Instead, the output I see from the final command says:
Error: UPGRADE FAILED: no Secret with the name "example" found

I think this might be related to/is a duplicate of #3785 and #1193. Can you check and confirm, or is there something else at play?

@bacongobbler I believe there are 2 separate problems. In #1193 there's a lot of talk about helm not cleaning up resources following a failed install/upgrade. While it's true that this would cause resources created by the failing install to become orphaned and cause a name collision during a subsequent upgrade, that is separate from the issue about the misleading error. In my example, there's no previously failed deployment.

I can confirm that the issue in #1193 is one way this can happen.  Someone has submitted #4871 which looks like a good fix for that situation (which is how this happened to me).
The error message makes a little more sense in context:
https://github.com/helm/helm/blob/d5c183e297aa0c445c4eca4ff4e6ebc3f44bc333/pkg/kube/client.go#L280-L284
Perhaps "no %s with the name %q found in previous release" would be slightly improved wording for that error message?

closing as inactive.


using ssh while web authentication is disabled

Gitea version (or commit ref): 1.12.6
Pullrequests: https://github.com/go-gitea/gitea/pull/13537 and https://github.com/go-gitea/gitea/pull/13527
Git version: 2.29.2
Operating system: Debian Buster (gitea gets started over systemd)
Database (use [x]):

[x] PostgreSQL
[ ] MySQL
[ ] MSSQL
[ ] SQLite

Can you reproduce the bug at https://try.gitea.io:

[ ] Yes (provide example URL)
[x] No (admin account needed for that)

Description
Since the update to version 1.12.6 it is not possible any more that accounts which have the permission flag "User Account Is Activated" and "Disable Sign-In" set to true to perform git operations over ssh. Before that it was only affecting the http(s), which was very useful as service accounts could only ssh for operating with gitea (fetch,pull,...) and the authentication via http(s) was disabled.
I set the flag "Disable Sign-In" to false in the admin site web interface and the service accounts were able to communicate over ssh again.
This sounds like a bug to me, that the "Disable Sign-In" is also disabling ssh operations. If not, why would there be two options to disable an account or are there plans to get a dedicated permission flag which permits external service accounts to interact with the repositories via ssh?

The bug was that accounts marked as disabled were in fact able to clone. For what you are looking at doing please use deploy keys.

Hi @techknowlogick , thanks for the reply.
Yea I know about the deploy keys, but these are can only be set to a dedicated project till now and if you have, lets say more than 100 repositories owned by each organisation, you have to deploy a lot of keys or am I wrong? And for such things it would make sense to have either a service user setting/attribute or to be able to create a deploy key in the organisation which can be used for all repositories inside of the organisation.