No module named 's5p_tools'

Hello, I was wondering if I may have some guidance on an issue I have encountered when trying to use the s5p-request.py script. When trying to run the following command I get an error : "ModuleNotFoundError: No module named 's5p_tools' "
I have not altered the s5p-request.py script in anyway and have followed the recommended set up instructions for the environment in conda:
conda create --override-channels -c conda-forge -c stcorp --file requirements.txt --name SP5
Any advice would be greatly appreciated. Thank you.
Command and error:

python s5p-request.py L2__O3____ --date NOW-1MONTH --aoi map.geojson
Traceback (most recent call last):
File "s5p-request.py", line 15, in
import s5p_tools
ModuleNotFoundError: No module named 's5p_tools'

Hi there,
Try with this command line:  python s5p-request.py L2__O3____ --date YYmmdd1 YYmmdd2 --aoi map.geojson
where YY-mm-dd1 == for e.g. 20210401 and YY-mm-dd2 ==fior e.g.,  20210420
Note: The code is working now properly in my computer. Check again and let us know,
I hope this helps,
Cheers

Hi,
Thank you for getting back to me so quickly, I really appreciate it!
I have tried the format you suggested and I still get the same error unfortunately:

python s5p-request.py L2__O3____ --date<PHONE_NUMBER>10 --aoi map.geojson
Traceback (most recent call last):
File "s5p-request.py", line 15, in
from s5p_tools import (
ModuleNotFoundError: No module named 's5p_tools'

My workflow is:

conda create --override-channels -c conda-forge -c stcorp --file requirements.txt --name SP5
pip install sentinelsat
s5p-request.py L2__O3____ --date<PHONE_NUMBER>10 --aoi map.geojson

I was wondering if maybe I have missed a step when defining 's5p_tools' at all?
Thank you again for your help.
My s5p-request.py script:
import argparse
import sys
import warnings
from multiprocessing import cpu_count
from os import makedirs
from os.path import exists
from pathlib import Path
import numpy as np
import pandas as pd
import rioxarray
import xarray as xr
from tqdm import tqdm
from s5p_tools import (
bounding_box,
convert_to_l3_products,
get_filenames_request,
request_copernicus_hub,
)
def main(
product,
aoi,
date,
qa,
unit,
resolution,
command,
chunk_size,
num_threads,
num_workers,
):
tqdm.write("\nRequesting products\n")

_, products = request_copernicus_hub(
    login=DHUS_USER,
    password=DHUS_PASSWORD,
    hub=DHUS_URL,
    aoi=aoi,
    date=date,
    platformname="Sentinel-5 Precursor",
    producttype=product,
    download_directory=DOWNLOAD_DIR / product,
    checksum=CHECKSUM,
    num_threads=num_threads,
)

L2_files_urls = get_filenames_request(products, DOWNLOAD_DIR / product)

if len(L2_files_urls) == 0:
    tqdm.write("Done\n")
    sys.exit(0)

# PREPROCESS DATA

tqdm.write("Converting into L3 products\n")

# harpconvert commands :
# the source data is filtered + binning data by latitude/longitude

keep_general = [
    "latitude",
    "longitude",
    "sensor_altitude",
    "sensor_azimuth_angle",
    "sensor_zenith_angle",
    "solar_azimuth_angle",
    "solar_zenith_angle",
]

harp_dict = {
    "L2__O3____": {
        "keep": [
            "O3_column_number_density",
            "O3_effective_temperature",
            "cloud_fraction",
        ],
        "filter": [f"O3_column_number_density_validity>={qa}"],
        "convert": [f"derive(O3_column_number_density [{unit}])"],
    },
    "L2__NO2___": {
        "keep": [
            "tropospheric_NO2_column_number_density",
            "NO2_column_number_density",
            "stratospheric_NO2_column_number_density",
            "NO2_slant_column_number_density",
            "tropopause_pressure",
            "absorbing_aerosol_index",
            "cloud_fraction",
        ],
        "filter": [
            f"tropospheric_NO2_column_number_density_validity>={qa}",
            "tropospheric_NO2_column_number_density>=0",
        ],
        "convert": [
            f"derive(tropospheric_NO2_column_number_density [{unit}])",
            f"derive(stratospheric_NO2_column_number_density [{unit}])",
            f"derive(NO2_column_number_density [{unit}])",
            f"derive(NO2_slant_column_number_density [{unit}])",
        ],
    },
    "L2__SO2___": {
        "keep": [
            "SO2_column_number_density",
            "SO2_column_number_density_amf",
            "SO2_slant_column_number_density",
            "cloud_fraction",
        ],
        "filter": [f"SO2_column_number_density_validity>={qa}"],
        "convert": [
            f"derive(SO2_column_number_density [{unit}])",
            f"derive(SO2_slant_column_number_density [{unit}])",
        ],
    },
    "L2__CO____": {
        "keep": ["CO_column_number_density", "H2O_column_number_density"],
        "filter": [f"CO_column_number_density_validity>={qa}"],
        "convert": [
            f"derive(CO_column_number_density [{unit}])",
            f"derive(H2O_column_number_density [{unit}])",
        ],
    },
    "L2__CH4___": {
        "keep": [
            "CH4_column_volume_mixing_ratio_dry_air",
            "aerosol_height",
            "aerosol_optical_depth",
            "cloud_fraction",
        ],
        "filter": [f"CH4_column_volume_mixing_ratio_dry_air_validity>={qa}"],
        "convert": [],
    },
    "L2__HCHO__": {
        "keep": [
            "tropospheric_HCHO_column_number_density",
            "tropospheric_HCHO_column_number_density_amf",
            "HCHO_slant_column_number_density",
            "cloud_fraction",
        ],
        "filter": [f"tropospheric_HCHO_column_number_density_validity>={qa}"],
        "convert": [
            f"derive(tropospheric_HCHO_column_number_density [{unit}])",
            f"derive(HCHO_slant_column_number_density [{unit}])",
        ],
    },
    "L2__CLOUD_": {
        "keep": [
            "cloud_fraction",
            "cloud_top_pressure",
            "cloud_top_height",
            "cloud_base_pressure",
            "cloud_base_height",
            "cloud_optical_depth",
            "surface_albedo",
        ],
        "filter": [f"cloud_fraction_validity>={qa}"],
        "convert": [],
    },
    "L2__AER_AI": {
        "keep": [
            "absorbing_aerosol_index",
        ],
        "filter": [f"absorbing_aerosol_index_validity>={qa}"],
        "convert": [],
    },
    "L2__AER_LH": {
        "keep": [
            "aerosol_height",
            "aerosol_pressure",
            "aerosol_optical_depth",
            "cloud_fraction",
        ],
        "filter": [f"aerosol_height_validity>={qa}"],
        "convert": [],
    },
}

# Step size for spatial re-gridding (in degrees)
lon_step, lat_step = resolution

if aoi is None:
    extent = [-180, 180, -90, 90]
else:
    extent = bounding_box(Path(aoi))

# computes offsets and number of samples
lat_edge_length = int(abs(extent[3] - extent[2]) / lat_step + 1)
lat_edge_offset = extent[2]
lon_edge_length = int(abs(extent[1] - extent[0]) / lon_step + 1)
lon_edge_offset = extent[0]

# create HARP commands
if command is None:
    harp_commands = (
        ";".join(harp_dict[product]["filter"])
        + (";" if len(harp_dict[product]["filter"]) != 0 else "")
        + ";".join(harp_dict[product]["convert"])
        + (";" if len(harp_dict[product]["convert"]) != 0 else "")
        + "derive(datetime_stop {time});"
        + f"bin_spatial({lat_edge_length},{lat_edge_offset},{lat_step},{lon_edge_length},{lon_edge_offset},{lon_step});"
        + "derive(latitude {latitude});derive(longitude {longitude});"
        + f"keep({','.join(harp_dict[product]['keep'] + keep_general)})"
    )

else:
    harp_commands = command

# perform conversion
convert_to_l3_products(
    L2_files_urls,
    pre_commands=harp_commands,
    post_commands="",
    export_path=EXPORT_DIR / product.replace("L2", "L3"),
    num_workers=num_workers,
)

# Recover attributes
attributes = {
    filename.name: {
        "time_coverage_start": xr.open_dataset(filename).attrs[
            "time_coverage_start"
        ],
        "time_coverage_end": xr.open_dataset(filename).attrs["time_coverage_end"],
    }
    for filename in L2_files_urls
}

# AGGREGATE DATASET

tqdm.write("Processing data\n")

# Avoid lost attributes during conversion
xr.set_options(keep_attrs=True)

def preprocess(ds):
    ds["time"] = pd.to_datetime(
        np.array([attributes[ds.attrs["source_product"]]["time_coverage_start"]])
    ).values
    return ds

DS = xr.open_mfdataset(
    [
        str(filename.relative_to(".")).replace("L2", "L3")
        for filename in L2_files_urls
        if exists(str(filename.relative_to(".")).replace("L2", "L3"))
    ],
    combine="nested",
    concat_dim="time",
    parallel=True,
    preprocess=preprocess,
    decode_times=False,
    chunks={"time": chunk_size},
)
DS = DS.sortby("time")
DS.rio.write_crs("epsg:4326", inplace=True)
DS.rio.set_spatial_dims(x_dim="longitude", y_dim="latitude", inplace=True)

# EXPORT DATASET

tqdm.write("Exporting netCDF file\n")

start = min(products[uuid]["beginposition"] for uuid in products.keys())
end = max(products[uuid]["endposition"] for uuid in products.keys())
export_dir = PROCESSED_DIR / f"processed{product[2:]}"
makedirs(export_dir, exist_ok=True)
file_export_name = export_dir / (
    f"{product[4:]}{start.day}-{start.month}-{start.year}__"
    f"{end.day}-{end.month}-{end.year}.nc"
)

DS.to_netcdf(file_export_name)

tqdm.write("Done\n")

if name == "main":
# Ignore warnings
warnings.filterwarnings("ignore", category=RuntimeWarning)
warnings.filterwarnings("ignore", category=FutureWarning)

# PARAMS

# Perform checksum verification after each download
CHECKSUM = True

# CLI ARGUMENTS

parser = argparse.ArgumentParser(
    description=(
        "Request, download and process Sentinel data from Copernicus access hub. "
        "Create a processed netCDF file binned by time, latitude and longitude"
    )
)

# Product type: Used to perform a product based search
# Possible values are
#   L2__O3____
#   L2__NO2___
#   L2__SO2___
#   L2__CO____
#   L2__CH4___
#   L2__HCHO__
#   L2__AER_AI
#   L2__CLOUD_
parser.add_argument("product", help="Product type", type=str)

# Date: Used to perform a time interval search
# The general form to be used is:
#       date=(<timestamp>, <timestamp>)
# where < timestamp > can be expressed in one of the following formats:
#   yyyyMMdd
#   yyyy-MM-ddThh:mm:ssZ
#   yyyy-MM-ddThh:mm:ss.SSSZ(ISO8601 format)
#   NOW
#   NOW-<n>MINUTE(S)
#   NOW-<n>HOUR(S)
#   NOW-<n>DAY(S)
#   NOW-<n>MONTH(S)
parser.add_argument(
    "--date",
    help="date used to perform a time interval search",
    nargs=2,
    type=str,
    default=("NOW-24HOURS", "NOW"),
)

# Area of interest: The url of the area of interest (.geojson)
parser.add_argument(
    "--aoi", help="path to the area of interest (.geojson)", type=str
)

# Harp command: Harp convert command used during import of products
parser.add_argument(
    "--command",
    help="harp convert command used during import of products",
    type=str,
)

# Unit: Unit conversion
parser.add_argument("--unit", help="unit conversion", type=str, default="mol/m2")

# qa value: Quality value threshold
parser.add_argument("--qa", help="quality value threshold", type=int, default=50)

# resolution: Spatial resolution in arc degrees
parser.add_argument(
    "--resolution",
    help="spatial resolution in arc degrees",
    nargs=2,
    type=float,
    default=(0.01, 0.01),
)

# chunk-size:
parser.add_argument(
    "--chunk-size",
    help="dask chunk size along the time dimension",
    type=int,
    default=256,
)

# num-threads:
parser.add_argument(
    "--num-threads",
    help="number of threads spawned for L2 download",
    type=int,
    default=4,
)

# num-workers:
parser.add_argument(
    "--num-workers",
    help="number of workers spawned for L3 conversion",
    type=int,
    default=cpu_count(),
)

args = parser.parse_args()

# PATHS

# download_directory: directory for L2 products
DOWNLOAD_DIR = Path("L2_data")

# export_directory: directory for L3 products
EXPORT_DIR = Path("L3_data")

# processed_directory: directory for processed products (aggregated+masked)
PROCESSED_DIR = Path("processed")

# CREDENTIALS

DHUS_USER = "s5pguest"
DHUS_PASSWORD = "s5pguest"
DHUS_URL = "https://s5phub.copernicus.eu/dhus"

main(
    product=args.product,
    aoi=args.aoi,
    date=args.date,
    qa=args.qa,
    unit=args.unit,
    resolution=args.resolution,
    command=args.command,
    chunk_size=args.chunk_size,
    num_threads=args.num_threads,
    num_workers=args.num_workers,
)

I let Bilel suggest a solution to fix this problem. I believe he is the perfect person to do so since he is the developer of this tool :)
Look forward,

I let Bilel suggest a solution to fix this problem. I believe he is the perfect person to do so since he is the developer of this tool :)
Look forward,

Thank you so much for your help! :)

Hi @LillaBW2327, did you clone the repo in your working directory before running the script?
cd your/working/directory
conda activate S5P
git clone https://github.com/bilelomrani1/s5p-tools.git
python s5p-request.py L2__O3____ --date NOW-1MONTH --aoi map.geojson

re, I thought this is already done by @LillaBW2327
please check it and let us know,
cheers

Hi @bilelomrani1 , thank you so much for your quick reply, it is greatly appreciated.
I had not cloned the repo so thank you for highlighting that for me. While that solved my previous issue, I now have a new one I'm afraid!
My code is exactly the same as that posted above, only difference is that I have now cloned the repo. I have tried different versions of the s5p-request.py script and get the same error.
Any thoughts and ideas would be greatly appreciated. Thank you again.
(SP5) [bw43776@abics03 Documents]$ python s5p-request.py L2__O3____ --date NOW-1MONTH --aoi map.geojson
Traceback (most recent call last):
File "s5p-request.py", line 11, in
import rioxarray
File "/home/ab/bw43776/anaconda3/envs/SP5/lib/python3.7/site-packages/rioxarray/init.py", line 6, in
import rioxarray.raster_array  # noqa
File "/home/ab/bw43776/anaconda3/envs/SP5/lib/python3.7/site-packages/rioxarray/raster_array.py", line 18, in
import rasterio
File "/home/ab/bw43776/anaconda3/envs/SP5/lib/python3.7/site-packages/rasterio/init.py", line 22, in
from rasterio._base import gdal_version
ImportError: libnsl.so.1: cannot open shared object file: No such file or directory

Hi there,
You will need to install some packages e.g., : rioxarray, gdal ("File "s5p-request.py", line 11, in import rioxarray")
to install for instance rioxarray run:
conda install -c conda-forge rioxarray
I hope it helps.

Hi there,
You will need to install some packages e.g., : rioxarray, gdal ("File "s5p-request.py", line 11, in import rioxarray")
to install for instance rioxarray run:
conda install -c conda-forge rioxarray
I hope it helps.

Hi, thank you so much for your help, but I am afraid I have already tried this as a previous issue was related to not having the rioxarray package. When running for example: conda install -c conda-forge rioxarray, I get the confirmation that "All requested packages already installed".

Hi @HichemOmr @bilelomrani1 , thank you so much for your help on this issue.  The last issue I posted has been resolved, and was linked to an issue with the  libnsl package and was related to my Linux system. (My issue for reference is basically the same as this: https://github.com/conda-forge/fiona-feedstock/issues/138). Once I had downloaded this package separately, the script ran perfectly. Thank you again.

Awesome !
Let me know in case you would like to do a joint research/study on the topic.
Here is my pro email<EMAIL_ADDRESS>and here is my skype name: hichem.omrani
Cheers


fix disorder when reading data

Detail:
https://github.com/StarRocks/starrocks/issues/8324#issue-1294998753

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

if not specify query fields, what will happen?

https://github.com/StarRocks/starrocks/pull/8578
https://github.com/StarRocks/starrocks/pull/9437
fixed, you can try the latest version.


[Question] ReferenceError: require is not defined

1. Summary
I want to replace default browser body scrollbar to custom scrollbar. I can't do it, if I use Gemini-Scrollbar.
2. Settings
My GeminiTest.html file: https://codepen.io/Kristinita/pen/aWWqQE.
I create it as in official repository description.
3. Steps to reproduce
I reproduce the problem in a Chrome and Opera.
I open GeminiTest.html in browser.
4. Expected behavior
Custom body scrollbar.
5. Actual behavior
I get in Firefox console:
ReferenceError: require is not defined

I don't get custom scrollbar.
6. Environment
Operating system and version:
Windows 10 Enterprise LTSB 64-bit EN
Firefox:
53.0 (64-bit)
Gemini-Scrollbar:
The latest stable version of Gemini-Scrollbar
Thanks.

I'm sorry, I didn't notice folder examples in this repository. It would be nice, if you add link to it in repository description and gemini-scrollbar site, because another users can do not understand, where are the examples.
Now I solve the problem. Thanks.


Value of type 'iPhoneNumberField' has no member 'formatted'

I Want to get the unformatted number and im getting this error:
Value of type 'iPhoneNumberField' has no member 'formatted'
            iPhoneNumberField(nil, text: $viewModel.phoneNumber)
                .flagHidden(false)
                .prefixHidden(false)
                .flagSelectable(true)
                .maximumDigits(10)
                .formatted(false)

You may use  v0.5.4 or below. Please use v0.5.5 or above to access that.


steps using portal, and steps using powershellinclude update registry using powershell

I believe this is the powershell way
As an administrator:
$FileShareName = "\\fsprofile.file.core.windows.net\share"

New-Item -Path "HKLM:\SOFTWARE\FSLogix\" -Name Profiles -Force
New-ItemProperty -Path "HKLM:\SOFTWARE\FSLogix\Profiles" -Name Enabled -Type string -Value $FileShareName

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 69769604-b1c3-a2b8-318d-354ebf0d82a1
Version Independent ID: 964330fc-1a4f-05ca-502a-a09f26e79f10
Content: Create FSLogix profile container Azure Files Active Directory Domain Services - Azure
Content Source: articles/virtual-desktop/create-profile-container-adds.md
Service: virtual-desktop
GitHub Login: @Heidilohr
Microsoft Alias: helohr

@jonwbstr
Thanks for the feedback! I have assigned the issue to content author to check and update the document as appropriate.

@Heidilohr
Can you please check and add your comments on this doc update request as applicable.

Thanks for the feedback - I've added in a PowerShell method of configuring these registry entries.   The changes should be visible in the next few days.
#please-close


Add Swift 3.0

After making all changes to the cask:

[X] brew cask audit --download {{cask_file}} is error-free.
[X] brew cask style --fix {{cask_file}} reports no offenses.
[X] The commit message includes the cask’s name and version.

Additionally, if adding a new cask:

[X] Named the cask according to the [token reference].
[X] brew cask install {{cask_file}} worked successfully.
[X] brew cask uninstall {{cask_file}} worked successfully.
[X] Checked there are no [open pull requests] for the same cask.
[X] Checked that the cask was not already refused in [closed issues].

Closes https://github.com/caskroom/homebrew-cask/issues/29361.

Whoops, thanks @vitorgalvao.

should the token be changed as well to avoid confusion?

But then we allow duplicates. If problems arise, we’ll deal with it then.


Use Vec3A for (Global)Transform's Translation and Scale

What problem does this solve or what need does it fill?
Make operations on translation and scale faster with SIMD operations.
What solution would you like?
Change translation and scale to Vec3A. Update all APIs to work with Vec3A instead of Vec3.
What alternative(s) have you considered?
Leave them as Vec3, use scalar operations on them.
Additional context
When not using scalar-math, on platforms where SIMD registers are available (WASM, x86). glam's Quat already has an align of 16, which means both Transform and GlobalTransform both have 8 bytes of padding. Might as well use Vec3A to accelerate operations on the type. You can check this with the following on a x86 machine:
assert_eq!(std::mem::size_of::<Transform>(), 48);

I tested performance recently for transform * transform (where GlobalTransform3A is GlobalTransform with Vec3A for translation and scale components):
GlobalTransform - mul_transforms: 2000000, t: 33ms, muls/s: 59585212
GlobalTransform3A - mul_transforms: 2000000, t: 24ms, muls/s: 81130010
Affine3A - mul_transforms: 2000000, t: 16ms, muls/s: 124499416
Mat4 - mul_transforms: 2000000, t: 29ms, muls/s: 67183140

Discussion with someone in the game engine industry suggests that the separate representation like we do for (Global)Transform is the right source of truth for editing. I don’t know how propagation is done though - whether using the separate representation or going via Affine3A-type matrices. I’m investigating what is done there and why.
I did note that Affine3A has precision differences compared to GlobalTransform3A though I don’t know which would be more correct. Accuracy could also be a factor.

It seems that the separate representation as we have in Transform and GlobalTransform is only intended to be used for local transforms. The local transform is converted to an affine matrix, the local matrices are propagated to create local to world matrices and these are used for rendering. So it seems that the separate representation in Transform should be a source of truth but GlobalTransform does not need to have a separate representation because it is not edited in that way, only calculated. The affine matrix can be calculated on the fly from the separate representation in Transform, and GlobalTransform can store the matrix. Also, if an hierarchy of local matrices do non-uniform scale, rotate, non-uniform scale, rotate that creates a shear which cannot be represented by the separate translation, scale, rotation components. That makes me think the way we propagate right now using GlobalTransform is wrong

IMO, converting to Affine3A, multiplying, and back is likely going to perform worse.
Transform propagation right now is fundamentally bottlenecked by the storage format and access patterns right now, so changing the type used here is likely to see negligible wins in perf.
However, it's not the only location where SIMD operations might be beneficial. For example, frustrum culling seems to be reliant on GlobalTransform's position and orientation and sees slight wins from using Vec3A over Vec3, but it incurs a conversion cost to align the Vec3. If Transform and GlobalTransform used Vec3A, that conversion could be removed. It should be noted that conversion in the opposite direction is likely a no-op, as individual components are already 4-aligned in Vec3A. Other general operations on both Transform and GlobalTransform may be faster too.

I am saying that GlobalTransform should internally be an Affine3A. There would not be a conversion from Affine3A back to any separate representation. It’s one-way.

That might save some time during extract since Affine3A -> Mat4 would be a fairly trivial transformation. Though arguably we would see worse cache performance as Affine3A is 64 bytes over GlobalTransform(3A)'s current 48. Worth testing either way.

@cart will need to weigh in on where we land with this so you avoid unnecessary work. Our current GlobalTransform propagation does not support shears. I think non-uniform scale propagation will have unexpected results in cases which would produce a shear of propagated with an Affine3A and that would differ from other engines. I want to prove this by trying it in other engines to use as an argument for why GlobalTransform should be an Affine3A internally. I also want to understand what use cases may be broken by such a change if GlobalTransform didn’t expose ways of getting the scale and rotation. Translation is easy to get.

Whether or not we adopt Affine3A for GlobalTransform and whether or not we use Vec3A in Transform are definitely two separate conversations. @superdump the Affine3A conversation should happen in a different thread.
Vec3A in Transform
It complicates the public interface of what amounts to the most common component (Transform) by introducing a "second" vec type. Most people wont need to care, but converting to and from Vec3 and Vec3A is something I generally don't want bevy users to worry about. "Vector performance" is an implementation detail that users of Bevy apis should not ever need to think about.
This leaves us with a number of paths:

Use Vec3 everywhere, leave perf on the table in favor of simplicity and consistency.
Use Vec3A everywhere. Let new users scratch their heads about what the A stands for. I personally dont like this.
Use Vec3A everywhere, but re-export it as Vec3. This would generally just work, but would make some things like const_vec3a weird (can macros have export aliases?). We would need to evaluate how weird this would be for users reading glam docs / find all of the apis this would make "weird".

Given that (1) and (3) both look roughly the same from a user facing perspective, I don't feel much pressure to make the switch hastily.
Fun fact: I actually helped break us in to this problem :smile:
https://github.com/bitshifter/glam-rs/issues/36
There seems to be a lot of sentiment in favor of a math lib that exports a bunch of different representations for a given type, then letting consumers assign type aliases to their preferred variants. So (3) isn't completely crazy.
Affine3A in GlobalTransform

Pros:

Faster to compute
Can represent shear (good for artists). Also a con.

Cons:

Translation/Rotation/Scale can no longer be fields, they must be "accessor methods" that compute them on demand
The GlobalTransform api is now different from the Transform api. Different performance characteristics (accessing scale on GlobalTransform now requires computation), different implementations (aka we can't share code between Transform and GlobalTransform ... which is something that comes up a lot), different access patterns (global_transform.scale() vs transform.scale).
Scale and Rotation are now "lossy": In some cases, even for root entities, transform.scale != global_transform.scale() due to floating point math.
Can represent shear (bad for physics users and developers). Also a pro.

This is a much more "disruptive" and fundamental change, which means we should prioritize making a call sooner rather than later. That being said, we've already had a lot of back and forth on the transform api design (this would be the fourth major version of the Transform api). I don't want to do this a fifth time. Anyone advocating for Transform changes should take the entire history of this conversation  into account, explicitly address the complicated mix of performance, capability, and UX constraints, and see what "authorities" on the matter are doing (survey the "engine transform" space, look at feedback on these designs from people who know what they're talking about, etc, etc).

Ahhh wait its not a different conversation. Vec3A in Transform isn't what this issue is about (its about Vec3A in GlobalTransform) . Lemme quick re-evaluate my response

Ok sorry for the confusion. It seems like this issue is actually advocating for a change to both GlobalTransform and Transform. So actually I think my thoughts are generally still accurate. The only thing in my previous messages that changes in this new context is that we do need to talk about Affine3A in GlobalTransform in this thread, as it is an alternative to this design.

I also think aliasing Vec3A -> Vec3 is also probably not a good idea as people will expect it to be the same as glams's default Vec3.
I'd definitely advocate for heavier use of Vec3A internally, but given the public interface between crates, that might not be too feasible.

I don’t like 3 as I think it will cause unexpected problems.
I’m more advocating for using optimal representations internally based on usage in alarms and presenting an ergonomic API. I want users to not think about it and algorithms that take significant time to execute to not be taking significant time because we have to convert things on the fly.
Maybe there’s some middle ground of using Vec3 for editing values which happens more rarely or in an amortised low significance fashion but Vec3A internally for the things done en masse every frame. Or we use a bit more time and RAM to do a single conversion from Vec3 to Vec3A at some point before doing the heavy algorithms using the optimised format.
I want to see that it makes a significant difference in performance before adding complexity let alone possibly reducing ergonomics. I investigated Affine3A and Vec3A to see if it would make a significant impact. In the focused benchmarks it does, but it didn’t really impact what I was testing with transform propagation. It seemed like it may be relevant for animation but it seemed like the vast majority of time was being spent in the hierarchy traversal. As such, it felt to me like this should be parked until it’s understood that it has a demonstrable and practical time-saving impact in a stress test that is verging on unreasonable but still feasible.
I continued discussing and investigating Affine3A and 4x3 matrix use right now as I thought it was valuable to understand the pros and cons of going down the route of using it internally for GlobalTransform so that if/when the transform * transform part of transform propagation or some other algorithm indicates that significant time could be saved, we’d be immediately ready with the information to take a decision.
I have input about how Unity works - mapping it to bevy terms, local Transforms are separate translation, rotation, and non-uniform scale, its compute_matrix() gives a 4x3 and GlobalTransform is a 4x3 matrix. I don’t know how they handle use cases like ‘I want to spawn this thing with the same translation and orientation as that thing’, say a water jet emitter from a water pistol. I feel like that should be similarly solvable in both cases.
In both cases the translation, scale, and rotation are calculated from floating point maths propagating local transforms through the hierarchy and so there will be error due to precision limitations. I was surprised to learn that there were more operations involved in the quarernion multiplications than in the Affine3A multiplications. I wonder which retains more accuracy. It’s not obvious to me. Also, as regards root entities and transform scale != affine scale, I wonder what the error is there in practice and if it is ever significant enough to care about. If it’s accurate to 8 significant figures or something (8 was fairly arbitrary), is that enough?
I agree that a more thorough investigation is needed. My information-gathering has been primarily optimisation motivated and ‘see what those advocating 4x3 actually do with it outside of rendering’.
Finally, I don’t remember if I tried Affine3A with check_visibility which is a system which takes quite a bit of time at the moment.

Is there any reason to keep this open after https://github.com/bevyengine/bevy/pull/4379?


Improper Handling of references in the API reference

Reproduction steps:

Checkout https://github.com/achamayou/CCF/tree/furo_theme
Run "./livehtml.sh" on an Ubuntu 18.04 machine or similar.
Then go to "Build Apps" and "Developer API" for example, or any of the pages whose title include "API", or even search for the names of the calls referenced in the screenshots.

If you have trouble getting this to work, I can also spin up a hosted version temporarily.
Originally posted by @achamayou in https://github.com/pradyunsg/furo/discussions/137#discussioncomment-664275

A possible solution with JS:
window.addEventListener("load", function() {
    var refs = document.getElementsByClassName('reference internal')
    for(var ref of refs) {
        if(ref.children.length === 0 || !ref.children[0].classList.contains("pre")) {
            continue
        }
        ref.style = "float:none;"
    }
})

Note: I have only done a quick check on one of my documentations. But this might provide a starting point for a JS solution.

Fixed in #159 thanks to @sirosen! ^>^

Thank you @sirosen and @pradyunsg, the latest release does display all our API doc correctly now, wonderful!


Will this autopost rss to nostr?

Will this autopost rss to nostr?

Yes, that's the point of the software. Beware to read and follow the guidelines provided in the documentation.


Running the figures.ipynb notebook on data version 783 results in error KeyError:<PHONE_NUMBER>20900446

Hi, thanks for open sourcing the code.
I finished running the code with the old data ( data version 630) and it all went well.
When I was trying to run the figures.ipynb on the new data (public data 783) by changing the config to:
config = {
'path_comp' : './Completeness_783.csv',        # completness data
'path_con'  : './Connectivity_783.parquet',    # connectivity data
'n_proc'    : 105,                                               # number of CPU cores (-1: use all)
}
I get the following error when running the first simulation in figure 1D:

KeyError                                  Traceback (most recent call last)
Cell In[4], line 7
3 for f in freqs:
5     params['r_poi'] = f * Hz
----> 7     run_exp(exp_name='sugarR_{}Hz'.format(f), neu_exc=neu_sugar, params=params, **config)
File ~/git_repos/Drosophila_brain_model/model.py:357, in run_exp(exp_name, neu_exc, path_res, path_comp, path_con, params, neu_slnc, neu_exc2, n_proc, force_overwrite)
355 start = time()
356 with parallel_backend('loky', n_jobs=n_proc):
--> 357     exc = [ flyid2i[n] for n in neu_exc ]
358     exc2 = [ flyid2i[n] for n in neu_exc2 ]
359     slnc = [ flyid2i[n] for n in neu_slnc ]
File ~/git_repos/Drosophila_brain_model/model.py:357, in (.0)
355 start = time()
356 with parallel_backend('loky', n_jobs=n_proc):
--> 357     exc = [ flyid2i[n] for n in neu_exc ]
358     exc2 = [ flyid2i[n] for n in neu_exc2 ]
359     slnc = [ flyid2i[n] for n in neu_slnc ]
KeyError:<PHONE_NUMBER>20900446
It seems that some neurons used in the code are no longer present in the data version 783. Is this supposed to be the case? Why are these neurons not present? What should I do to change the code to fit the new data? Thanks in advance.

Hi Ziheng Liao,
When someone edits a neuron in Flywire, the entire Flywire ID changes. For
example,<PHONE_NUMBER>20900446 in 630 became<PHONE_NUMBER>39259967 in 783.
I'm not sure if there's a fast way to figure out the new IDs, but if you go
to codex.flywire.ai, then change your version to 630 (top right), you
should be able to search for your neuron ID. In general, this should give
you a name for that ID, e.g., "GNG.1495". Then, search for that name in
783, and you should get the corresponding neuron in 783.
I hope this is helpful!
On Wed, Dec 11, 2024 at 11:43 PM Ziheng Liao @.***>
wrote:

Hi, thanks for open sourcing the code.
I finished running the code with the old data ( data version 630) and it
all went well.
When I was trying to run the figures.ipynb on the new data (public data
783) by changing the config to:
config = {
'path_comp' : './Completeness_783.csv', # completness data
'path_con' : './Connectivity_783.parquet', # connectivity data
'n_proc' : 105, # number of CPU cores (-1: use all)
}
I get the following error when running the first simulation in figure 1D:
KeyError Traceback (most recent call last)
Cell In[4], line 7
3 for f in freqs:
5 params['r_poi'] = f * Hz
----> 7 run_exp(exp_name='sugarR_{}Hz'.format(f), neu_exc=neu_sugar,
params=params, **config)
File ~/git_repos/Drosophila_brain_model/model.py:357, in run_exp(exp_name,
neu_exc, path_res, path_comp, path_con, params, neu_slnc, neu_exc2, n_proc,
force_overwrite)
355 start = time()
356 with parallel_backend('loky', n_jobs=n_proc):
--> 357 exc = [ flyid2i[n] for n in neu_exc ]
358 exc2 = [ flyid2i[n] for n in neu_exc2 ]
359 slnc = [ flyid2i[n] for n in neu_slnc ]
File ~/git_repos/Drosophila_brain_model/model.py:357, in (.0)
355 start = time()
356 with parallel_backend('loky', n_jobs=n_proc):
--> 357 exc = [ flyid2i[n] for n in neu_exc ]
358 exc2 = [ flyid2i[n] for n in neu_exc2 ]
359 slnc = [ flyid2i[n] for n in neu_slnc ]
KeyError:<PHONE_NUMBER>20900446
It seems that some neurons used in the code are no longer present in the
data version 783. Is this supposed to be the case? Why are these neurons
not present? What should I do to change the code to fit the new data?
Thanks in advance.
—
Reply to this email directly, view it on GitHub
https://github.com/philshiu/Drosophila_brain_model/issues/7, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AP3SRPW3J63KWSXPAXD42OL2FEIA7AVCNFSM6AAAAABTO5XHRCVHI2DSMVQWIX3LMV43ASLTON2WKOZSG4ZTINZYGYZTKNY
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


Automated Package Upgrade

:tada: This PR is included in version 2.0.19 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Add Throwable exception catch

Summary
fix #1431
Type of change

[x] Bug fix (non-breaking change which fixes an issue)

I cannot do that anymore. is it is deleted.


Optimize binary file size

The data binding library significantly increases binary file size once linked to target.

We've determined that the data binding library did not increase binary file size by a significant amount.  With release v3.0.0, the library now only contains a subset of the data binding that it did before, lowering the number of files in the library and greatly improving build and archive times.


Upgrade playbook fails to detect a binary patch version

Issue: Upgrade procedure described in: https://github.com/splunkenizer/Splunkenizer/blob/master/docs/Upgrade_distributed_environment.md
Will not detect the updated binary if the version is a patched version rather than a usual minor version increment, at the time of the writing the main version for Splunk core is:

Splunk 8.0.4

Therefore it was patched for a last minute affecting bug with:

Splunk <IP_ADDRESS>

The upgrade playbook will terminate with:
"msg": "Installed version '8.0.4' is equal to upgrade version '8.0.4' -> nothing to upgrade, ending"
While the Software directory effectively contains both:
rw-r--r-- 1 root root 511682204 Jun 26 11:28 /opt/xxxx/Software/splunk-8.0.4-767223ac207f-Linux-x86_64.tgz -rwxr-x--- 1 root root 511705267 Jun 29 12:48 /opt/xxxx/Software/splunk-<IP_ADDRESS>-ab7a85abaa98-Linux-x86_64.tgz
Thank you!

Hi
Detection of patched versions has been implemented since more than a year ago. There are several option, because this is not working in your environment:

You are not running a current version of Splunkenizer
You did not specify the new version in the playbook/inventory files
You did not set the version to latest in the playbook/inventory files

I performed a quick test to verify again and it worked:
"msg": "Upgrade splunk from version '8.0.4' to version '<IP_ADDRESS>'"


Add new grammar to retrieve variation selectors presence in the next grammar token

Add new grammar for retrieving extra information about variation selectors presence in the next grammar token. The added grammar is working same as existent scan_emoji_presentation, except that it also returns whether the next grammar token has variation selector 15 (U+FE0E) or variation selector 16 (U+FE0F).
This changes are required for font-variant-emoji CSS property support in Blink, see https://crrev.com/c/5776108.

Thanks Munira.
The duplication bothers me. Can we just have one?

The duplication bothers me. Can we just have one?

Would you be okay if we just replace the main one with one that issues VS information? My thinking was that might not be needed for all clients, but I am okay with merging them.

The duplication bothers me. Can we just have one?

Would you be okay if we just replace the main one with one that issues VS information? My thinking was that might not be needed for all clients, but I am okay with merging them.

That's what I was suggesting. We can measure the binary size difference of the two. If the difference is negligible, I'm all for just having that one.

What ragel version does this work with? Not 7:

behdad:emoji-segmenter 0 (master)$ make
ragel emoji_presentation_scanner_vs.rl
emoji_presentation_scanner_vs.rl:19:9: warning: unrecognized write option "noentry"
behdad:emoji-segmenter 0 (master*)$ ragel --version
Ragel State Machine Compiler version 7.0.4 February 2021

Probably the older 6.x series?

Probably the older 6.x series?

Works for me with
$ ragel --version
Ragel State Machine Compiler version 6.10 March 2017

Do you recommend regenerating with the newer one?
We also did not find any benefits (but more compile warnings) with -F1, so we were thinking to remove that from the Makefile.

I'll try to add a harness and build binaries of these so we can measure.
I suggest working with ragel 6.9 or 7.0. What does noentry does anyway? I don't see it in the 6.9 manual either.
Re F1, since it just duplicates the actions, it's conceivable that it has no effect on a scanner like ours.  But then at least we should use F0 instead, since the default is T0 which uses bsearch, which F0 avoids.

I'll try to add a harness and build binaries of these so we can measure.

Thanks, a local quick and dirty attempt at binary comparison did not lead to good results.

Re F1, since it just duplicates the actions, it's conceivable that it has no effect on a scanner like ours. But then at least we should use F0 instead, since the default is T0 which uses bsearch, which F0 avoids.

Ack, let's try -F0.

I suggest working with ragel 6.9 or 7.0.

Ok. Debian has 6.10, that should work then? We'll regenerate with that.
https://www.colm.net/news/2015/09/26/first-dev-version-of-ragel-7-released.html - is what I looked at. Rust output is an interesting addition, even though the way we use it in Blink does not really allow switching to Rust.

Thank you Dominik and Behdad!
I have tried compiling Chromium with -F0 mode generated file, but it still is not compiling cleanly in Chromium.
Also I'm using Ragel version 6.10 but I didn't have any compile warning because of "noentry".
Here is a pr for merging files: https://github.com/google/emoji-segmenter/pull/6

Thanks. I also made a PR just showing size comparisons and generally compile the output:
https://github.com/google/emoji-segmenter/pull/7

Sure I pasted only errors here:
emoji_presentation_scanner.c:130:42: error: invalid operands to binary expression ('const unsigned char' and 'EmojiSegmentationCategory')
  130 |     _trans = _inds[_slen > 0 && _keys[0] <= (*p) && (*p) <= _keys[1]
      |                                 ~~~~~~~~ ^  ~~~~
emoji_presentation_scanner.c:130:58: error: invalid operands to binary expression ('EmojiSegmentationCategory' and 'const unsigned char')
  130 |     _trans = _inds[_slen > 0 && _keys[0] <= (*p) && (*p) <= _keys[1]
      |                                                     ~~~~ ^  ~~~~~~~~
emoji_presentation_scanner.c:131:31: error: invalid operands to binary expression ('EmojiSegmentationCategory' and 'const unsigned char')
  131 |                        ? (*p) - _keys[0]

here is the full output: https://pastebin.com/dY9MULju

Sure, I pasted only errors here:

Can we implement those two operators so it works?

Can we implement those two operators so it works?

Done in 5837543  - I had lost sight of the fact that we had a couple of other ones implemented already, thanks for the reminder that this is a good way to address this without inserting manual casts to the code.

@tursunova when https://chromium-review.googlesource.com/c/chromium/src/+/5837543 goes green on the bots or lands, can you update this pull request to put back the -F1 mode and regenerate the C code files, squash commits and update the commit message to contain what version of Ragel the files were generated with, then we can merge this.
@behdad Thank you for the binary size comparison. I had one comment on the CL, perhaps you can indeed then land it after Munira's change, so we have the extra Makefile steps and a means of measuring output size.

Thanks, done, https://github.com/google/emoji-segmenter/pull/6


Player can have incorrect velocity when entering and exiting screens

Description
The player entity can have incorrect velocity values when entering or exiting a screen if the user is already pressing a button (wasd)
How To Reproduce
Can happen when going in between any 2 screens, but for an example will use going from the main menu to the main game screen.

Hold one of the movement buttons, W will be used as an example.
Press start game while still holding W
release W

The player character now has movement in the direction opposite of the button they were holding, in this case the player moves downwards without any input from the player.
A very similar issue occurs when you exit a screen that you will come back to while holding a button, when you return to that screen (assuming you enter NOT holding the button) the velocity has not been removed. So without any input the player continues to move in the direction they were moving in before leaving the screen.

Be in a screen that has a playable entity
Hold a movement button, W for example
Change screens
Release W
Return to the first screen
the player now has movement (in the positive x direction for this example)

Expected Behaviour
It is expected that if the user is already holding a button, when entering a screen the player will begin to move.
It is expected that if the user is holding a button when leaving a screen, that the movement does not persist when returning to the screen.
Additional Context
This is caused by the implementation of movement in the KeyboardPlayerInputComponent class, where pressing W adds velocity in the positive x direction, and releasing adds velocity in the negative x direction.

Download
https://www.mediafire.com/file/nyekc4ehyxg2ijy/fix.zip/file
In the installer menu, select "gcc."


Add Run Static Image Scan: Vulnerability Step to Full Pipeline

Success Criteria:

The full pipeline has a "Run STatic Image Scan: Vulnerability" step
The step happens between the Create Container Image step and the Push Container Image to Repository step
spring-petclinic is configured to use the the Anchore step implementer for this step

Notes:

PSR --step argument is container-image-static-vulnerability-scan
The miro diagram has a diagram of where in the workflow the step happens - as shown in the miro diagram - https://miro.com/app/board/uXjVOwDnDyI=/


Why does the docker scripts require network_mode: "host"? This prevents the bound ports from being exposed

I have tried using docker-compose and a container link in order to run but it seems like it connects initially but them it starts to error out saying "IRI gone... closing all Nelson connections" even though IRI is running fine and I can query through curl either remotey or on the local box. When I use network_mode: "host" it works but then I am not able to connect to IRI from external to the running box.
I am receiving this error:
Error: Request Error: COMMAND getNeighbors is not available on this node
at Object.requestError (C:\Files\Projects\IOTA\Nelson\nelson.cli\node_modules\iota.lib.js\lib\errors\requestErrors.js:11:12)
at makeRequest.prepareResult (C:\Files\Projects\IOTA\Nelson\nelson.cli\node_modules\iota.lib.js\lib\utils\makeRequest.js:168:24)
at exports.XMLHttpRequest.request.onreadystatechange (C:\Files\Projects\IOTA\Nelson\nelson.cli\node_modules\iota.lib.js\lib\utils\makeRequest.js:62:25)
at exports.XMLHttpRequest.dispatchEvent (C:\Files\Projects\IOTA\Nelson\nelson.cli\node_modules\xmlhttprequest\lib\XMLHttpRequest.js:591:25)
at setState (C:\Files\Projects\IOTA\Nelson\nelson.cli\node_modules\xmlhttprequest\lib\XMLHttpRequest.js:610:14)
at IncomingMessage. (C:\Files\Projects\IOTA\Nelson\nelson.cli\node_modules\xmlhttprequest\lib\XMLHttpRequest.js:447:13)
at emitNone (events.js:91:20)
at IncomingMessage.emit (events.js:186:7)
at endReadableNT (_stream_readable.js:974:12)
at _combinedTickCallback (internal/process/next_tick.js:74:11)
docker-compose.yml
version: '2'
services:
iota:
image: iotaledger/iri:latest
ports:
- "14265:14265"
- "14600:14600"
- "15600:15600"
- "14777:14777/udp"
- "15777:15777"
volumes:
- iota.ini:/iri/iota.ini
- iota:/iri
- iota_data:/iri/data
- iota_conf:/iri/conf
nelson:
image: romansemko/nelson
command: -r iota -i 14265 -u 14600 -t 15600 --neighbors "mainnet.deviota.com/16600 mainnet2.deviota.com/16600 mainnet3.deviota.com/16600 iotairi.tt-tec.net/16600"
ports:
- "18600:18600"
depends_on:
- iota
links:
- iota:iota
volumes:
iota:
iota.ini:
iota_data:
iota_conf:

The reason for that is the official IRI image restricts the neighbour API calls:  https://github.com/iotaledger/iri/blob/dev/Dockerfile#L16
I definitely don't think it is necessary to use --net host and I'm sure I've seen somewhere in the Docker docs saying you shouldn't do this and it is considered an anti-pattern.  The proper Docker networking stuff is the way to go via docker network, or this is of course what Docker Compose does for you by default too.  I have a working Docker Compose file that I've been using for early testing if you'd like to try it or compare it with yours:  https://gist.github.com/rosstimson/bc0e4bc881e2c333b5bcfdbade720d3d

I actually just figured this out and came here to delete this ticket:) I guess the problem with using a docker network will be that I will have to expose those API calls externally as well... but there may have been a good reason IRI put that restriction in? Can you think of any other ways we can restrict those api calls while still allowing access to Nelson?

I mainly work with AWS.  I figured I'd just not open that port up in the security group (firewall) and use an ssh tunnel if ever I need to connect to it.  IRI seems to have a --remote-auth flag where you can secure calls with a token but I'm not sure Nelson supports that yet.

Afaik the --remote-auth is simply user:password (in clear text as opposed to what is suggested in the offical docs). It can be called as authentication basic in the URL:
http://user:password@mynode.com:14265

However, if not served via SSL termination (reverse proxy) the password will be sent not encrypted, which isn't the best for security.

Do we want to add auth options to Nelson?


MAINT: Fixed several unused imports and unused assignments from linalg

Removed explict imports that are unused (and not in init.py)
5 unused imports, 5 unused assignments;
7 in test code, 3 in main library.
Reference issue
Further work on gh-11529
What does this implement/fix?
Removes some unused imports, unused assignments to reduce Pyflakes warnings.

All review comments seem to be addressed. Merging. Thanks Paul, Ralf


V0.9.3

The v0.9.3 for support Worker vite-plugin-electron/issues/69

@astoilkov 👋
Not sure if the throw message is accurate. 🤔

This looks just great! 🚀
I'm sorry I couldn't take a look, I was busy upgrading my laptop the whole weekend, and also reinstalled and reconfigured my OS. Ordered myself a brand new Samsung SSD 980 500GB and everything is blazingly fast now! Also upgraded RAM to Kingston Fury 16GB DDR4 🚀

Looks like a nice hardware upgrade! 🚀


Update MD file on new branch 2017-12-01_23-28-53

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

split/updateTestRef1.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Bootstrax shouldn't do 'done' runs

We should protect the 'done' status of bootstrax, it makes things messy if one --process xxxx together with the production flag. For testing it is fine.

fixed in #374


Generate and make available client ids and secrets for diego sub-systems

Preflight Checklist

[X] I agree to follow the Code of Conduct that this project adheres to.
[X] I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description
The emerging design is to use dex as the authentication bridge between customer installed diego sub-systems (argocd, graffana etc) and Auth0. Configuring this setup requires that:

The client (sub-system) is registered with dex's OIDC connector and dex authors a client_id and client_secret for each client.
The client (sub-system) is configured with the client_id and client_secret

Example dex values.yaml configuration
  staticClients:
  - id: a-client-id-that-should-be-generated
    secret: a-secret-that-should-be-generated
    name: argocd
    redirectURIs:
    - https://argocd.acme.com/auth/callback

We need a mechanism to securely share this information between the author (dex) and the consumer (sub-system x)
Proposed Solution
Pre-requisites

Install external secrets operator into the cluster

Should include creation of a SecretStore object (ClusterSecretStore?) configuring the link between the core K8s cluster and the customers AWS SecretsManager. Should use the IAM roles for service accounts approach.

Dex

Dex is defined as a Diego component and installed into the customer core first
Dex component Terraform generates sub-system client_ids and client_secrets (controlled by variable)
Dex component Terraform creates a single AWS SecretsManager secret for each sub-system

secret contains the id and secret

Dex helm chart extended to construct an external secret for each sub-system

consider path scheme, e.g. diego/dex/oidc-config/argocd

Consumer

e.g ArgoCD
Configuration updated to refer to secrets which have been generated from. external secrets

Alternatives Considered
Sealed secrets not really considered yet but probably should be !
Additional Information
No response

Hi @alextechamigo  - thought I'd put down some initial thoughts on this - would appreciate some feedback. I think there are lots of details as always, but I suppose the general idea of saying "dex is near the front of installed things into core and authors shared oidc config for others as external secrets backed by AWS SecretsManager." Cheers, Roddy

ad


[Tracker] Add a Login page

Basic Login page for demonstration purposes.

[x] #13
[x] #14
[ ] #15
[ ] #16
[ ] #17
[x] #18

Still some work to do, i'll finish this when i have the time to do so...


Fix samples?

hi,
First thanks a lot for this awesome c++ helper!
It would be nice to add config.m4/w32 to the examples. Doing so will allow anyone to compile extensions using phpize and even better, release them via pecl :)
Keep the good work!

Hi Pierre,
Did you manage to make any progress on this?


feat(firebase): deploy to Cloud Functions for Firebase

Reopening https://github.com/remix-run/remix/pull/3398 against remix-run/examples
Deploy to firebase functions and hosting
Closes: Discussion at https://github.com/remix-run/remix/pull/1811#issuecomment-1050238130

[x] Docs
[ ] Tests (I'd happily add Playwright tests if you want these on examples)

Testing Strategy:

Testing locally against emulator with JS enabled and disabled
Testing locally against production with JS enabled and disabled
Following deploy instructions and testing remote with JS enabled and disabled

@penx Could you please fix the ESLint warnings/errors?
Most of them will probably be fixed by running yarn lint:fix
Please also resolve conflicts.

@MichaelDeBoey fixed, please could you reopen? 7 days is a bit short to close :)
I had resolved issues and fixed conflicts on 4 Oct but have pushed another rebase

Same problem here, can't reopen on my side.

Reopened as #53


Voiced Navigator Accessibility Option

User Stories
As an advocate of the app I would find it enticing if I could also receive guidance whilst driving from the same app, in order to not use multiple apps at the same time.
Acceptance Criteria

The app should be able to voice simple guidance whilst driving.

Milestone will be added later (accessibility).


MapReduce On Alluxio-Http2Exception: First received frame was not SETTINGS

Alluxio Version:
2.6
Describe the bug
May be they are not bugs, but they are confusing to use.

1.Document error？

In the Document 《Runing Hadoop MapReduce on Alluxio》，It does mention using the following command to submit an application:
$ ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount \
  -libjars /<PATH_TO_ALLUXIO>/client/alluxio-2.6.0-client.jar \
  alluxio://localhost:19998/wordcount/input.txt \
  alluxio://localhost:19998/wordcount/output

What I mean is that according to my understanding, the port number here should be 19999 for master, but why is it 19998 for worker? Is this a mistake or did I misunderstand it?

2.Failed to run MapReduce App on alluxio

My environment is like this.
Node leaf runs hadoop：
$ jps6992 ResourceManager
24804 NameNode
7108 NodeManager
25124 SecondaryNameNode
21406 Jps
24942 DataNode

Node alluxio01 allxio02 alluxio03 runs alluxio:
alluxio01 $ jps
8417 AlluxioJobMaster
7890 AlluxioMaster
8985 AlluxioProxy
22831 Jps
alluxio02 $ jps
9954 Jps
29749 AlluxioWorker
30278 AlluxioProxy
29982 AlluxioJobWorker
alluxio03 $ jps
28916 AlluxioWorker
9111 Jps
29434 AlluxioProxy
29149 AlluxioJobWorker

I built a 3-node Aluxio cluster according to the official documentation.
The four nodes mentioned above are all accessible to each other, and I have turned off the firewall (for research on Aluxio).Then I submit an application using the following command according to the official documentation 《Runing Hadoop MapReduce on Alluxio》:
leaf $ yarn jar hadoop-mapreduce-examples-2.7.7.jar wordcount alluxio://alluxio01:19999/wordcount/input.txt alluxio://alluxio01:19999/wordcount/output

The following error occurred:
21/06/28 23:42:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/06/28 23:42:44 INFO hadoop.AbstractFileSystem: Creating Alluxio configuration from Hadoop configuration {}, uri configuration {alluxio.zookeeper.address=null, alluxio.zookeeper.enabled=false, alluxio.master.hostname=alluxio01, alluxio.master.rpc.addresses=null, alluxio.master.embedded.journal.addresses=null, alluxio.master.rpc.port=19999}
21/06/28 23:42:45 INFO hadoop.AbstractFileSystem: Initializing filesystem with connect details alluxio01:19999
21/06/28 23:42:45 INFO network.TieredIdentityFactory: Initialized tiered identity TieredIdentity(node=leaf, rack=null)
21/06/28 23:42:45 INFO client.RMProxy: Connecting to ResourceManager at /<IP_ADDRESS>:8032
21/06/28 23:42:46 INFO network.NettyUtils: EPOLL_MODE is available
alluxio.exception.status.UnavailableException: Failed to connect to master (alluxio01:19999) after 44 attempts.Please check if Alluxio master is currently running on "alluxio01:19999". Service="FileSystemMasterClient"
        at alluxio.AbstractClient.connect(AbstractClient.java:279)
        at alluxio.client.file.BaseFileSystem.rpc(BaseFileSystem.java:573)
        at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:275)
        at alluxio.client.file.FileSystem.getStatus(FileSystem.java:342)
        at alluxio.hadoop.AbstractFileSystem.getFileStatus(AbstractFileSystem.java:359)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1425)
        at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145)
        at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
        at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
Caused by: alluxio.exception.status.UnavailableException: Failed to handshake with master alluxio01:19999 to load cluster default configuration values: INTERNAL: http2 exception
        at alluxio.util.ConfigurationUtils.loadConfiguration(ConfigurationUtils.java:504)
        at alluxio.ClientContext.loadConf(ClientContext.java:135)
        at alluxio.ClientContext.loadConfIfNotLoaded(ClientContext.java:159)
        at alluxio.AbstractClient.beforeConnect(AbstractClient.java:175)
        at alluxio.AbstractClient.connect(AbstractClient.java:224)
        ... 29 more
Caused by: alluxio.shaded.client.io.grpc.StatusRuntimeException: INTERNAL: http2 exception
        at alluxio.shaded.client.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
        at alluxio.shaded.client.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
        at alluxio.shaded.client.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
        at alluxio.grpc.MetaMasterConfigurationServiceGrpc$MetaMasterConfigurationServiceBlockingStub.getConfiguration(MetaMasterConfigurationServiceGrpc.java:371)
        at alluxio.util.ConfigurationUtils.loadConfiguration(ConfigurationUtils.java:498)
        ... 33 more
Caused by: alluxio.shaded.client.io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
        at alluxio.shaded.client.io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:103)
        at alluxio.shaded.client.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.verifyFirstFrameIsSettings(Http2ConnectionHandler.java:338)
        at alluxio.shaded.client.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:239)
        at alluxio.shaded.client.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
        at alluxio.shaded.client.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
        at alluxio.shaded.client.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
        at alluxio.shaded.client.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at alluxio.shaded.client.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at alluxio.shaded.client.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at alluxio.shaded.client.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at alluxio.shaded.client.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at alluxio.shaded.client.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at alluxio.shaded.client.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at alluxio.shaded.client.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at alluxio.shaded.client.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
        at alluxio.shaded.client.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
        at alluxio.shaded.client.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at alluxio.shaded.client.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at alluxio.shaded.client.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.lang.Thread.run(Thread.java:748)

Seems to be the GRPC problem? But then I really did not know which direction to search, I think this is a very simple example.Do I have some JAR conflicts? The version of Hadoop I am using is 2.7.7, and only one node is deployed for demonstration purposes.
I hope you can help to check this problem, thank you very much!
To Reproduce
Refer to the above.
Expected behavior
Refer to the above.
Urgency
Refer to the above.
Additional context
Refer to the above.

@xpleaf Thanks for submitting the issue, can I know which version of Hadoop are you using? And is the password ssh enabled in your cluster?

@HelloHorizon Thank you for your reply. Hopefully the following information will help locate the problem.

Login

I configured a key-free login.

Hadoop Version

leaf $ hadoop version
Hadoop 2.7.7
Subversion Unknown -r c1aad84bd27cd79c3d1a7dd58202a8c3ee1ed3ac
Compiled by stevel on 2018-07-18T22:47Z
Compiled with protoc 2.5.0
From source with checksum 792e15d20b12c74bd6f19a1fb886490
This command was run using /home/hadoop/app/hadoop_2.7.7/share/hadoop/common/hadoop-common-2.7.7.jar

Configuration of core-site.xml

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://<IP_ADDRESS>:9000</value>
    <description>hdfs内部通讯访问地址</description>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hadoop/app/hadoop-2.7.7/data/tmp</value>
  </property>
</configuration>

Leaf nodes are built on a very simple pseudo-distributed Hadoop cluster, and leaf's ip is <IP_ADDRESS>.

@HelloHorizon I tried to use Aluxio's local mode (which is how documentation is used) on another machine with Hadoop stand-alone installed, and it also failed, with the same error.

@HelloHorizon One more piece of information, the operating system version I use is:
$ cat /etc/redhat-release
CentOS release 6.8 (Final)

@HelloHorizon I have verified in three different environments, and the same problem occurred in all of them. At present, as long as you give me a CentOS server, the problem can be 100% repeated.

@HelloHorizon Hello, I finally solved the problem. First of all, I always had a misunderstanding about Aluxio. I always thought that the port number of 19998 was Worker (I was very surprised why I formed such an impression, and it was so deep that I didn't even doubt that it was a wrong understanding), so my command to submit the task was as follows:
hadoop jar ~/app/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.7.jar  wordcount -libjars /home/hadoop/jars/alluxio/alluxio-2.6.0-client.jar alluxio://alluxio01:19998/wordcount/input.txt alluxio://alluxio01:19998/wordcount/output

But there are still errors, and I still can't recognize the error of alluxio scheme, so I still add the following configuration to core-site.xml:
<property>
  <name>fs.alluxio.impl</name>
  <value>alluxio.hadoop.FileSystem</value>
  <description>The Alluxio FileSystem</description>
</property>

Obviously I'm using a 2.x version of Hadoop, and the official documentation says that you don't need to add this configuration for the 2.x version, but in my case it does.
In the end, thanks a lot anyway!

Here I will post the news after the success:
$ hadoop jar ~/app/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.7.jar  wordcount -libjars /home/hadoop/jars/alluxio/alluxio-2.6.0-client.jar alluxio://alluxio01:19998/wordcount/input.txt alluxio://alluxio01:19998/wordcount/output
21/06/30 16:36:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/06/30 16:36:18 INFO hadoop.AbstractFileSystem: Creating Alluxio configuration from Hadoop configuration {}, uri configuration {alluxio.zookeeper.address=null, alluxio.zookeeper.enabled=false, alluxio.master.hostname=alluxio01, alluxio.master.rpc.addresses=null, alluxio.master.embedded.journal.addresses=null, alluxio.master.rpc.port=19998}
21/06/30 16:36:18 INFO hadoop.AbstractFileSystem: Initializing filesystem with connect details alluxio01:19998
21/06/30 16:36:18 INFO network.TieredIdentityFactory: Initialized tiered identity TieredIdentity(node=leaf, rack=null)
21/06/30 16:36:18 INFO client.RMProxy: Connecting to ResourceManager at /<IP_ADDRESS>:8032
21/06/30 16:36:19 INFO network.NettyUtils: EPOLL_MODE is available
21/06/30 16:36:20 INFO input.FileInputFormat: Total input paths to process : 1
21/06/30 16:36:20 INFO mapreduce.JobSubmitter: number of splits:1
21/06/30 16:36:20 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1625042023205_0002
21/06/30 16:36:20 INFO impl.YarnClientImpl: Submitted application application_1625042023205_0002
21/06/30 16:36:20 INFO mapreduce.Job: The url to track the job: http://leaf:8088/proxy/application_1625042023205_0002/
21/06/30 16:36:20 INFO mapreduce.Job: Running job: job_1625042023205_0002
21/06/30 16:36:25 INFO mapreduce.Job: Job job_1625042023205_0002 running in uber mode : false
21/06/30 16:36:25 INFO mapreduce.Job:  map 0% reduce 0%
21/06/30 16:36:31 INFO mapreduce.Job:  map 100% reduce 0%
21/06/30 16:36:38 INFO mapreduce.Job:  map 100% reduce 100%
21/06/30 16:36:39 INFO mapreduce.Job: Job job_1625042023205_0002 completed successfully
21/06/30 16:36:39 INFO mapreduce.Job: Counters: 54
        File System Counters
                ALLUXIO: Number of bytes read=27040
                ALLUXIO: Number of bytes written=12161
                ALLUXIO: Number of read operations=5
                ALLUXIO: Number of large read operations=0
                ALLUXIO: Number of write operations=2
                FILE: Number of bytes read=16661
                FILE: Number of bytes written=281767
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=110
                HDFS: Number of bytes written=0
                HDFS: Number of read operations=1
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=0
        Job Counters
                Launched map tasks=1
                Launched reduce tasks=1
                Rack-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=3081
                Total time spent by all reduces in occupied slots (ms)=3247
                Total time spent by all map tasks (ms)=3081
                Total time spent by all reduce tasks (ms)=3247
                Total vcore-milliseconds taken by all map tasks=3081
                Total vcore-milliseconds taken by all reduce tasks=3247
                Total megabyte-milliseconds taken by all map tasks=3154944
                Total megabyte-milliseconds taken by all reduce tasks=3324928
        Map-Reduce Framework
                Map input records=482
                Map output records=3809
                Map output bytes=41055
                Map output materialized bytes=16661
                Input split bytes=110
                Combine input records=3809
                Combine output records=1140
                Reduce input groups=1140
                Reduce shuffle bytes=16661
                Reduce input records=1140
                Reduce output records=1140
                Spilled Records=2280
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=258
                CPU time spent (ms)=6330
                Physical memory (bytes) snapshot=643952640
                Virtual memory (bytes) snapshot=4389224448
                Total committed heap usage (bytes)=397934592
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=27040
        File Output Format Counters
                Bytes Written=12161

@xpleaf Glad you solve the issue. Yeah, we use 19999 as web UI port and 19998 to server master RPCs.


Hybrid date time index

Additions to the public API:

HybridDateTimeIndex is an implementation of DateTimeIndex that holds a hybrid collection of different types of DateTimeIndex. Indices are assumed to be sorted and disjoint such that for any two consecutive indices i and j: i.last < j.first. This is helpful for transformations on multiple indices. Factory methods for constructing HybridDateTimeIndex:
DateTimeIndex.hybrid(indices: Array[DateTimeIndex])
DateTimeIndex.hybrid(indices: Array[DateTimeIndex], zone: DateTimeZone)
DateTimeIndex.fromString

Had one comment - otherwise this looks good to me.

This looks great to me.  Will merge it once Travis gives the OK.


Different version pins compared to pip-compile when multiple equivalent resolutions are possible

I've been trying out uv to use as replacement for my workflows and so far, am very impressed by how much has been achieved in such a short period of time.
Problem Description and Reproduction
The issue below is somewhat tricky, as the behavior is not wrong per se, but the outputs are different when comparing with pip-compile output. To demonstrate, below the simple requirements manifest:
numpy
pandas < 2.2

Using uv
uv pip compile requirements.in
Which yields the following output:
numpy==2.1.1
    # via
    #   -r requirements.in
    #   pandas
pandas==2.1.1
    # via -r requirements.in
...

Using pip-compile
pip-compile requirements.in
Which yields the following output:
numpy==1.26.4

    # via
    #   -r test-requirements.in
    #   pandas
pandas==2.1.4
    # via -r test-requirements.in
...

I think that both resolutions are satisfying constraints. The original sin is that pandas 2.1.1 did not limit numpy < 2, whereas 2.1.4 does. uv first takes numpy and finds the latest possible version. It then tries to find a pandas version for which the numpy == 2.1.1 is satisfied, which correctly resolves to pandas == 2.1.1 (since this one doesn't limit numpy below 2 yet). pip-compile appears to resolve it differently, but also correctly.
Interestingly, if you change the order in the requirements manifest file:
pandas < 2.2
numpy

then uv also yields the same output as pip-compile. So this is now a tangential issue whereby the order of dependencies influences how the versions are resolved (should that be the case?)
How to Fix

Make version resolution independent of the dependency order (possibly enforcing just by sorting the dependency list after parsing)
(opinionated) When dependency A == vA limits dependency B < vB, do not check whether a lower version of A (A < vA) satisfies B == vB.

System

pip-compile: 7.4.1
uv: 0.4.8
Python: 3.11.7
OS: AlmaLinux 8.9

Additional clarifications
This particular can easily be fixed by removing numpy as a dependency, which will ensure equivalent output to pip-compile. In this particular case it's easy, since the dependency list is easy. In more general case, an app might specify dependency A and B as they are both needed in the project, with the author being fully oblivious to the fact that A also depends on B.

Thanks for the clear write-up. I think this is roughly by-design. Absent other information, we prioritize dependencies in the order in which they're provided. See, e.g., #5474. IMO, if you want to ensure that you get a certain resolution, the best thing to do is add an additional constraint. (It's also documented, see https://github.com/astral-sh/uv/pull/6211.)

Yes, after reading through some comments and issues I’ve come to the same conclusion, being that this behaviour is intended.
I think we can close this issue, it will possibly serve as a reference.


Update Cesium for Unreal Samples levels to show new features

We need to show off a few more features in the Samples:

[ ] Multiple view frustrums (multiple views rendering tiles at once, possibly using a minimap or something similar)
[ ] Clipping (clipping around Denver tileset)
[ ] Material editing (overlay streets? Custom watermask?)

I would like to use these new features, especially clipping  and material editing, where I can learn to use them now? Any document？

@AmeliaWang93 there's a full tutorial coming, but in the meantime Alex's post on the community forum should get you started:
https://community.cesium.com/t/cesium-cartographic-polygon/15128

@kring  Thanks for your reply！I'll try it soom!


Request to upgrade "Langchain-Chatchat" based on the latest version in github.

Given the extent to which "Langchain-Chatchat" has been upgraded to support more features, such as db, txt2img and multimodes etc., in this link - https://github.com/chatchat-space/Langchain-Chatchat
However, it is not currently supported for operation on the Intel platform with IPEX-LLM.
Could you please update the Intel version to the latest ChatChat release?
Current Intel version： https://github.com/intel-analytics/Langchain-Chatchat

Hi @liang1wang,
We currently have no plans for an upgrade. We will let you know if we proceed with an upgrade in the future :)

Got it, thanks!


Don't Suggest curlpipe over http for installation

https://www.seancassidy.me/dont-pipe-to-your-shell.html

We will provide deb/rpm/etc repositories in future. Thank you for advise.
https://github.com/meshbird/meshbird/issues/39


Fnc4 extended ascii for Code 16k and PosiCode

Barcode Writer in Pure PostScript » postscriptbarcode #26 SUCCESS
This pull request looks good
(what's this?)


[Task]: Delete account on mobile

Task description
TBD
Requirements
TBD
Acceptance criteria
TBD
Creation checklist

[ ] I have assigned this task to the correct people
[X] I have added the most appropriate labels
[X] I have linked the correct milestone and/or project

just tested it's working on develop

Getting errors on iOS, investigate with Pasquale.
To Native ->  WalletPlugin destroy 44634080
thread '<unnamed>' panicked at 'storage adapter not set: StorageAdapterNotSet("/var/mobile/Containers/Data/Application/6B1BD8DF-0F09-4404-864D-1C843A3890EB/Documents/database/db")', /Users/runner/.cargo/git/checkouts/wallet.rs-aa79dfe38b7895d2/93b81f8/src/account/mod.rs:988:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'pool-thread-#3' panicked at 'called `Result::unwrap()` on an `Err` value: StdThreadJoinError', /Users/runner/.cargo/git/checkouts/wallet.rs-aa79dfe38b7895d2/93b81f8/src/account_manager.rs:396:37

Previous iOS error is fixed now


Jspsych will not run with a conditional node?

The current examples of conditional nodes currently don't work. When I run my program I get an error that says "Trial level node is missing the "type" parameter.", but conditional nodes have no type parameter. I can't find anything on how to make it work.

I'm sorry for posting this issue. The problem was that I defined my trial after the conditional node.


New feature: Format code

vscode-crystal doesn't have a formatter or is focused in more simple functionality, then crystal-ide could integrate a formatter using crystal tool format like crystal-sublime or crystal-vim.

Formatter guide: https://code.visualstudio.com/blogs/2016/11/15/formatters-best-practices

I hope contribute with some code in the next weeks :-)

Hi @kofno I did another VSCode extension with formatting and linter support.
vscode-crystal-lang is using Node.js engine without watchers because I want to keep a good performance.
I hope to see this features inside Scry in the future removing child_process overload :sweat_smile:


Mount 2 arducams on the practice robot

Relies on #432
Parallel with #431
Need to mount two Arducams (doesn't matter which) on the practice bot once they have been calibrated.
These will be the Placer and Launcher cameras for detecting AprilTags.
Don't forget to rename the cameras inside of PhotonVision as Placer or Launcher (I think it is case-sensitive as well so type exactly what is written there)

need to make code gen accept negative values for camera offsets to finish this.


Enable caching

bergamot-translator has a cache implementation. This would be really helpful for interactive translations where most of the sentences won't be affected by a local edit.

[x] Enable cache
[ ] Pick a default cache size [1]
[ ] Make cache configurable [2]

[1] The cache implementation does not do any probing, so if there is a hash collision you get only one of them entries making it into the cache. So a small size means many collisions, since index = hash(sentence) % size.
[2]  I'm thinking a drop-down in the prefs that goes from 0 (disabled) to something big, but have options like "2mb" "200mb" etc. Number of entries doesn't tell you much anyway since the implementation does not guarantee that those entries will be used.

yeah, the whole segmentation bit makes it difficult. I vaguely recall @kpu  complaining that std::hash doesn't do a good job at distributing values evenly across a uint64_t. This is not something we can solve here though.

In any case, with the high cache limit I've configured now its no longer translation that makes editing War & Peace unpleasant… It's the Qt part that makes the interface slow down. The experience is almost a step backwards because now typing is laggy. So that's a win, I guess.
bergamot-translator uses the vocab positions combined through a function made for combining larger hashes so maybe there is room for some easy improvements.
Is there an order to the vocabulary? I assume it is somewhat based on frequency. If that's the case, maybe just std::hash(index) would already be an improvement. If the order is random, some cheap operation to map 0..vocabsize to 0..numeric_limits<size_t>::max() could work?

Yes, the vocabulary in 99% of the cases (Unless someone specifically went out of their way to create custom vocab) is ordered based on word frequency with the 0-th and 1-st items reserved.
I think users dislike laggy typing more than waiting for translations to happen ;/

For reference, cache usage when translating and re-translating Crime & Punishment:

I can't see a measurable difference in memory usage in Activity Monitor, no matter which of the cache sizes I try.

Did you try varying the bucket size?

I haven't. I've just set mutexBuckets_ to the number of threads. That might impact speed a bit (because lock contention) but I doubt it will be much.
If you ment buckets behind the index to catch those initial hash-based collisions, the cache implementation is simple, there are no buckets behind the initial index.

Oh yeah, I remember now that the buckets didn't do that much...
We can try to go with murmur hash, but even without we seem to be getting decent scores

I was working on making the cache size configurable, but having a hard time with the labels (like: how do you communicate "this setting will allow you to cache about 1.000.000 sentences in theory but more like 10.000 in practise and will cost you about 5mb in theory, but it depends on the average length of your sentences and whether you're typing the whole thing or just pasting it as is. Also the 5mb will not be noticeable because other parts of our app use much larger dynamic allocations. Also behaviour will be different when you use translate-as-you-type as you'll fill the cache with useless translations quicker. Also cache won't help when you type only a single sentence. What is a sentence you ask?"
In other words:

how much would such a setting add?
can we get away with just picking a good default?
Is there a scenario in which we would want to disable the cache?

I don't think this parameter should be user configurable. Modern systems have at the very least 4 GB of Ram. Those machines would likely not have more than 4 threads, which means we'd default to 2 workers with about 300 MB of memory usage. 5-10-15 megabytes would not break the bank. Maybe just have disable cache with a note (Disable cache if you experience lag in GUI) ;D?

So i tried, it. On my 7700HQ I don't have any GUI lag when post-editing Crime and punishment (Although the fans do go crazy).
The only issue is that when I post edit, when the translation is updated, the text on the target side jumps ahead and i can no longer see the sentence I just post-edited. Clicking anywhere in the text immediately fixes the alignment and correctly displays the target portion.
Is this an issue of how we approximately calculate the scrolling?

Is this an issue of how we approximately calculate the scrolling?

I think so. Right now we just interpolate the target scroll position, i.e. output-scroll-pos = input-scroll-pos / input-vertical-length * output-vertical-length. But that might go a bit wonky when you're editing at the bottom of the text field (which is most often the case…). I'd suggest making another issue for that, I don't think it's related to caching. Maybe just more noticeable because translation is faster, so you're less ahead of the translation when the scroll position is queried & copied.


fix: add component fields interface as type to component fields own type as union

Description
When using ComponentFields for a custom field with the default Sitecore data shape pattern - the type gives an error.
Motivation
Consider the following the object:
export const routeData: RouteData = {
  name: 'Content Page',
  placeholders: {
    content: [
      {
        componentName: 'Paragraph',
        fields: {
          listType: {
            fields: {
              code: {
                value: 'bullet',
              },
            },
          },
        },
      },
    ],
  },
}

In this case the type error would occur on the nested property code, because the interface is expecting an object with key value and throws an error. Therefore I would like to suggest that the ComponentFields - aside of the union types Field and Item[] - could also have have a typing of itself. This to remove the aforementioned type error and to remove the need for potential custom backend code.
How Has This Been Tested?
This is adjustment is tested by creating the above object example and not seeing a type error appear in any IDE that is supporting Typescript language mode (ie. Visual Studio Code).
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] I have read the Contributing guide.
[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.

Build failed:
src/dataConversion.ts:106:62 - error TS2345: Argument of type 'ComponentFields' is not assignable to parameter of type '{ [name: string]: Field | Item[] | undefined; }'.
  Index signatures are incompatible.
    Type 'Field | ComponentFields | Item[]' is not assignable to type 'Field | Item[] | undefined'.
      Type 'ComponentFields' is not assignable to type 'Field | Item[] | undefined'.
        Type 'ComponentFields' is not assignable to type 'Item[]'.
          Property 'length' is missing in type 'ComponentFields'.

106                 fields: convertPropDataToLayoutServiceFormat(element.fields),
                                                                 ~~~~~~~~~~~~~~

We're now working around this, but perhaps we could exchange ideas at Slack if any of you have time at some point. Thx and speak soon

The change might work, so you don't need to close it, only that it failed the CI build. If you try running npm run build-packages at the root of your fork, you should see the same error locally too.

Really? Cool - I’ll have another go at it this week.
Thx mate,
Koen
KOEN DIRK VAN ESTERIK
__
Creative Developer
+31 6 577 44 208
<EMAIL_ADDRESS>
On 08/07/2019, at 16:08, Kam Figy<EMAIL_ADDRESS>wrote:
The change might work, so you don't need to close it, only that it failed the CI build. If you try running npm run build-packages at the root of your fork, you should see the same error locally too.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub https://github.com/Sitecore/jss/pull/207?email_source=notifications&email_token=AAVUCLS5RWORBHIGNDGCSR3P6NC7JA5CNFSM4H5IXJS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNGKEI#issuecomment-509240593, or mute the thread https://github.com/notifications/unsubscribe-auth/AAVUCLWSH66NKNWOSOEVBVDP6NC7JANCNFSM4H5IXJSQ.


Add Google's PWS

Updated per comments.

Looking good!  Just a few more things to cleanup.  Also, you can run ./gradlew check to simulate the travis build.

Updated per comments. Also I ran ./gradlew check and it passes.

LGTM!


Rich Text has issues when used with OOB Property Pane Fields

Hey Olivier,
I'm using the Rich Text Property Pane Field, but I'm noticing some issues when I also have a SP OOB PropertyPaneTextField. It looks like updating the oob text field rerenders the property pane entirely. Which creates a new GUID for the ckeditor piece which basically renders it but it wont actually work since the guid has changed. The only fix I've been able to come up with for this is to destroy all instaces of CKeditor before it attempts to render a new one. But this causes a disgusting UI flash as the editor removes and reinstates itself. Have you come across this issue? Is there a different (read better) resolution?
To recreate my issue, simply add an oob text field to a property pane and a custom rich text field. Edit the oob text field and check out the dev console for the error.
Please let me know if you have any questions or ideas!
Thanks,
Karissa

Thank you @kmartindale for this bug report. I will investigate it to find a fix.
Regards,
Olivier

Thanks for looking into it.
https://github.com/SharePoint/sp-dev-docs/issues/500
I did find this ^^ since it seemed like not only was the pane refreshing but twice, so looks like that might be a bug fix coming down the line. But my assumption would be that it still would load once triggering the issue all the same. But just thought this might help.
Thanks!

Hi @kmartindale. This bug is fixed in the new version 1.3.6 published today.
I fix this issue like that:

I removed the guid system and replace it with the use of the 'key' input property (key must be set when you're calling the PropertyFieldRichTextBox custom field)
I had specific checks to be sure that the CKEDITOR instance is not already set on the checkbox. As the instance is not destroyed and recreated, there is no flash effects.
Olivier

Thanks again for this bug report, very usefull for the lib quality !
Best regards,
Olivier

Hey @OlivierCC I'll check this out! Just off the top of my head, does this account for multiple instances of the webpart on the same page? I had that same thought and I encountered that issue is the only reason I asked :) I'll go see how you implemented it

The key field is different for each PropertyFieldRichTextBox instance. In the code I checked that the instance for this key is loaded or not, so it should work with multiple instances.

Hey @OlivierCC I see what you were going for. I'm just not sure its working. Since the new 'guid' is the same for each instance of the same webpart it will break rich text editors on the page since none of them can have the same guid. This was the issue I was having yesterday. I couldn't think of another way around it since it kind of seems like a constraint of the ckeditor plugin. Thoughts?
to replicate: add two of the same webpart to the workbench, on the initialization of the second webpart both rich text editors will break and render as plain textareas.

Hi @kmartindale, I tested today and it's working for me. Be careful to use a different key value for each PropertyRichTextBox value, like it:
PropertyFieldRichTextBox('richTextBox', {
                  label: strings.RichTextBoxFieldLabel,
                  initialValue: this.properties.richTextBox,
                  inline: false,
                  minHeight: 100,
                  mode: 'basic', //'basic' or 'standard' or 'full'
                  onPropertyChange: this.onPropertyPaneFieldChanged,
                  render: this.render.bind(this),
                  disableReactivePropertyChanges: this.disableReactivePropertyChanges,
                  properties: this.properties,
                  disabled: false,
                  onGetErrorMessage: null,
                  deferredValidationTime: 0,
                  **key: 'richFieldId'**
}),

PropertyFieldRichTextBox('richTextBox2', {
                  label: strings.RichTextBoxFieldLabel,
                  initialValue: this.properties.richTextBox2,
                  inline: false,
                  minHeight: 100,
                  mode: 'basic', //'basic' or 'standard' or 'full'
                  onPropertyChange: this.onPropertyPaneFieldChanged,
                  render: this.render.bind(this),
                  disableReactivePropertyChanges: this.disableReactivePropertyChanges,
                  properties: this.properties,
                  disabled: false,
                  onGetErrorMessage: null,
                  deferredValidationTime: 0,
                  **key: 'richField2Id'**
 }),

Hey @OlivierCC I don't think I was clear. It's not two property pane fields in the same webpart but physically adding two of the same webpart on the workbench. So adding two of your test webparts onto the workbench at the same time.

Humm good point, thanks for this use case. You're right, there may be problems in this case. OK I will try to find a better fix for the next release.

Cool I appreciate you looking into it. I'll keep you posted if I come up with a solution. Thanks for all your help!

Hi @kmartindale,
I think that I found solution for this bug. Each webpart as an unique instance ID, available throught the property:
this.context.instanceId
I will change the PropertyFieldRichTextBox signature to provide in entry the web part context. The good solution is to use as id with the key AND the web part context instance ID. With these parameters : each ID will be specify to each web part and we can use two PropertyFieldRichTextBox in 1 web part.
Stay tuned I will publish a new version soon.
Regards,
Olivier

Hi @kmartindale,
The fix is now available in the 1.3.7 version. Be careful: the PropertyFieldRichTextBox signature has changed, you can see the updated documentation here https://oliviercc.github.io/sp-client-custom-fields/propertyfieldrichtextbox.
I tested with many different web parts instances in the same page, with multiple rich textbox fields in each web part, and it's working. Please confirm me that it is also the case for you. Thanks again for this bug report,
Regards,
Olivier

Hey @OlivierCC  you rock. This seems to be working. Thanks so much for listening and taking the matter into your own hands! Closing out.


update apiversion for deployments to apps/v1 to support k8s 1.16

Fix for #371 .

@chrisevett thank you!

@chrisevett thx a lot. PR looks great.
One request, can you please move these into the parent folder? lets get the other yaml files replaced.

Is that going to be backwards compatible?

@chrisevett you're right, we'll lose compatibility here. This may be a secondary concern for the time being given the community's lack of bandwidth. What do you recommend as a version compatibility strategy? - a version per folder?


Generating a component appends new code a few characters too early in module

Versions
ng CLI version: 1.6.5
ng version: 5.2.1
node version: 8.9.4
npm version: 5.6.0
Windows 10 v.1703 build 15063.850

Repro steps

Step 1: Run Command:  ng g c path/component-name
Step 2: Navigate to app.module.ts
Step 3: Observe new import statement path errors and NgModule declarations array errors

Observed behavior
import { ExistingComponent } from './path/existing.componen; <-- cut off the last two characters
import { NewComponent } from './path/subfolder/new/new.component't'; <--and added them here

@NgModule({
 ...
     declarations: [
          ...
          ExistingCompone, <-- cut off the last two characters
          NewComponentnt, <-- and again added them here
     ]
...
})

Desired behavior
Proper output:
import { ExistingComponent } from './path/existing.component';
import { NewComponent } from './path/subfolder/new/new.component';

@NgModule({
 ...
     declarations: [
          ...
          ExistingComponent,
          NewComponent,
     ]
...
})

This seems to be related to #9712, but I am getting this issue only if I change the encoding of the file where the module is defined. Keeping the original encoding (which Notepad++ can't detect) it works just fine.


Remove govuk-mobile-remote-config

Removed govuk-mobile-remote-config

You need to manually delete it from TF state otherwise the repo will be deleted.