Add support for DICOM dermoscopy objects

This follows up on the discussion with @Punzo on slack.
The samples can be accessed in https://challenge2020.isic-archive.com/.
If it turns out the effort required is significant, we will need to reconsider.

FYI: @dclunie

@fedorov it turns out that OHIFv2 can load dermoscopy data without any issue.
I loaded locally the DICOM corrected Test Data (6,7 GB) from https://challenge2020.isic-archive.com/ .
There is an issue: in the metadata the PhotometricInterpretation is set to YBR_FULL_422, however when downloading the image the length is rows * columns * 3 (instead * 2) (each pixel has 2 values, which can be converted to RGB/RGBA values, see link).

if I force the PhotometricInterpretation to RGB, the image is loaded properly

I could check if the image array is and in case convert thhe array as an RGB in https://github.com/cornerstonejs/cornerstoneWADOImageLoader/blob/master/src/imageLoader/convertColorSpace.js, but I would prefer not putting hacks in place for deffefctive datasets.
3DSlicer loads the images without problems, but I think it ignores the PhotometricInterpretation, and it just uses the SamplePerPixel attribute (which for YBR_FULL_422 is 3 by the standard, even if before the conversion to RGB/RGBA the array values for each pixel are 2).
Anyway, for datasets with proper metadata will be loaded with no problem with OHIFv2 with latests cornerstoneWADOImageLoader (see PR: https://github.com/OHIF/Viewers/pull/2944)

@punzo, AFAIK, all these images are JPEG baseline compressed (Transfer Syntax 1.2.840.10<IP_ADDRESS>.50).
The DICOM Photometric Interpretation describes what is encoded in the compressed JPEG bitstream, not the RGB that might (or might not) come out of a JPEG decoder that converts the YCbCr chrominance downsampled compressed data into uncompressed RGB. See:
https://dicom.nema.org/medical/dicom/current/output/chtml/part05/sect_8.2.html#para_4bcb841e-c6bf-4e26-82a5-3fad3c942da0
and the notes after the table.
So, the logic in OHIF (or the image loader it is using) needs to account for whatever the JPEG decoder it is using actually does, and not be literally applied to the decompressed pixel data (i.e., it should know there are 3 pixels per sample in the decompressed data not 2 if chrominance downsampling has already been reversed), and that the color space has already been transformed by the decoder from YCbCr to RGB (both are usually the case for baseline JPEG decoders such as those that are libpeg-based).
Or to put it another way, each DECOMPRESSED pixel should have 3 values not 2 as you are assuming, regardless of the PhotometricInterpretation of the COMPRESSED pixel data being YBF_FULL_422. The only time that the PixelData length should be considered in this respect is when the encoding is "Native" (uncompressed), not "Encapsulated" (compressed), i.e., Note 2:
https://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_C.7.6.3.html#para_442b66ec-6d91-42ba-bfeb-738c94d2d6d3
does not apply to these JPEG compressed images.
Your decoder should not be guessing that Photometric Interpretation might be "wrong", rather it should know exactly what the compressed pixel data decoder has done, and behave accordingly.
Be aware that not all JPEG compressed pixel data is YCbCr - sometimes it really was compressed as RGB components (e.g., Aperio WSI).
Also, for JPEG compressed images, though YBR_FULL_422 is always used when a YCbCr color transformation has been applied before compression,  the actual extent of chrominance downsampling varies in practice, and is not always 2x2:
https://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_C.7.6.3.html#para_054ed567-c63e-4450-a128-0c8af7adc764
Further, some of the YBR family of Photometric Interpretations are used for non-JPEG compressed pixel data (e.g., some RLE or uncompressed ultrasound images, in which case a color space transformation +/- a chrominance channel upsampling may be required). That should not apply to these ISIC dermatology challenge images, though changes to your logic might affect handling of other images, which is why I mention it.
@chafey might want to comment on what Cornerstone does/should do in this respect for compressed images.

Found the issue. The wado parser in cornerstoneWADOImageLoader sends a XMLHttpRequest  as type=application/octet-stream; transfer-syntax=*
Then it uses the mime information from the server response to guess the transfer syntax of the donwloaded frame.
https://github.com/cornerstonejs/cornerstoneWADOImageLoader/blob/master/src/imageLoader/wadors/loadImage.js#L79
https://github.com/cornerstonejs/cornerstoneWADOImageLoader/blob/master/src/imageLoader/createImage.js#L238
However in the case of requests as type=application/octet-stream; transfer-syntax=* the right mime will not be known
(we had similar issues with SLIM: https://github.com/herrmannlab/dicom-microscopy-viewer/blob/master/src/pyramid.js#L426-L433). In fact in the case of this dermoscopy datasets, we would not get an a "image/jpeg" mime response (i.e. the frame is decompressed by the server and returned already as uncompressed color converted rgb image).
Therefore the cornerstoneWADOImageLoader will load internally the frame as if the Transfer Syntax is 1.2.840.10008.1.2 instead of 1.2.840.10<IP_ADDRESS>.50. This finallly generates issues, because color transformations are applied (not all decoders apply automatically the color transformations, so for a set of transfer syntaxes corresponding to such decoders, the color treansformations are appllied as in https://github.com/cornerstonejs/cornerstoneWADOImageLoader/blob/master/src/imageLoader/createImage.js#L238).
We could improve the cornerstoneWADOImageLoader either:

using the dicom-webclient instead to use a simple XMLHttpRequest (e.g., in Slim we use it to pass an array of transfer syntaxes https://github.com/herrmannlab/dicom-microscopy-viewer/blob/83739f1ba11840df7f4487cdf8015b7450ee419d/src/pyramid.js#L436-L491 and the first compatible one will used). In this way (having the "image/jpeg" mime of higher priority than application/octet-stream), we can get the original compressed jpeg file for the frame and we will decompress/apply color transformations client side.

or:
2) After fetching the arrayBuffer, we could guess the right frame mime from the array buffer rather then from the mime information from the server response. E.g. similar to the approach that we use in SLIM: https://github.com/herrmannlab/dicom-microscopy-viewer/blob/83739f1ba11840df7f4487cdf8015b7450ee419d/src/webWorker/decodeAndTransformTask.js#L99-L150
@sedghi @swederik, it would be nice to have your feedback as well. Thanks!

@fedorov moving this back in Progress

open PR with fix at https://github.com/cornerstonejs/cornerstoneWADOImageLoader/pull/477

Per @igoroctaviano, could it be related to https://github.com/OHIF/Viewers/issues/3354 ?

Per @igoroctaviano, could it be related to #3354 ?

Doesn't seem to be the same. It is not fixed with the update to cornerstone tools. From the data, it looks like the issue may be related to the way ohif viewer decode the RGB overlaid mask (I'll add more details about that directly in #3354). Also, in #3354 the thumbnails look ok, but not the images in the viewer, here instead, the images looked wrong in both, thumbnail and main viewer

A latest update to ohif (v3-stable) contains updates to cornerstone that solved the issue with the RGB display. Cornerstone's CPU-based thumbnails rendering didn't interpret correctly the RGB dermoscopy images, affecting how the thumbnails were displayed in OHIF. It was solved by changing the behaviour to GPU rendering by default (cornerstonejs/cornerstone3D-beta#586). This feature was transferred to OHIF in PR 3372 and now available in v3-stable.
This was tested with a subset from "DICOM corrected Test Data (6,7 GB)", available at (https://challenge2020.isic-archive.com/) OHIF/Viewer (v3-stable, Cornerstone Tools v0.66.2):
OHIF/Viewer (v3-stable, Cornerstone Tools v0.67.1):
Same data loaded through https://v3-demo.ohif.org

As shown in this test, this issue is solved with the latest updates to v3-stable and the data is now displayed correctly (https://v3-demo.ohif.org/).
I think this issue can be closed, @fedorov @igoroctaviano

Is it fixed in both v2 and v3, or only in v3?

Yes, I checked it works ok in both, v2 and v3

Closing, per discussion today.

@jlulloaa reopening - please confirm that this works with a Google DICOM Store!

please confirm that this works with a Google DICOM Store!

@igoroctaviano I guess @rodrigobasilio2022 will need to finish this up now?

@fedorov that's right, @rodrigobasilio2022 will continue this work.

Due to my lack of permission, @igoroctaviano will test

TODO for myself - add Rodrigo to

@igoroctaviano the DICOM store in the above is not in a project that is in IDC, or which I can access.
I created a new DICOM store, imported all ISIC 2020 test images, and added @rodrigobasilio2022 and @igoroctaviano: projects/idc-sandbox-000/locations/us-central1/datasets/isic/dicomStores/isic2020_test
Please confirm you can access.

@fedorov looks like the dicom store does not contain the same data used here: https://github.com/OHIF/Viewers/issues/2934#issuecomment-1559695180
@rodrigobasilio2022 can you confirm that?

This image is named ISIC_0520611.dcm

The store used in https://github.com/OHIF/Viewers/issues/2934#issuecomment-1559695180 was set up by Jose, and I do not know anything about what it contained.
The store I set up should have all of the DICOM dermoscopy files included in the test dataset from https://challenge2020.isic-archive.com/.

@igoroctaviano, you showed a bucket with few images, isn't it?

I am confused. Are you looking for a bucket or for a DICOM store? Can't you test against the dicom store I set up for you?

@fedorov I tried (unsuccessful): http://localhost:3000/projects/idc-sandbox-000/locations/us-central1/datasets/isic/dicomStores/isic2020_test/study/<IP_ADDRESS>.4.1.59<IP_ADDRESS>5.2571.16203982705<IP_ADDRESS>

I don't know if OAuth will work from localhost. Did that ever work?

I don't know if OAuth will work from localhost. Did that ever work?

Yes, It works fine for other studies. localhost:3000 is a valid callback for this OIDC

Permissions should be fine, I don't know what is going on then.

Here's the corresponding bucket: https://console.cloud.google.com/storage/browser/isic2020-test-sample.

I don't have permission to list any item in this bucket

This was confirmed to work against a GCP DICOM store.

Code cloned from https://github.com/OHIF/Viewers ( master branch) on July 20th still doesn't render RGB properly
( independently of modality and transfer syntax). Does it mean that the fix just wasn't merged to master branch?

Do you have an example image?

I do, cannot attach DICOM to comment though.

There is an unresolved known issue related to v3 incorrectly handling multichannel images: https://github.com/OHIF/Viewers/issues/3354.
I guess in the context of this issue, the question is: why is it that dermoscopy images render fine, and other RGB examples in #3354 do not?

My examples are also US  ( as in 3354). Also PALETTE _COLOR doesn't work.

IMO dermoscopy images ( at least seen in OHIF v3 worklist) have photometric interpretation YBR_ICT

I need to see the image to understand the problem. If you can upload to the dicom store ( I dont have the autority to let you), I can take a look in tje image.

Please look at 3354,
sample 3 displays the same presentation on similar US modality file

You are from IDC? I don't have full access so I need the url to the image so I can access it

No, I do not think @vicongit23 is from IDC. As suggested above, let's continue the discussion in #3354. Of note, there are links to the public samples from IDC in that issue.

No, I do not think @vicongit23 is from IDC. As suggested above, let's continue the discussion in #3354. Of note, there are links to the public samples from IDC in that issue.

Correct

This was broken https://d3w0jbfdn2j3pj.cloudfront.net/projects/idc-sandbox-000/locations/us-central1/datasets/isic/dicomStores/isic2020_test/study/<IP_ADDRESS>.4.1.59<IP_ADDRESS>5.6460.1620398329<IP_ADDRESS>

This was broken https://d3w0jbfdn2j3pj.cloudfront.net/projects/idc-sandbox-000/locations/us-central1/datasets/isic/dicomStores/isic2020_test/study/<IP_ADDRESS>.4.1.59<IP_ADDRESS>5.6460.1620398329<IP_ADDRESS>

I opened this one today and it looked good, at least the first series. There's a specific broken series?

I tried again with master and I got different results this time, it's still broken:

I tried again with master and I got different results this time, it's still broken:

@sedghi FYI, looks like a regression

@sedghi do we have automation in place for the codec library? running snapshot testing and checking if the images change? I think this is very useful to avoid regression.

sad :(
No we don't have really but we should really. Can you give me that data in .dcm format?

sad :( No we don't have really but we should really. Can you give me that data in .dcm format?

@fedorov is it ok to share this dataset?

Yes! But note that it will be prudent for you to keep track of where a given test dataset is coming from, and make sure you are in compliance with the license. This dermoscopy sample is from ISIC: https://challenge2020.isic-archive.com/

We do not seem to have any images in IDC that would be suitable to reproduce this issue.


Added full model description parsing, refactoring in progress...

@JuliaRegistrator register

Comments on pull requests will not trigger Registrator, as it is disabled. Please try commenting on a commit or issue.


chore: fix master build by merging alembic migration heads

See title, this happens when merging two database migration without rebasing first
ERROR
ERROR [flask_migrate] Error: Multiple head revisions are present for given argument 'head'; please specify a specific target revision, '<branchname>@head' to narrow to a specific head, or 'heads' for all heads

BTW @mistercrunch https://github.com/apache/superset/pull/28361 was just merged today and we could likely just change the revision keys in the associated migration file if we didn't want to add another migration.

@mistercrunch I wonder if there could be a GitHub action—something which I'm not overly well versed with—that could enforce that any PR which touches a migration file must be rebased against master before merging?

I wonder if there could be a GitHub action ...

I think there's a timing issue here where while things may be ok at the time the PR is update (and CI runs), it doesn't mean things are ok after when the main branch has changed. We'd need something like pre-merge hook but I don't think they exist.

One approach would be to try and "auto-fix" with a script that would run when pushing on master, but not great to push this straight on the main branch...
That or maybe merge queues? https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue


nvm exec not parsing command properly

Hey,
if I take the following (ember cli) command and run:
node node_modules/ember-cli/bin/ember test --filter="Team Test |"

the arguments passed to the test command are Team Test |, which is correct.
But if I do the following:
nvm exec node node_modules/ember-cli/bin/ember test --filter="Team Test |"

the arguments are split like so "Team", "Test", "|" which is not correct
Do you have any suggestion on how to escape the command to be parsed correctly?
I'm running nvm 0.29
Thanks

Argument passthrough is tricky - what happens if you do nvm run node_modules/ember-cli/bin/ember test --filter="Team Test |"?

Nope, that didn't work. It didn't pick up the filter arg at all.
I'm now doing nvm use 5; node node_modules...... instead which solves my particular predicament.

Glad you have a workaround. Your use case should definitely work, with both nvm run and nvm exec, so I'll get to this as soon as I can!

This was probably fixed with this commit: https://github.com/creationix/nvm/commit/b76d43e9f8f0fa5fc1ef6a4934da36d763e80dae

@patocallaghan please reopen if this is still an issue.


Proposed Recipes for Global Drifter Program

Source Dataset
I'm trying to port a code that converts the hourly GDP dataset (~20'000 individual netCDF files) into a ragged array to a pangeo-forge recipe.

AOML website link.
The file format is netCDF.
One file per drifter.
Accessible through ftp

ftp://ftp.aoml.noaa.gov/phod/pub/lumpkin/hourly/v2.00/netcdf/drifter_101143.nc

Transformation / Alignment / Merging
The files contain a few variables and metadata that should in fact be stored as variables. I have a draft recipe that cleans up the file, parses the date, converts metadata to variables. The files have two dimensions ['traj', 'obs'], where each file (one ['traj']) contains a different number of observations ['obs'] (this ranges from ~1000-20000+). More precisely, scalar variables['traj'] are: type of buoy, dimensions, launch date, drogue loss date, etc., and vector variables['obs'] are: lon, lat, ve, vn, time, err, etc.
To create the ragged array, scalar variables should be concatenated together, and the same goes for the various vector variables. My two issues are:

is it possible to concatenate on multiple dimensions?
is it possible to set a non-uniform chunk size? Chunks would be set to the number of observations per trajectory, which is needed to merge the nth chunk and allow for efficient calculations per trajectory.

Cheers,
Output Dataset
Single netCDF archive, or a Zarr folder.

Thanks for posting this Philippe! Pinging @selipot who I believe has NSF Earthcube funding to work on this topic!

is it possible to concatenate on multiple dimensions?

Not currently, but this is in the works (see https://github.com/pangeo-forge/pangeo-forge-recipes/issues/140)

is it possible to set a non-uniform chunk size?

On the target Zarr dataset? No, Zarr does not support this.

Given the ragged nature of the data you described, Xarray / Zarr may not be a good fit at all for the target format. Have you considered either

Awkward Array (cc @jpivarski) - what is the ideal on-disk format?
Using a tabular data model (e.g. Pandas dataframe) + Parquet storage?

Have you considered

Thank you for pinging me. Yes, @philippemiron's work on this is funded through my CloudDrift EarthCube project :)

Thanks for the suggestion, I will take a look at Awkward Array. I have not considered pandas, the data set is not that large ~17 GB, but I believe xarray would be easier to work with. I realized that ragged arrays are not really a good fit for Xarray but I have been testing using groupby() operation from flox by setting nonuniform Dask chunks per trajectory and hope to make operations per trajectory efficient.

For working with Lagrangian data (which we do a lot in our lab), I have always preferred Pandas, due to the ability to group on any of the different attributes (float id, lat / lon, time, etc). 17 GB is not so bad if you use dask.dataframe + Parquet.
I would recommend the following approach:

Define 3 representative workflow tasks
Try to do each of them with the 3 different libraries (Xarray, Awkward, Pandas)
Assess on a few different metrics including

performance (raw speed)
how confusing / complicated the code is

For working with Lagrangian data (which we do a lot in our lab), I have always preferred Pandas, due to the ability to group on any of the different attributes (float id, lat / lon, time, etc). 17 GB is not so bad if you use dask.dataframe + Parquet.
I would recommend the following approach:
* Define 3 representative workflow tasks

* Try to do each of them with the 3 different libraries (Xarray, Awkward, Pandas)

* Assess on a few different metrics including

  * performance (raw speed)
  * how confusing / complicated the code is

Thanks, that is a good plan. I will follow up here (or with you soon).

Thanks for the cc!

Awkward Array (cc @jpivarski) - what is the ideal on-disk format?

It's looking more and more like the ideal on-disk format is Parquet. pyarrow support for converting rich data types between Arrow and Parquet is getting better and better, and in pyarrow 6, Awkward 2, we carry over enough metadata to do lossless Awkward ↔ Arrow ↔ Parquet round-trips.
Also, we've been thinking/hoping to add a Zarr v3 extension (zarr-developers/zarr-specs#62), though this would require the Awkward library to interpret a Zarr dataset as ragged. An advantage of Parquet is that the raggedness concept is built into the file format—any other library reading it would interpret it correctly.
We're in the midst of developing a Dask collection for Awkward Arrays (https://github.com/ContinuumIO/dask-awkward/) and would be eager to help you with your use-case as a way to get feedback for our development. I'm on https://gitter.im/pangeo-data/Lobby and https://discourse.pangeo.io/ .

Hi Ryan,
We are still testing out file format and actually putting together a Notebook for the EarthCube annual meaning benchmarking (xarray, pandas, and awkward array). We are now wondering if it will be possible to host the dataset on the Pangeo Cloud. Currently, I have a set of preprocessing tools that:

replace the missing values by np.nan (not recognized because of errors in the NOAA files Alexander-Barth/NCDatasets.jl#166);
parse the date/time variable;
fix some data types;
retrieve some of the variables stored as attributes;
allocate and fill the np.array;
save to an xr.Dataset (I also have a variant to create a parquet archive).

With those tools, I can create the ragged array as follow:
from os.path import join, isfile
from preprocess import create_ragged_array
import urllib.request

# download the data
list_id = [101143, 101144, 101509, 101510, 101511]  # subset of the 17324 files

input_url = (
    'ftp://ftp.aoml.noaa.gov/phod/pub/lumpkin/hourly/v2.00/netcdf/'
)

folder =  'raw/'  # output folder
files = []
for i in list_id:
    file = f'drifter_{i}.nc'
    url = join(input_url, file)

    if not isfile(join(folder, file)):
        req = urllib.request.urlretrieve(url, join(folder, file))

    files.append(join(folder, file))

# ragged array
ds = create_ragged_array(files).to_xarray()
ds.to_netcdf('array.nc')

My guess is that this could possibly be converted to a pangeo-forge recipe but to be honest I'm not totally sure how to proceed (and if it's possible) with the current model.

@philippemiron - we would love to help support this use case. If the data can be stored as netCDF, then we can make it work here (with zarr).
However, I'm confused about what this ragged array looks like in xarray? My understanding is that xarray does not support ragged arrays. Can you share the xarray repr of ds as generated by your code?

See here. I'm also working with @jpivarski on a parquet representation that would be handled by awkward array.

Could you please just do print(ds) and copy-paste the results on this issue?

<xarray.Dataset>
Dimensions:                (traj: 5, obs: 10345)
Coordinates:
    ID                     (traj) int64 ...
    longitude              (obs) float32 ...
    latitude               (obs) float32 ...
    time                   (obs) datetime64[ns] ...
    ids                    (obs) int64 ...
Dimensions without coordinates: traj, obs
Data variables: (12/54)
    rowsize                (traj) int64 ...
    location_type          (traj) bool ...

Attributes: (12/15)
    title:             Global Drifter Program hourly drifting buoy collection
    history:           Version 2.00.  Metadata from dirall.dat and deplog.dat
    Conventions:       CF-1.6
    date_created:      2022-03-24T11:31:24.444885
    publisher_name:    GDP Drifter DAC
    publisher_email<EMAIL_ADDRESS>    ...                ...
    metadata_link:     https://www.aoml.noaa.gov/phod/dac/dirall.html
    contributor_name:  NOAA Global Drifter Program
    contributor_role:  Data Acquisition Center
    institution:       NOAA Atlantic Oceanographic and Meteorological Laboratory
    acknowledgement:   Elipot et al. (2022) to be submitted. Elipot et al. (2...
    summary:           Global Drifter Program hourly data

I am very interested in how this get solves.
I just wanted to raise the question of whether this is the best/good format to save the data in?
It looks like instead of using a ragged array, the dataset is stored as a single vector (obs). This is the same approach that Argopy (https://argopy.readthedocs.io/en/latest/data_manipulation.html) took with its point data set (@gmaze). When Argopy goes to their "profile" dataset, they basically fill the dataset with a lot of NaNs.
It is definitely the simplest and most intuitive format, and the differentiation into different drifters will have to be done by the analyst in code or by going to a "trajectory" dataset (like "profile"). Does the parquet representation handled by awkward array take care of that in a smarter way? and if so, then maybe Argopy can also benefit.

Hi @dhruvbalwada,
The variables are indeed each stored into a 1-d ragged array of size ['obs']. The separation can be easily done using the rowsize['traj'] variable that contains the number of ['obs'] per trajectory. The main advantage of using awkward array is that the indexing is already taken care of by the library, e.g. sst[400] returns the sea surface temperature along trajectory #401. I believe it is worth investigating as it can make the implementation of operations per trajectory simpler.

Oh wow, that sounds amazing! Can't wait to play with this.
Maybe Argopy would also like to move to this format in future releases.
I am sure the pros will have some better comments, but the way the xarray is formatted it seems like it should be little trouble getting it into Pangeo Forge.
One question might be, how does one go about chunking this dataset? If there are some better approaches than just chopping it arbitrarily.
Have you tried following the tutorial to see if you can create a recipe out of this, it seems like most of the elements are already there in the code you used to create the ragged array. Is there a specific step in making the recipe where you are getting stuck?

To perform operations per trajectory, you can chunk it like this:
# align chunks of the dask array with the trajectories
chunk_settings = {'obs': tuple(rowsize.tolist())}
ds = xr.open_dataset(path_gdp, chunks=chunk_settings)

and then use map_blocks() and apply_ufunc().

After examining the code and the ouput dataset, I believe it should be possible to produce this dataset using a standard Pangeo Forge XarrayZarrRecipe. You are basically just concatenating all of the netcdf files along the obs dimension, as we usually do with time dimensions. The only complication would be how to handle the traj dimension, which has a different (smaller) shape.
We need to play around with this a bit, but I don't see any fundamental blockers here.

I'm not 100% certain of what is the issue. I managed to create the XarrayZarrRecipe:
recipe = XarrayZarrRecipe(pattern,
                          target_chunks={'obs': tuple(obs_per_file)},
                          process_chunk=preprocess_gdp,
                          xarray_open_kwargs={'decode_times':False})

recipe

XarrayZarrRecipe(file_pattern=<FilePattern {'obs': 5}>, inputs_per_chunk=1, target_chunks={'obs': (417, 2005, 1970, 1307, 4646)}, target=None, input_cache=None, metadata_cache=None, cache_inputs=True, copy_input_to_local_file=False, consolidate_zarr=True, consolidate_dimension_coordinates=True, xarray_open_kwargs={'decode_times': False}, xarray_concat_kwargs={}, delete_input_encoding=True, process_input=None, process_chunk=<function preprocess_gdp at 0x13f0a1750>, lock_timeout=None, subset_inputs={}, open_input_with_fsspec_reference=False)

But then when trying to retrieve one chunk, I get the following error:
with recipe.open_chunk(all_chunks[0]) as ds:
    display(ds)

ValueError: Chunks do not add up to shape. Got chunks=((417, 2005, 1970, 1307, 4646),), shape=(417,)

I will try to spend some time playing around with it.

A single 'obs' vector is definitely the simplest and most intuitive format, and the separation into different drifters will have to be done by the analyst in code or by going to a "trajectory" dataset (like "profile"). Does the parquet representation handled by awkward array take care of that in a smarter way? and if so, then maybe Argopy can also benefit.

Parquet does that in a smarter way: the construction of a ragged array from rowsize and obs is built into the Parquet format. It knows about ragged arrays, but also nested records, missing data, heterogenous unions, etc.
>>> import awkward._v2 as ak
>>> dataset = ak.from_parquet("global-drifter/global-drifter-0000.parquet", "obs.l*itude")
>>> dataset.show(limit_rows=6)
{obs: [{longitude: 119, latitude: 21.2}, ..., {longitude: 115, ...}]},
{obs: [{longitude: -67.3, latitude: 41.9}, ..., {longitude: -52.6, ...}]},
{obs: [{longitude: -50, latitude: -32.5}, ..., {longitude: -51.8, ...}]},
...,
{obs: [{longitude: 177, latitude: 56.1}, ..., {longitude: 176, ...}]},
{obs: [{longitude: -109, latitude: -55.3}, ..., {longitude: -96.4, ...}]}]
>>> print(dataset.type)
907 * {obs: var * ?{longitude: float32, latitude: float32}}
>>> ak.num(dataset)
<Array [{obs: 1488}, {obs: 4136}, ..., {obs: 44962}] type='907 * {obs: int64}'>

But since this structure is relatively simple, ragged arrays with only one level of depth, distributing them with the existing NetCDF infrastructure and building that structure on demand is pretty reasonable:
>>> import h5py
>>> import numpy as np
>>> file = h5py.File("gdp_v2.00.nc")   # a different data sample
>>> content = ak.zip({"longitude": np.asarray(file["longitude"]), "latitude": np.asarray(file["latitude"])})
>>> dataset = ak.unflatten(content, np.asarray(file["rowsize"]))
>>> dataset.show(limit_rows=6)
[{longitude: -17.7, latitude: 14.7}, {...}, ..., {longitude: -16.9, ...}],
[{longitude: -17.3, latitude: 14.5}, {...}, ..., {longitude: -17.3, ...}],
[{longitude: 136, latitude: 10}, {...}, ..., {longitude: 125, latitude: 13.7}],
...,
[{longitude: -25.7, latitude: 65}, {...}, ..., {...}, {longitude: -30.2, ...}],
[{longitude: -24.9, latitude: 64.8}, {...}, ..., {longitude: -30.4, ...}]]
>>> print(dataset.type)
17324 * var * {longitude: float32, latitude: float32}
>>> ak.num(dataset)
<Array [417, 2005, 1970, 1307, ..., 4572, 17097, 3127] type='17324 * int64'>

(The last numbers are the lengths of the trajectories.) Although Parquet does this stuff in general, this particular dataset isn't complex enough to really need it.
Actually, I'd like to get involved in this for Argopy, for exactly the same reasons that it applies to @philippemiron's project. I did some early tests converting Argo's NetCDF files into Parquet, and the Parquet versions were 20× smaller, as well as faster to load. But even there, Parquet isn't strictly needed: the same rowsize + obs trick could be applied, and I think the gains would be similar. (By decoupling a physically meaningful variable, the sequence length, from file boundaries, we reduce overhead. But the rowsize + obs trick does the same thing.)
For this particular project, the main thing that I think Awkward + Dask provides over xarray + Dask is that when you
chunk_settings = {'obs': tuple(rowsize.tolist())}
ds = xr.open_dataset(path_gdp, chunks=chunk_settings)

the chunking size for parallelization becomes tied to the physically meaningful variable, "trajectory length." Datasets with a large number of small trajectories would have a per-task overhead that you can't get rid of. Consider that the rowsize is a potentially large array; converting it to a Python tuple might not be feasible, let alone creating that many Dask tasks.
With Awkward + Dask, a single task could be run on dataset[100:200], for example, which would have 100× less per-task overhead because it applies to 100 trajectories. How large that slice is would be determined by the number of available processors, memory constraints, and maybe networking if distributed.
Meanwhile, however, I need to learn the "+ Dask" part of this to actually develop some examples. @philippemiron and I have also considered some examples with Numba, and that's a third part to consider. @douglasdavis tried an example of Awkward + Dask + Numba this morning and it worked, though it evaluated the first partition locally to determine the output type and I think we can streamline that.

I will try to spend some time playing around with it.

I've modified the preprocess function to take a xarray.Dataset as a parameter so we can pass it directly to process_chunk parameter of XarrayZarrRecipe, see this gist. Here are the first few steps of the recipe to save you some time.
input_url_pattern = (
    "ftp://ftp.aoml.noaa.gov/phod/pub/lumpkin/hourly/"
    "v2.00/netcdf/drifter_{id}.nc"
)

list_id = [101143, 101144, 101509, 101510, 101511]
obs_per_file = [417, 2005, 1970, 1307, 4646]
input_urls = [
    input_url_pattern.format(id=id)
    for id in list_id
]
print(f"Found {len(input_urls)} files!")
input_urls

from preprocess import preprocess_gdp

recipe = XarrayZarrRecipe(pattern,
                          target_chunks={'obs': tuple(obs_per_file)},
                          process_chunk=preprocess_gdp,
                          xarray_open_kwargs={'decode_times':False})
recipe

Cheers,

I just got an update from @selipot about this

We have derived a good format for one of the datasets of the NOAA GDP (the hourly dataset, now with SST). Now, as we planned, could we get this dataset up on the Pangeo catalog?
The dataset is not big in itself (17GB as NetCDF or ~6.2 GB as parquet) but I think its presence in a cloud catalog will be beneficial as it lays next to other Eulerian big datasets. If you agree, how can we make this happen? The initial dataset is available right now from NOAA NCEI as a single NetCDF file (see https://github.com/Cloud-Drift/gdp-get-started), but we have a better version we would like to share with Pangeo.
I am copying Philippe Miron to this email since he led the efforts that made this happened. By the way, we just released officially the first version of our clouddrift library (https://github.com/Cloud-Drift/clouddrift) which contains the tools and recipe that were used to create this dataset.

Sounds like we are ready to move ahead with creating a recipe. The dataset is just a single netCDF file at
https://www.nodc.noaa.gov/archive/arc0199/0248584/1.1/data/0-data/gdp_v2.00.nc

@selipot - I browsed around the website, but I couldn't find any information about the GDP data license. Without a clear license, we may be in violation of copyright law if we host the data. Can you say any more about the license under which GDP is distributed?

Working on finding this out with with the GDP. I suspect it would be what the license is for NOAA data which is unknown to me. Again reaching out to NOAA.

From NOAA and NCEI: All environmental data managed/archived at NCEI is publicly available without restriction.  NCEI doesn't currently employ a data usage license.


Function Localization

The existing function is not completely Chinese

Hello @L-GROSS
We do not have a native speaker on our team, however, you can help us with translation:
https://github.com/dbeaver/dbeaver/wiki/Localization
Or you can specify here in this ticket functions with translation, and we will add it to the project.

Hello @L-GROSS
We do not have a native speaker on our team, however, you can help us with translation:
https://github.com/dbeaver/dbeaver/wiki/Localization
Or you can specify here in this ticket functions with translation, and we will add it to the project.

I suppose it's much better now (thanks to DBeaver community), so we can close this issue.


'pattern' does not support 'modifier' substatement along with 'error-message' substatement

The issue raised in https://github.com/freeconf/yang/issues/37  is still present
The yang module loading is failing if modifier and 'invert-match' keywords are present.
example:
            pattern '[Ii][Nn][Tt][Ee][Rr][Nn][Ee][Tt]$' { // mesh-tag with value="INTERNET" cannot be created
                modifier invert-match;
                error-message "Mesh-tag 'INTERNET' already exist and cannot be created.";
            }

Oh sorry.   I have noticed that  modifier invert-match is supported,  but the loading fails when  both 'modifier'  and 'error-message' is present together are substatements for pattern.
If only one of the substatement is present under 'patter', then loading is succesful

ok, i did not test that and i can see i got the grammar too limited
grammar:
https://github.com/freeconf/yang/blob/master/parser/parser.y#L834
test case
https://github.com/freeconf/yang/blob/master/parser/testdata/types/leaf.yang#L34
should be easy fix, I can get this in next day or so.
On Fri, Apr 14, 2023 at 8:37 AM David Abraham Mathews <
@.***> wrote:

Oh sorry. I have noticed that modifier invert-match is supported, but the
loading fails when both 'modifier' and 'error-message' is present together
are substatements for pattern.
If only one of the substatement is present under 'patter', then loading is
succesful
—
Reply to this email directly, view it on GitHub
https://github.com/freeconf/yang/issues/41#issuecomment-1508443361, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAACA7WV3V4XDSMVBJLHVHDXBFAKFANCNFSM6AAAAAAW54FDVE
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

should be all set now on master


Please add releases info / rss feed about updates

Hi @moudey,
just have a little question / request. So I see you don't have created any releases page on your project and you just pointing us to your website to download your app. Just wanna ask whether you have any rss feed on your webpage or if not could you add any on your website or on this project site? Just to check for updates from outside of the browser. I'am using a RSS tool and check different channels on internet & github too...like this...
https://github.com/portapps/brave-portable/releases.atom
...to see new updates without to visit every single channel manually. Would be nice if you could add any I can check for. A simple info about new version XY is released is ok. Thanks.

@Dean-Corso
Thank you for your feedback.
I will work on that in updating the site.
You can use this link to check the latest shell version: https://nilesoft.org/shell/version

@moudey,
Ok thanks but make a rss feed on your site or here which I can include in my RSS tool. I have many rss channels to check everyday (+20 on github) and I do not check every single URL manually for this you know. In a rss tool you can check all rss channels right away without to visit any website to get the actually status. Just look...

....pretty simple to get all update states of any github projects who also using the releases page itself. If you don't wanna share your source then you can still using the releases page to put your installer / portable packages into or just a info about the update XY and then I can use this page as rss feed what github supports already. Otherwise if you don't like this then just do it on your website later if possible so that would be better for users like me who using rss tools. Thank you.

Hi @Dean-Corso
Please use these feed channels
releases
https://nilesoft.org/feed/releases
news
https://nilesoft.org/feed/new
all
https://nilesoft.org/feed


IJulia heartbeat error when jupyter is launched from emacs

If I start my Jupyter process from emacs, then everything works fine with all my kernels except Julia. And the Julia kernel works if I comment out the line
 start_heartbeat(heartbeat[])

in IJulia/src/init.jl. So perhaps there is some issue with start_heartbeat which only shows up when Jupyter is launched in certain ways.
I'm on macOS Sierra, using Julia 0.6.0 and Emacs 25.1.1, and my MWE is putting
(start-process "Jupyter" "*jupyter-buffer*" "~/anaconda/bin/jupyter" "notebook")
in an emacs buffer and doing C-x C-e.

See also #489.  Apparently we don't need the heartbeat for the notebook, only for certain remote-process situations, so it should be safe to disable.   I still don't know why it is breaking on some machines, though.

Thanks, stevengj. I will just keep it disabled.


Re-wrote project structure and Object-Oriented http fuzzer

Great that you're taking under this big refactoring! Much obliged :)
I did a first overview review, and commented on some things.
There's quite a bit of changes in two commits.. If you choose not to split these changes into smaller commits, at least update the commit messages to state what all changes you have intended to make; this first version contained some "undocumented changes".
I didn't look carefully into your httpfuzzer refactoring yet to see whether all functionality is still intact that existed before. Taking the example *.feature file and translating that into the new API you introduced would help significantly both this one-time review, but especially anyone wanting to take mittn/fuzzer into use.
As for the 'scanner' and 'tls', folder restructuring I suggest you retain the example *.feature files for now, as they serve as documentation of the available features and guidance on how to use them.
The documentation files that were simply moved are no longer that relevant, as they talk about BDD, Behave, Gherking, environments.py etc.. So the documentation needs an overhaul. Perhaps even start with the documentation (http://tom.preston-werner.com/2010/08/23/readme-driven-development.html). That would enable more contributors to join in on the effort!
You necessarily don't need to make all these changes in one pull request, but if you have a plan in which order you'd plan to do some changes, it would help anyone wanting to review your commits on this bigger undertaking.

Alright, as discussed this was a PR to give hints about the direction we should probably go. Closing this now and opening separate smaller PRs.


🗺 Roadmap update

@jpechane, @Naros, @bibryam, feedback welcome on this community roadmap update for 1.3. Proably also a good idea to talk about it in next Monday's call.

@gunnarmorling LGTM. Just for 1.x I think you could include a replacement for embeddedengine

Good point; added.

Applied, thanks for feedback.


Restrict dynamic config file permissions

Proposed changes

Change permissions of dynamic config file 0644 -> 0640

Checklist
Before creating a PR, run through this checklist and mark each as complete.

[x] I have read the CONTRIBUTING document
[x] I have run make install-tools and have attached any dependency changes to this pull request
[x] If applicable, I have added tests that prove my fix is effective or that my feature works
[x] If applicable, I have checked that any relevant tests pass after adding my changes
[x] If applicable, I have updated any relevant documentation (README.md)
[x] If applicable, I have tested my cross-platform changes on Ubuntu 22, Redhat 8, SUSE 15 and FreeBSD 13

Codecov Report
Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (6d098af) 66.28% compared to head (2432399) 66.10%.
Report is 1 commits behind head on main.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #496      +/-   ##
==========================================
- Coverage   66.28%   66.10%   -0.18%
==========================================
  Files         118      118
  Lines       13446    13446
==========================================
- Hits         8913     8889      -24
- Misses       3938     3962      +24
  Partials      595      595

Files
Coverage Δ

src/core/config/config.go
69.02% <50.00%> (ø)

... and 2 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Generate Documentation/Blog using Wyam

Once we have the Cake PR pulled in, this should be simple.
/cc @daveaglick

Wyam all the things!

@daveaglick said...
Wyam all the things!

That's the plan :-)


Prestart director job without recursive chown on storage folder

What is this change about?
Change director pre-start job to just chown the root store folder and do not do it recursively.
Please provide contextual information.
On one of our landscapes we noticed that this usage of find + chown causes few minutes to complete. Before merging let discuss if this change could cause some implications.
What tests have you run against this PR?
We have tested newly deployed bosh and an updated bosh with the change and it worked fine.
How should this change be described in bosh release notes?
Not sure.
Does this PR introduce a breaking change?
It should not.
Tag your pair, your PM, and/or team!
@friegger @c0d1ngm0nk3y

It's hard for me to reason about what the effect of this might be. Have you tested this out? I assume that perhaps some child directories would be owned by root and with -rw-r----- permissions, that would mean that vcap could only read (I assume it's in the same group). I don't know if those child directories are important or if everyone of value is created after the pre-start. It does look like an attempt was made it speed it up: https://github.com/cloudfoundry/bosh/commit/2c3ad0339ebc2b0e8ea2ec82f672fcef9af93550 previously. Is your system significantly larger than what was tested?

Yes we've tested this. Directors with this change already run on 4 different infrastructures. Pre-start is down to 1s. Biggest impact is on Azure landscapes, where disk performance seems to be slower. Here we previously had ~4-6min for this step. It's real downtime.
Sample Usage:
/var/vcap/store# df . -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdd1       252G  188G   52G  79% /var/vcap/store

/var/vcap/store# du . -h -d 1
4.0K	./health_monitor
19G	./director
165G	./blobstore
16K	./lost+found
1.6G	./postgres-9.4
3.0G	./postgres-10
188G	.

We also tested the scenario with a new disk and then updating the Director with that disk.
The previous change did not really improve the situation here, probably find is rather slow the very first time it runs. Afterwards it seems to rely on cache.
The recursive chown is there since the project has moved to the cloudfoundry org. We assumed it might be there, because in ancient times things where not executed as vcap, but were also wondering whether there could be another reason.
We'll report back, when the change has reached all our systems.


Remove extras for backends, migrate to dependencies.yaml.

This PR changes to "bring your own backend" proposed by @vyasr by removing the extras.
This PR also migrates all dependencies in pyproject.toml to be managed by dependencies.yaml.
There is no longer anything specific to setuptools or scikit-build-core in the repo. You could use any other backend and it would work just fine. Also, extras don't work in a conda build environment. We prefer to force the user to specify the backend and avoid hiding this as "magic."

You'll probably need to update some of the tests to also install the necessary backends now.


Support for hidden cells executed only at test time

Is your feature request related to a problem? Please describe.
Many notebooks are extremely time intensive to run. In machine learning, notebooks may involve downloading very large datasets or training a model over hundreds of epochs. A single notebook may take hours to run, but we would still like to test these notebooks quickly in CI.
Describe the solution you'd like
We already have nbmake support for skipping cells, it would be interesting to be able to add hidden cells that are executed only by nbmake. With this, I could have a normal cell like:
download = True
num_epochs = 100

that is executed by all users, then a hidden cell containing:
download = False
num_epochs = 1

that is only executed by nbmake. This would keep testing time down while keeping the notebook as simple as possible.
Hidden cells definitely aren't the only way to implement this. For my use case, I really just need to change the values of a couple of variables. If nbmake offered a way to do this in a configuration file where it automatically replaced variables with these values that would also suffice. Instead of a configuration file, I would be fine with including this metadata in the notebook itself.
My goal here is that the notebook appears as simple as possible and doesn't include any visible code that is specific to nbmake testing. Any solution that offers this is equally valid to me, I definitely welcome alternative proposals.
Describe alternatives you've considered
Some alternatives I've come across:

Use sed to replace the variable value during CI (difficult on Windows, doesn't work locally)
Use environment variables like num_epochs = os.environ.get('NUM_EPOCHS', 100) (ugly, doesn't work locally)

Additional context
Some examples of these alternatives in the wild:

TorchGeo
GPyTorch

Hi Adam, this makes sense as an ask.
Do you have a preference as to how you would like to specify these hidden cells?
E.g. we could (a) inject some expressions into metadata tags, or (b) reference another notebook on the filesystem on include.

The former sounds simpler to me.

Ok, I'm having a think about this one for now.
Increasingly, I can see that it's necessary for nbmake users to customise execution in order to mock dependencies/config and assert things.
As a result, I'm considering exposing the nbclient instance so that you can execute python code during the test.
From your perspective, this may mean you tag a cell with my_module.set_epochs so you can invoke a python function to run this code.
Input welcomed!

We would write test code outside of the ipynb file that you are testing (e.g. in a python script or another notebook).
My assumption is that your users may want to run the ipynb file after seeing the docs, therefore it should not contain test code.

Yes, the ipynb file should be able to run normally without the test code.

@adamjstewart I have released v1.3.3a1 which allows you to mock variables after they are defined in a cell.
To do so, please add cell metadata to the cell which defines the variables. Nbmake will apply your mocks after the cell succeeds.
Example:
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbmake": {
     "mock": {
      "x": 2,
      "y": "fish",
      "z": {
       "x": 42
      }
     }
    }
   },
   "outputs": [],
   "source": [
    "x = 5\n",
    "y = 'y'"
   ]
  },

Please provide feedback! It will determine if we continue in this direction.

Will test this out when I get a chance!

Will test this out when I get a chance!

Awesome, I've added docs in the readme and made this feature available in 1.3.3


Build server: Logging cleanup

Meta issue - #1921
Currently we log all over the place with different types of loggers (MSBuild logging, CompilerServerLogger). We need to clean it up and make it better.

8b78b4f42158723b7f2d60f17bf4c8a44c27168a


Version 2.10.6?

Is there a tentative date for the release of the next version? I'm already using the new changes in the ConnectionPool.from_url that correctly parses the timeout values as floats, but it would be better to use a released version. :)

I am after the very same feature. 2.10.5 was released over a year ago and there are also several other fixes in master.
Ping @andymccurdy.

Me too, I assume this is why there is this fork on github and PyPI, with higher version number!

Is this repo no longer maintained @andymccurdy?

https://twitter.com/andymccurdy/status/735713770446585857

Hey @andymccurdy any thoughts on moving the project to https://github.com/jazzband/?

2.10.6 is on PyPI now!

Would it be possible to add this new release to this repository's releases page? It currently has 2.10.5 as the latest version.


Make it possible to upload icon files to GeoServer

Many styles that we want to use with GeoServer require separate image files for their icons.
Whilst the GeoServer REST API supports this it is limited to only one type of style file,

You can also use PUT with a ZIP file to upload a SLD 1.0 (application/vnd.ogc.sld+xml) file and any associated icon files.

Also the Java library that is currently being used to make the GeoServer REST API calls does not support this. A simple class that extends the existing one could be written to add this functionality though.
The alternative option is to just add a separate step that copies the images into the correct directory in the GeoServer container.

Side note: when client-side styling is supported (i.e. in Mapbox), the DTVF currently requires icons to be loaded as separate files. Icons are loaded from URLs, so if this can be achieved in a way that Geoserver can host these icons and make them accessible, then it can be used for client-side styling as well as server-side.


Add profile to website

Hey @zhmiao : Just thought of this since I am just now on this website-related stuff. Could you take a look at the steps and PR your profile to the group website? Thanks!

Closed by #45 .


⚠ Bofrost.fr has degraded performance

In ba71946, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 1911 ms

Resolved: Bofrost.fr performance has improved in 43d06c8 after 6 minutes.


geostyler-style/dist/style.d.ts:135:32 - error TS1110: Type expected.

geostyler-style/dist/style.d.ts:135:32 - error TS1110: Type expected.
I encountered this error when I was trying to install  geostyler-sld-parser using Angular 10.
No errors showing up in Visual Studio Code in the style.d.ts file, but compilation fails with the error:
ERROR in node_modules/geostyler-style/dist/style.d.ts:135:32 - error TS1110: Type expected.
135 export declare type FontSpec = ttf://${string}#0x${string};
I have typescript v4.0.5 installed.
Angular CLI: 10.2.0
Node: 14.17.0
OS: win32 x64
Angular: 10.2.3
Package                         Version
@angular-devkit/architect       0.1002.0
@angular-devkit/build-angular   0.1002.0
@angular-devkit/core            10.2.0
@angular-devkit/schematics      10.2.0
@angular/cli                    10.2.0
@schematics/angular             10.2.0
@schematics/update              0.1002.0
rxjs                            6.6.3
typescript                      4.0.5

Unfortunately, I cannot reproduce your error.
Could you try upgrading your typescript version? geostyler-sld-parser expects typescript=^4.2.4.
Also, it might help to look at https://github.com/geostyler/geostyler-demo/ and compare your webpack (or other) build configuration. When installing via npm, you do not have to compile geostyler-sld-parser.

@jansule , Thank you for the reply. I tried to upgrade the typescript version  but Angular 10 Compiler
requires  TypeScript >=3.9.2 and <4.1.0 .
So the problem still exists , is there a version of geostyler-sld-parser  expects typescript<4.1.0 ?

<EMAIL_ADDRESS>runs on<EMAIL_ADDRESS>you can try that.
Before that, better try setting "skipLibCheck": true in your tsconfig.json. See also https://www.typescriptlang.org/tsconfig#skipLibCheck.

Seems like this issue was solved. If not, feel free to reopen this issue.


Stolen Bitcoin and Lost Crypto Retrieval Hacker

There has been a lot of scamming online when it comes to Bitcoin investment in an online platform. I was a victim of an online  Bitcoin investment where I had a investment worth $79,000 btc in total. I couldn’t allow all my effort to waste when I discovered I was being scammed. For weeks I was depressed and was thinking of what I should not. Then I did some research on how i can recover my Bitcoin and then I came across Cyber Asset Recovery, a cybercrime investigator and a recovery expert firm. As a result of being able to explain my issues to Cyber Asset Recovery, everything I lost to these Bitcoin scam company was recovered in few days of working with them. Thanks to Cyber Asset Recovery. Their team of experts are pro at what they do. Send a message to them and get the assistance you require.
email.  Cyberassetrecovery@ protonmail. com
Telegram Username.   @CyberAssetRecovery
Website.       cyberassetrecovery. pro/

PRO ADVANCED RECOVERY is a private investigative, financial regulation, and funds recovery company. Cryptocurrency-related scenarios, phony investment
schemes, recovery scams, and ethical hacking are among our specialties. We look into the things that impact your score because we are experts in
credit restoration as well. All the software needed to do RECOVERIES from start to finish is available. Please be aware that the tools that are being
supplied are not eligible for CREDIT FIX. Be wary of fraudulent online reviews and testimonials. Together, the authors and Criminals form a syndicate.
Please use the email address below to contact our team as soon as possible to arrange a mail meeting with an ethical hacker.
<EMAIL_ADDRESS>TELEGRAM:@proadvancedrecovery
WHATSAPP:+1(657)5436038

Are you one of the unfortunate individuals who has fallen victim to a crypto theft? Do not despair, as there are solutions out there to help you recover your stolen funds. I will share my journey of recovering my stolen crypto of 160,000 USD with the help of Wizard Web Recovery. Cryptocurrency theft is an all too common occurrence in the digital age. With the rise of decentralized finance and the increasing popularity of cryptocurrencies, hackers and scammers are constantly on the lookout for opportunities to steal funds from unsuspecting individuals. I was one of those individuals who fell prey to crypto theft, losing a significant amount of money in the process. Feeling completely powerless and disoriented after having my crypto stolen, I scoured everywhere for a solution. Unfortunately, most recovery services seemed sketchy and undependable. Then, like a stroke of luck, I discovered Wizard Web Recovery. Their website exuded professionalism, their services were transparently laid out, and, crucially, they had a solid history of successfully recovering stolen crypto. I bravely decided to contact them for help. When I contacted Wizard Web Recovery, I knew I was in good hands. They were responsive, transparent, and empathetic to my situation. They explained their recovery process in detail and kept me updated every step of the way. Their experts worked tirelessly to track down the perpetrators and recover my stolen funds. After weeks of hard work and dedication, I received the news I had been hoping for - my crypto of 160,000 USD had been successfully recovered. Receiving my recovered crypto was an unparalleled experience. I was overwhelmed with relief, finally putting an end to the nightmare of being a theft victim. Wizard Web Recovery's unmatched expertise and dedication made it all possible. Their professionalism, efficiency, and effectiveness in reclaiming stolen crypto distinguish them from the competition. If you ever face the unfortunate situation of having your cryptocurrency stolen, don't lose hope! There are trustworthy recovery services like Wizard Web Recovery available to help you get your funds back. I successfully recovered my stolen 160,000 crypto with the expert assistance of Wizard Web Recovery. Remember, with the right help, reclaiming what is rightfully yours is entirely possible. Don't hesitate to seek assistance from professionals like Wizard Web Recovery – you won't regret it. Their contact info is written below.
WEBSITE - wizardwebrecovery.net
TELEGRAM - wizardwebrecoveryteam
EMAIL - wizardwebrecovery@ programmer (.)  net


Windows terminal: Copying data (with click or ctrl-c/ctrl-shift-c) pastes as a single string in other output.

Description of the problem or steps to reproduce
No SSH. When copying multiple lines (a block of code) in windows terminal and then pasting it in another file buffer the entire output is pasted as a single string.
When used the copy with click-selected feature :
a) The output is pasted but in a stair case manner
b) the initial source file (from where the data is copied) "scrolls up" and reveals clipboard/(other) data - still in a staircase manner.
Specifications

Commit hash:5044ccf6
OS: Windows 10 20H2 [Build: 19042.662]
Terminal: Windows Terminal [ Command Prompt selected ]
Staircase paste when using select to copy -

Normal ^-C or ^-Shift-C copy paste

Garbage data in source file when select copy is used and buffer is scrolled up.

Are you using strictly ctrl-c and ctrl-v to copy and paste?
To avoid the staircase paste you can also try enabling the paste option when you perform a paste.

Like I mentioned I tried the combinations for copy paste . The copy being with Ctrl-C and Ctrl-Shift-C and the select to copy (which windows terminal offers)
I will try your tip for the staircase paste but is there anything i can do about the single string conversion ?

Are you using strictly ctrl-c and ctrl-v to copy and paste?
To avoid the staircase paste you can also try enabling the paste option when you perform a paste.

The set paste didn't help . I get the same results

If you use ctrl-v to paste, do you still get the staircase paste? If so, try running > raw and then pressing ctrl-v. What does the output look like? Does it say EventKey: Ctr-v or something else?

If you use ctrl-v to paste, do you still get the staircase paste? If so, try running > raw and then pressing ctrl-v. What does the output look like? Does it say EventKey: Ctr-v or something else?

Nope , not staircase . I get it all as a block of continuous string when i use ctrl-v

I only get the staircase if I copy with the "select to copy" feature provided by the terminal itself (which in turn also shows the garbage data in the source file above , look at a screenshot example i posted at the forefront)
When i use ctrl-v with > raw enabled here is what i get:

a) I cannot scroll up so i cannot show the entire dump but basically raw throws out every single character copied along with a lot of whitespaces I see . I am not sure if I can pipe a dump to see it so i can scroll and say if the Eventkey: Ctrl-v is there.


BUG: artifcats left after deleted the observatorium CR

*-observatorium-thanos-receive-controller-tenants-generated, this configmap not deleted after the CR deleted

this seems like an issue for the github.com/observatorium/thanos-receive-controller-issue. Seems like we need to set owner references to that generated resources get deleted


defaultsTo on integer throwing error: Unexpected Error. Error is not defined in Database.

I'm using Postgres on AWS and seeing this error when attempting to use defaultsTo on an integer:
Unexpected Error. Error is not defined in Database.
field in the model is set to this:
amount: {
type: 'integer',
defaultsTo: 0
},
I've tried different values for kicks. Nothing works. I get this nebulous error.

Please reopen if this is still an issue.


Report sizes seem to increase over days after a restart

Noticed in data from a lot of probes - this is the average report size over the last 30 days:

The two drops are at points where all probes were restarted.
Pattern seems to suggest the probes are accumulating stuff in data structures which does not come back on restart, and therefore probably unnecessary.

Two things I realized by trying to download some older reports with the help of #3581 in Weave Cloud:

When downloading raw reports with timestamp e.g. 2019-02-12T17:52:00Z, I noticed that their JSON contained a lot of entries with the current timestamp i.e. 2019-03-14T15:03:55Z. That seems like a bug with the AWS reporter since the historic raw reports clearly differ from the ones that would have been downloaded at that time in the past.
Furthermore, downloading a few raw reports with a fixed timestamp 2019-02-12T17:52:00Z, I noticed that their size oscillates about 5% up and down within a few seconds, suggesting more than just timestamp entries are taken from the current state, rather than historic.

With these observations in mind, it seems quite hard and unreliable to analyze historic reports through Scope as we might a bug or a few with the historic lookup.
@bboreham Am I missing something? Are any of the above issues already known? If not, I'd just file new issues.
I imagine it would be hard to tackle this issue in the meantime as my understanding is that S3 only stores parts of reports and collecting and merging them manually at a given timestamp would be quite impossible.

For the second point, are you sure you're looking at reports from the same node?

Maybe you are looking at the merged report for the whole instance?
What we store in S3 is the exact data that came in from each probe; it's a part of the whole instance, but it's a better place to start because it separates any issues in the probe from issues in the merging process.

Another idea would be to use -probe.publish.stdout so you get the whole report in the log, and just leave the probe running for a while then see if you can spot it growing.
Maybe modify the code of StdoutPublisher to only print one in 100 or one in 1000 reports to save on disk space.

We got another drop in report sizes after the release of 1.11 3 days ago.
This time we also have per-user stats, and I was seeing a lot of dead containers in the large reports from one particular user, so that is probably the next place to look.

So that didn't fix it.
I added a log line to probe/docker/registry.go to print out events as they come in, then ran a command to create and destroy a container, several times:
for x in {1..100} ; do docker run --rm bash ; done

Here is the normal sequence of events:
<probe> INFO: 2019/06/04 18:04:34.004180 event create: 5050066d1a1417138e8367858abfa6c32b24ba9c26aba88c0130e0905e66fac0
<probe> INFO: 2019/06/04 18:04:34.010732 event attach: 5050066d1a1417138e8367858abfa6c32b24ba9c26aba88c0130e0905e66fac0
<probe> INFO: 2019/06/04 18:04:34.034367 event network:connect: 227fe2d486e77f56b7409f9ca15996aab446df3f687aabed902ced8768b44766
<probe> INFO: 2019/06/04 18:04:34.517102 event start: 5050066d1a1417138e8367858abfa6c32b24ba9c26aba88c0130e0905e66fac0
<probe> INFO: 2019/06/04 18:04:34.657756 event die: 5050066d1a1417138e8367858abfa6c32b24ba9c26aba88c0130e0905e66fac0
<probe> INFO: 2019/06/04 18:04:34.738521 event destroy: 5050066d1a1417138e8367858abfa6c32b24ba9c26aba88c0130e0905e66fac0

But sometimes we miss the destroy event:
<probe> INFO: 2019/06/04 18:04:34.907947 event create: 08b61f48790c7a6ae4ea1b084d112f1a1b224033bbb4960a7ae58cdf1fb581c5
<probe> INFO: 2019/06/04 18:04:34.926912 event attach: 08b61f48790c7a6ae4ea1b084d112f1a1b224033bbb4960a7ae58cdf1fb581c5
<probe> INFO: 2019/06/04 18:04:34.937667 event network:connect: 227fe2d486e77f56b7409f9ca15996aab446df3f687aabed902ced8768b44766
<probe> INFO: 2019/06/04 18:04:35.254530 event start: 08b61f48790c7a6ae4ea1b084d112f1a1b224033bbb4960a7ae58cdf1fb581c5
<probe> INFO: 2019/06/04 18:04:35.338868 event die: 08b61f48790c7a6ae4ea1b084d112f1a1b224033bbb4960a7ae58cdf1fb581c5

Sometimes we lose both die and destroy:
<probe> INFO: 2019/06/04 18:05:51.305000 event create: 5d8afb01b1345a193fcf944a7b4b9062849109ec4e3925f248195fe8b749930a
<probe> INFO: 2019/06/04 18:05:51.315773 event attach: 5d8afb01b1345a193fcf944a7b4b9062849109ec4e3925f248195fe8b749930a
<probe> INFO: 2019/06/04 18:05:51.332189 event network:connect: 227fe2d486e77f56b7409f9ca15996aab446df3f687aabed902ced8768b44766
<probe> INFO: 2019/06/04 18:05:51.944405 event start: 5d8afb01b1345a193fcf944a7b4b9062849109ec4e3925f248195fe8b749930a

and here's a normal one again:
<probe> INFO: 2019/06/04 18:06:22.699808 event create: acb3eccbf48fb486785229fed3ebe98f257392aa646c2e2e21430cefce72af12
<probe> INFO: 2019/06/04 18:06:22.716019 event attach: acb3eccbf48fb486785229fed3ebe98f257392aa646c2e2e21430cefce72af12
<probe> INFO: 2019/06/04 18:06:22.737031 event network:connect: 227fe2d486e77f56b7409f9ca15996aab446df3f687aabed902ced8768b44766
<probe> INFO: 2019/06/04 18:06:23.070987 event start: acb3eccbf48fb486785229fed3ebe98f257392aa646c2e2e21430cefce72af12
<probe> INFO: 2019/06/04 18:06:23.206659 event die: acb3eccbf48fb486785229fed3ebe98f257392aa646c2e2e21430cefce72af12
<probe> INFO: 2019/06/04 18:06:23.300760 event destroy: acb3eccbf48fb486785229fed3ebe98f257392aa646c2e2e21430cefce72af12


Timeout when invoke lambda

I'm trying to invoke a function from aws lambda
Rust version: 1.43
Docker image: clux/muslrust
    let client = LambdaClient::new(config.get_region());
    let request = InvocationRequest {
        function_name: config.get_function_name(),
        invocation_type: Some("Event".to_string()),
        payload: Some(serde_json::ser::to_string(&event).unwrap().into()),
        ..Default::default()
    };
    client.invoke(request).await
        .map(|_| Response {
            status: Status::Ok,
            message: None,
        })
        .map_err(|e| Error::message(format!("{}", e).as_str()))

with dependencies
rusoto_core = { version = "0.43", default_features = false, features=["rustls"] }
rusoto_lambda = { version = "0.43", default_features = false, features=["rustls"] }

how ever it stuck at
[2020-05-06T03:23:22Z DEBUG rustls::anchors] add_pem_file processed 175 valid and 0 invalid certs
[2020-05-06T03:23:22Z DEBUG rusoto_core::request] Full request:
 method: POST
 final_uri: https://lambda.eu-west-1.amazonaws.com/2015-03-31/functions/function-name/invocations
......
[2020-05-06T03:23:22Z DEBUG rusoto_core::request] user-agent:"rusoto/0.43.0 rust/1.43.0 linux"
[2020-05-06T03:23:22Z DEBUG hyper::client::connect::dns] resolving host="lambda.eu-west-1.amazonaws.com"
[2020-05-06T03:23:22Z DEBUG hyper::client::connect::http] connecting to <IP_ADDRESS>:443

Does anybody have this issue ? Please help
p/s: run the function in local can invoke the lambda without stuck (so it might be the ssl issue, I'm not sure how to debug it)

tested with the hyper lib itself on the lambda, sound like it's the network issue
START RequestId: 9ff31842-180b-4f05-adf1-042f6f73a0fb Version: $LATEST
[2020-05-06T07:41:00Z DEBUG hyper::client::connect::dns] resolving host="httpbin.org"
[2020-05-06T07:41:00Z DEBUG hyper::client::connect::http] connecting to <IP_ADDRESS>:80
END RequestId: 9ff31842-180b-4f05-adf1-042f6f73a0fb
REPORT RequestId: 9ff31842-180b-4f05-adf1-042f6f73a0fb	Duration: 5005.18 ms	Billed Duration: 5100 ms	Memory Size: 1024 MB	Max Memory Used: 27 MB	Init Duration: 22.92 ms
2020-05-06T07:41:05.764Z 9ff31842-180b-4f05-adf1-042f6f73a0fb Task timed out after 5.01 seconds


Update to v1.5.1

Updating the package version to v1.5.1.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

The osx build is failing due to what I assume is a systematic error, should I rerender this feedstock? @ping ocefpaf

The osx build is failing due to what I assume is a systematic error, should I rerender this feedstock?

Re-rendering will fix that but will also add Python 3.6 to the matrix. Not sure if we have all the dependencies ready for windspharm, but I am working on it.

OK, yeah you called it perfectly, the rerender fails because pyspharm has no 3.6 build.

I am working on pyspharm now. Hold on...

This look right @ocefpaf?

This look right @ocefpaf?

Perfect!
@ajdawson in conda-forge updates like this are OK to self-merge. We only expect review when there is a non-automatic (re-rendering) functional change to the feedstock. However, feel free to ping me anytime.

OK cheers, thanks for the help.


Angular extension cannot be installed in Fedora Linux

Following the instructions from the README.md, I clonned the repo, opened the "Install Dev Extension", I chose the project folder but nothing happened.
I found the following log in the ~/.local/share/zed/logs:
[INFO] compiling Rust extension ~/Documents/zed-angular [ERROR] failed to compile Rust extension Caused by:     0: failed to run rustc     1: No such file or directory (os error 2)
Any ideas?

Do you have rustc installed? rustc --version

Sorry for the late response!
I wasn't aware you needed Rust to compile the extension in order to be installed. I installed Rust and everything worked fine.
Thanks!


Add images: Android Basics with Compose

URL of codelab
https://developer.android.com/codelabs/basic-android-kotlin-compose-add-images?continue=https%3A%2F%2Fdeveloper.android.com%2Fcourses%2Fpathways%2Fandroid-basics-compose-unit-1-pathway-3%23codelab-https%3A%2F%2Fdeveloper.android.com%2Fcodelabs%2Fbasic-android-kotlin-compose-add-images#3
In which task and step of the codelab can this issue be found?
Add images to your android app, step 3, and an image composable
Describe the problem
After adding the background image I kept receiving the error "Render problem" and it wouldn't display the image on either the design preview or the emulator.
Steps to reproduce?

Follow the steps in the codelab

Versions
Android Studio version:
Android Studio Iguana | 2023.2.1
Build #AI-232.10227.8.2321.11479570, built on February 21, 2024
Runtime version: 17.0.9+0--11185874 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
Windows 10.0
GC: G1 Young Generation, G1 Old Generation
Memory: 2048M
Cores: 4
Registry:
ide.experimental.ui=true
API version of the emulator:
Additional information
Include screenshots if they would be useful in clarifying the problem.
By changing "image" to "img1" it works fine.

@seringl as seen in the first image , the compose is still rendering in the preview , please let the render happen , then send the screenshot again , if the the preview is out of date then kindly rerun the configuration or preview.

I have the exact same issue. As you can see, the build is complete but the preview is lost because of the rendering issue. It happens right after I add the Image composable, and call the GreetingImage function in preview.

Hi @seringl
Thank you for writing about your issue.
Have you tried complete closing Android Studio and then reopening to see if this resolves the issue?
I have  seen this error and it was an issue with Android Studio and not the actual code.
Regards.


Knowledgebase images gallery

Requires https://github.com/ABI-Software/sparc-api/pull/12 to work.

It has been merged upstream .


Prevent tests from failing due to Android 4.4 browser timing out

The coverage level has reduced because we don't actually have any ES6 mobile browsers to test in BrowserStack anymore. I'll have to force this PR through and I'll speak to Leon about upgrading our BrowserStack or changing to SauceLabs.

 Chrome Mobile 30.0.0 (Android 4.4.2) tests are always timing out

 Samsung Galaxy S5 (android 4.4) fails to connect on browserstack, slowing Travis builds


"itemscope" boolean attribute is not added to the DOM element

While trying to add an itemscope attribute to an element, I noticed that no matter what I do, It is never added to the DOM element in the rendered template...
I've tried :

...and It never worked. Am I missing something ?
Code exemple here

What's happening is that it passes the booleanAttribute test here, but earlier when it tests the node for that property it doesn't exist.
So when it goes to set it here it's setting an undefined property.
I don't know if the answer is to correctly set the propertyName on the node, or use setAttribute instead.

Thanx for the investigation ! I am not familiar with Ractive codebase but I just made a test and realized that setting the property itemScope to a DOM element dosen't set it's attribute automatically, could it be the problem ?
var test1 = document.createElement('div');
console.log('itemscope - before : ', test1.itemscope, test1.attributes);
test1.itemscope = true;
console.log('itemscope - after : ', test1.itemscope, test1.attributes);

var test2 = document.createElement('input');
console.log('required - before : ', test2.required, test2.attributes);
test2.required = true;
console.log('required - after : ', test2.required, test2.attributes);

Which prints :
itemscope - before :  undefined, {}
itemscope - after :  true, {}
required - before :  false, {}
required - after :  true, {0: required}

So the attribute is set using the more "regular" property required, and it is not with more "obscure" properties... There is probably other properties like this, I just don't know exactly how they are called :open_mouth:


Adds GCP backend to forward/orbit

Merges GCP backend functionalities in ft/GCP to forward/orbit

PR has been updated. Reviewers, please be cautious.

PR has been updated. Reviewers, please be cautious.

PR has been updated. Reviewers, please be cautious.

CONFLICT (content): Merge conflict in docker-entrypoint.sh
CONFLICT (content): Merge conflict in config.json

PR has been updated. Reviewers, please be cautious.

CONFLICT (content): Merge conflict in docker-entrypoint.sh
CONFLICT (content): Merge conflict in config.json

CONFLICT (content): Merge conflict in docker-entrypoint.sh
CONFLICT (content): Merge conflict in config.json

PR has been updated. Reviewers, please be cautious.

PR has been updated. Reviewers, please be cautious.


Add partial evaluation

http://okmij.org/ftp/tagless-final/course/PE-dB.html

I am trying out the PE-dB stuff now ... looks like I can simply copy over everything

https://gist.github.com/izgzhen/9e9555ecb08456c3007b7c935b96fad3
Not sure how to handle liftEnv...

We dont have real HOAS, and HOAS is bad for PE.

We dont have real HOAS

Yes, while being called hoas, it looks like de Brujin index for me.

Regarding liftEnv, can you explain a bit why it exists in the AST?

It is De Brujin inside, but it is very painful to write DBI by hand, so we present a HOAS interface, which get translated to DBI.

Hi, it is originally there since I though it is useful.
We dont really need it anywhere.

We dont really need it anywhere.

Then I can't wait to drop it so my PE derivation might work somehow :)

go ahead :)

I am a bit busy, preparing for lambdaconf, so cant work on code this week. Sorry about that, but will look at them once I am done.


xxxx.worker.js load wrong path, pls add  into html file it will work correctly

If you are using django, the quick fix is to add an entry in urls.py to redirect the *worker.js to the STATIC_URL
from django.http import HttpResponseRedirect
from django.conf import settings
import os

app_name = 'myapp'

urlpatterns = [
    ....
    url('worker.js$', view=lambda request: HttpResponseRedirect(os.path.join(settings.STATIC_URL, os.path.basename(request.path))))
    ....
]


TS-4295: Add Priority Queue in lib/ts/

TS-4295

This looks clean to me, but I haven't not checked for correctness. +1 from me.

There should be a test case for decreasing weights.

@maskit Please take another look.

Looks good except the comparator part.

@maskit Comparator is added.

Cool! +1

@masaori335 you can use TestBox.h for a simple regression test harness. There's other examples of doing this in the tree. The test output is better and you get a more structured set of test cases.

@jpeach I agree with using many asserts is ugly. But if I understand correctly, when we use TestBox.h, we need to write tests as Regression Tests. Right?
Because of PriorityQueue is a basic data structure, I'd like to keep tests as Unit Tests.

@jpeach I didn't know we can use TestBox.h in Unit Tests:p I'll fix tests to use it.
https://github.com/apache/trafficserver/blob/edf9ff13db4e63ee9b5003b603439bf2aa6ff19b/lib/ts/test_X509HostnameValidator.cc#L190


ModuleNotFoundError: No module named 'flask' on RHEL 8

After days of attempting to solve the problem myself I'm forced to realize that I don't have the skills to debug Python or Apache.
On a brand new RHEL8 system with @minimal installed, I am unable to get Puppetboard installed properly due to an error about the flask module not being found.
Using a simple:
class { 'puppetboard':
  manage_git        => true,
  manage_virtualenv => true,
}

Apache barks with:
[Tue Mar 29 08:57:24.729351 2022] [wsgi:error] [pid 6074] [remote <IP_ADDRESS>:58516] mod_wsgi (pid=6074): Failed to exec Python script file '/srv/puppetboard/puppetboard/wsgi.py'.
[Tue Mar 29 08:57:24.729453 2022] [wsgi:error] [pid 6074] [remote <IP_ADDRESS>:58516] mod_wsgi (pid=6074): Exception occurred processing WSGI script '/srv/puppetboard/puppetboard/wsgi.py'.
[Tue Mar 29 08:57:24.729764 2022] [wsgi:error] [pid 6074] [remote <IP_ADDRESS>:58516] Traceback (most recent call last):
[Tue Mar 29 08:57:24.729823 2022] [wsgi:error] [pid 6074] [remote <IP_ADDRESS>:58516]   File "/srv/puppetboard/puppetboard/wsgi.py", line 14, in <module>
[Tue Mar 29 08:57:24.729831 2022] [wsgi:error] [pid 6074] [remote <IP_ADDRESS>:58516]     from puppetboard.app import app as application
[Tue Mar 29 08:57:24.729840 2022] [wsgi:error] [pid 6074] [remote <IP_ADDRESS>:58516]   File "/srv/puppetboard/puppetboard/puppetboard/app.py", line 7, in <module>
[Tue Mar 29 08:57:24.729845 2022] [wsgi:error] [pid 6074] [remote <IP_ADDRESS>:58516]     from flask import render_template, Response
[Tue Mar 29 08:57:24.729863 2022] [wsgi:error] [pid 6074] [remote <IP_ADDRESS>:58516] ModuleNotFoundError: No module named 'flask'

Now, Puppetboard does seem to install properly but it's the Python module (I hope I'm describing it properly) that's giving me trouble.
Apache is running, but I get the following from curl/browser:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

A manual run of the WSGI yeilds:
[root@rhel8gold ~]# python3.8 /srv/puppetboard/puppetboard/wsgi.py
Traceback (most recent call last):
  File "/srv/puppetboard/puppetboard/wsgi.py", line 14, in <module>
    from puppetboard.app import app as application
  File "/srv/puppetboard/puppetboard/puppetboard/app.py", line 7, in <module>
    from flask import render_template, Response
ModuleNotFoundError: No module named 'flask'

It's been about four years since I've implemented a Puppet environment so I'm a bit rusty.  I've tried the recommendations here:
https://github.com/voxpupuli/puppetboard/issues/527
and here:
https://github.com/voxpupuli/puppetboard/issues/582
but to no avail.
I've had even less success with RHEL7.

Here's my apache vhost config:
<VirtualHost *:80>
  ServerName rhel8gold-puppetboard

  ## Vhost docroot
  DocumentRoot "/srv/puppetboard/puppetboard"

  ## Directories, there should at least be a declaration for /srv/puppetboard/puppetboard

  <Directory "/srv/puppetboard/puppetboard">
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Require all granted
  </Directory>

  ## Logging
  ErrorLog "/var/log/httpd/rhel8gold-puppetboard_error.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/rhel8gold-puppetboard_access.log" combined

  ## WSGI configuration
  WSGIDaemonProcess puppetboard group=puppetboard python-home=/srv/puppetboard/virtenv-puppetboard threads=5 user=puppetboard
  WSGIProcessGroup puppetboard
  WSGIScriptAlias / "/srv/puppetboard/puppetboard/wsgi.py"
</VirtualHost>

Hi @bschonec!
Please recreate the issue in voxpupuli/puppet-puppetboard as it’s a problem with the module, not the app.


not running

doesnt run and says process with an id of 13344 is not running. what am I doing wrong ? :)

This problem seems to be caused by moving a project between workstations, environments, or versions of Visual Studio
Follow steps in my article
https://medium.com/@alibenchaabene/build-a-chatgpt-client-an-image-generator-in-blazor-using-openai-api-315ce939e8ed


Feature/request scoped event emitters

PR Checklist
Please check if your PR fulfills the following requirements:

[x] The commit message follows our guidelines: https://github.com/nestjs/nest/blob/master/CONTRIBUTING.md
[x] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Other... Please describe:

What is the current behavior?
Request Scoped listeners are not available right now.
What is the new behavior?
We can inject the EVENT_PAYLOAD symbol and make our listeners Request Scoped:
class EventListener {
  constructor(@Inject(EVENT_PAYLOAD) private readonly eventPayload: any) { }
}

Does this PR introduce a breaking change?

[ ] Yes
[x] No

Other information
You can follow the commits to understand how it became from the first version to the refactored one, which I used to reduce the complexity of the EventSubscribersLoader class.

Fantastic PR @thiagomini, thank you 🙌

Fantastic PR @thiagomini, thank you 🙌

Thank you, Kamil! I added a few other commits here to resolve the review suggestions. I also refactored a bit the test code to use constants instead of manually set objects. In the final commit, I refactored a bit of the EventsProviderRequestScopedConsumer to look cleaner. Before this commit, I was adding more and more static variables to the class to handle different event payloads, so I decided to move this logic away to another object, the RequestScopedEventPayload class.


Problem accessing "external helpers"

Hi,
I would like to use serenity-js but I have to use some kind of external libraries of "helpers".
Following the instructions here https://serenity-js.org/handbook/design/interactions.html, I created an example based on the serenity-js-jamsmine-protracotr-template

the ./lib/ExternalLibrary.ts (the one I wan't to use)
the ./src/abilities/UseExternalLibrary.ts (as the ability to use the external lib)
the ./src/interactions/DoSomeMagic.ts (as the interaction to use the ability)
enhanced the ./spec/example.spec to instruct the actor to DoSomeMagic()
enabled the actors in ./src/actors.ts to use the ability UseExternalLibrary

When running the test, this test will raise a:
"ConfigurationError: Jasmine can't UseExternalLibrary yet. Did you give them the ability to do so?"
(I think this was done in step 5)
I also created a "native" example (folder spec-native) which runs without erros.
What am I doing wrong?
You can view and clone the code from here https://github.com/viper3400/serenity-js-playground/tree/UseExternalLibrary

I think I see what's going on.
In Actors we have:
export class Actors implements Cast {
    prepare(actor: Actor): Actor {
        return actor.whoCan(
            BrowseTheWeb.using(protractor.browser),
            UseExternalLibrary        // <- this should be an instance of the ability, so:
            UseExternalLibrary.of(externalLibrary)
        );
    }
}

The ability itself looks good; Remember you'll also need to return from doSomeMagic to avoid ignoring the Promise an async function returns in your ExternalLibrary.
You're on the right track, though!
Is the custom integration you're working on something you'd like to share with the wider community?

Hi Jan,
thanks for that, that did the trick. I didn't got this.
Actually, it's not very sophisticated what I'm trying and there is nothing worth to share. I tried to wrap up some existing page objects and navigation helpers (and their helpers ...) stuff from our outer business framework protractor tests (like navigation and login stuff) in order to reuse it in a new module to give the screenplay approach a chance.
I first thought I could wrap around it the way serenity-js/protractor - BrowseTheWeb.manage() ability does but finally ended up to wrap the functions I really need within interactions and using my own targets and task.
Nevertheless, thanks for you support, it helped me either way.
Regards, Jan


Myabe I find a spelling mistake when define struct DefalutServiceSDK

In the location [https://github.com/nervosnetwork/muta/blob/master/framework/src/binding/sdk/mod.rs#L23]
you maybe want to spell the word "DefaultServiceSDK" instead of "DefalutServiceSDK".

Thanks


Why project still 0.XX? Multiplayer and Deterministic Replay?

Hello @sschmid and others,
Why the version is still 0.XX? is there a full roadmap?
And now, i want to developing multiplayer game which implementing Deterministic Replay concept. Is Entitas good enough for it? and why?
Thanks

Hi,
I recently posted sth about the roadmap in the chat:
https://gitter.im/sschmid/Entitas-CSharp?at=58cdae0efe6a638b1add8d63
I'm currently working on a roslyn code generator and @mzaks is about to release Entitas-Lang which both attempt to solve the issues you might have with code generation. I also just updated the build pipeline and the Entitas code base to be deployed as dlls.
I want to have those things in before I consider going 1.0.0 since those changes might have an effect on the public api.
Having said that:

Entitas is ready for production

We used it for multiple different games in different genres over the last 3 years and there are a few games already out in the AppStore made with Entitas. Take a look at the Wiki - #madeWithEntitas
I used Entitas in a synchronous multiplayer game and in a deterministic game where I could replay the game just by providing inputs. It's really easy to achieve this with Entitas.

Is Entitas good enough for it?

In my opinion Entitas it currently one of the best options to build a game with a solid architecture. I'm currently working on a game with hundreds of systems and growing complexity, but it's still a pleasure to work with, and the complexity is manageable very well. We have thousands of unit tests and the game runs very smooth.

and why?

It's fast and optimized
I uses way less memory than Unity
It uses object pools and caching by default to ensure to have almost no allocations
Data Oriented approach, composition over inheritance
Systems that follow the Single Responsibility Principle
It's really easy to write Unity Tests with Entitas, which can be veeery beneficial if you want to create a deterministic game
It has proven to scale well
Me and the great community are happy to help if you have any questions
It's plain fun to work with :D

Hi @sschmid ,
Thanks very much for your explanation.
I will start to explore more.
(y)


Resolve spotless complaints

spotless is a real pain for drive-by-contributors (c.f. https://github.com/apache/logging-log4j2/issues/1216#issue-1538041705, there was an attempt in #1196, https://github.com/apache/logging-log4j2/issues/1319)
My understanding is that the advice to everyone is that they'll need to run spotless on any files they change as part of any effort to make a pr and to tolerate any merge conflicts. That should mean that running spotless on the entire repo is fair game as it's equivalent to anyone making a PR and losing a race against any other PR.
There's one notable problem with this PR, namely: https://github.com/apache/logging-log4j2/issues/1317, but someone can fix spotless for that and then fix the entire codebase. (That someone might be me if someone can explain what spotless is doing/why)
Checklist

[x] use main otherwise
[ ] ./mvnw verify succeeds (if it fails due to code formatting issues reported by Spotless, simply run spotless:apply and retry)
[ ] Changes contain an entry file in the src/changelog/.2.x.x directory
[x] Tests for the changes are provided -- spotless passes
[ ] Commits are signed (optional, but highly recommended) -- happy to do this if requested, but atm it's just overhead, and this took ~50 attempts.

@jsoref,
When I introduced Spotless last September, I was looking for a fast way to remove most useless differences (whitespace and import ordering) between 2.x and main, without one big commit. So I used the "ratchet" feature provided by Spotless. After 6 months of usage I must agree with you: we need to fix all files at once: "ratchet" does not work outside of a Git working tree and requires the CI to clone multiple commits instead of just the last one.
If we are going to do that, we need to:

discuss all aspects of code style on the<EMAIL_ADDRESS>mailing list: the current Spotless configuration has a minimal set of rules and I would want to merge a big commit every time we add a code style rule. A complete Java style guide is on our web site. This could be a starting point for a discussion, but I am sure that some rules are no longer supported.
Line length, modifier order, final keyword, chained invocations wrapping: these are only some examples of rules that are not currently enforced by Spotless, but need to be agreed on.
propose a Spotless configuration in one PR,
apply the configuration to all files in a second PR (so that committers can verify it more easily).

Hello. Would you consider https://github.com/marketplace/cleanthat/ to apply spotless automatically to incoming PRs ?
(I'm the author of Cleanthat, and a Spotless contributor).

Line length, modifier order, final keyword, chained invocations wrapping: these are only some examples of rules that are not currently enforced by Spotless, but need to be agreed on.

Line-length can be handlded in spotless with one of the formatting options (Eclipse, GoogleJavaFormat, etc)
Modifier Order is supported through Cleanthat. Unnecessary final keywords also.
I would need more details about expectations around chained invocations wrapping.

This has been solved in #1446.


Merge to publish

Renamed training to tutorials

OPS Build status updates of commit 607c218:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

.openpublishing.publish.config.json
:white_check_mark:Succeeded

.openpublishing.redirection.json
:white_check_mark:Succeeded

breadcrumb/toc.yml
:white_check_mark:Succeeded

docfx.json
:white_check_mark:Succeeded

tutorials/aspnet.yml
:white_check_mark:Succeeded

tutorials/flow.yml
:white_check_mark:Succeeded

tutorials/node.yml
:white_check_mark:Succeeded

tutorials/php.yml
:white_check_mark:Succeeded

tutorials/python.yml
:white_check_mark:Succeeded

tutorials/react.yml
:white_check_mark:Succeeded

tutorials/ruby.yml
:white_check_mark:Succeeded

tutorials/uwp.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


bug with spring gateway

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
[√ ] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?

Bug

Which version of SkyWalking, OS and JRE?
SW :8.2.0
OS: window 10
JRE: 1.8.0_261
spring boot 2.3.4
spring gateway 2.2.0.RELEASE

Which company or project?
(bsp) spring cloud project

What happened?
If possible, provide a way to reproduce the error. e.g. demo application, component version.

from the image ,the node "abnomal node" should not have appeared here。it was "interconnect" node.

Requirement or improvement

Please describe your requirements or improvement suggestions.

All data will be removed after the TTL reached.


Updating help for az webapp new

Fixes #90
This checklist is used to make sure that common guidelines for a pull request are followed.

If this PR is for a new extension or change to an existing extension, use the following to try out the changes in this PR:
docker run -it microsoft/azure-cli:latest
export EXT=<NAME>
pip install --upgrade --target ~/.azure/cliextensions/$EXT "git+https://github.com/panchagnula/azure-cli-extensions.git@sisirap-newHelp#subdirectory=src/$EXT&egg=$EXT"


Ansible multitenant playbook does not deploy plans

Fixed in https://github.com/EnMasseProject/enmasse/commit/2f72178becafa55cbc529dc9f14fd52d5027dc36


ft2-clone: 1.36 -> 1.37

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/8bitbubsy/ft2-clone/releases.
meta.description for ft2-clone is: "A highly accurate clone of the classic Fasttracker II software for MS-DOS"
meta.homepage for ft2-clone is: "https://16-bits.org/ft2.php"
meta.changelog for ft2-clone is: ""
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 1.37 with grep in /nix/store/95x8qpa6gng5gna10y6ixjx9kdrg9z13-ft2-clone-1.37

directory tree listing: https://gist.github.com/7f22dc3863923391942eaf0b1ed6eff3

du listing: https://gist.github.com/4ec2993231281744a3cba372fbca5a47

Rebuild report (if merged into master) (click to expand)

4 total rebuild path(s)

1 package rebuild(s)

1 x86_64-linux rebuild(s)
1 i686-linux rebuild(s)
1 x86_64-darwin rebuild(s)
1 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
ft2-clone

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/95x8qpa6gng5gna10y6ixjx9kdrg9z13-ft2-clone-1.37 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A ft2-clone https://github.com/r-ryantm/nixpkgs/archive/cce77e34a419574907205ff3d0c85d972446ef7a.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/95x8qpa6gng5gna10y6ixjx9kdrg9z13-ft2-clone-1.37
ls -la /nix/store/95x8qpa6gng5gna10y6ixjx9kdrg9z13-ft2-clone-1.37/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review 1

1 package built:

ft2-clone

Maintainer pings
cc @fgaz for testing.

Result of nixpkgs-review pr 102338 run on x86_64-darwin 1

1 package failed to build:

ft2-clone

Yes, darwin is broken since some time. I planned to fix it soon but with ofborg down I guess I should just mark darwin as broken

not a regression, marking broken can be done in another PR


⚠ Roblox Devforum has degraded performance

In 8d0314d, Roblox Devforum (https://devforum.roblox.com) experienced degraded performance:

HTTP code: 200
Response time: 4045 ms

Resolved: Roblox Devforum performance has improved in 94e2b2a.