Mass conversion of raw data to plot-level tiles

in brief discussion between @dlebauer and I last Friday, we talked pros & cons of just splitting all our imagery data into plot tiles, like for every sensor we would:

transfer raw data to Roger
level 1 extractors convert raw data into GeoTIFF
we have process that scans a day of GeoTIFFs from a sensor and creates plot-level tiles
level 2+ extractors operate on plot GeoTIFFs
later on we remove original raw files and only keep plot-level

idea being that in step 3 we use a more generalized version of the stereoTop full field mosaic extractor that operates on geoTIFFs. Thus we funnel all extractors -> GeoTIFF format (we can still make JPGs for quick previewing). To kick things off:
PROS

scientists likely to care about plot-level analysis in most cases; easier to work with and understand what you're seeing
easier to compare same plot location across time/sensors w/ aligned imagery
reduce stitching/overlap handling required on data scientist side if we don't perform this clip for them, or reduce computational overhead if we otherwise would dynamically query plots from netCDF files (for example)
we could consistently organize data by sensor x time x plot across entire project structure, not just in geostreams/bety GIS databases (e.g. in Clowder and in filesystem directories)

CONS

muddles data organization somewhat; datasets as organized on Roger disk and in Clowder would no longer apply. instead each plot could include 2+ distinct timestamps that were stitched together to form the plot image you're seeing so we'd have:

raw_data/
  stereoTop/
    2016-02-02/
      2016-02-02_12-32-32-123 Range 23 Pass 6/
        files

Now, maybe if we apply givens about the gantry we could assume we wont have more than 1 capture per plot, per sensor, per minute, so we drop the milliseconds from the timestamp or something. or we sort by plot:
raw_data/
  stereoTop/
    2016-02-02/
      Range 23 Pass 6/
        2016-02-02_12-32-32-123
          files

impact on storage plans. effectively plot tiles would become defacto "lowest level raw data" that is in the data release, so the actual raw imagery could be archived and deleted. this would be a complex process with all the issues JD and i face elsewhere in terms of making sure a full day of data is present before performing the tiling,
less flexibility if, theoretically, one wanted to query some arbitrary ground footprint later on that is not a specific plot boundary - if we dont implement mechanisms to dynamically query netCDF, for example. if we DON'T precompute the tiles, perhaps we can offer a dynamic way to chop things on the fly.

I think one way or another we need to provide imagery at the plot-level; either pre-clipped, or some means to quickly clip on demand (e.g. for plot-level extractors). so maybe a better question is, what's easier: a bunch of complexity to achieve the clipping above, or something like THREDDS to juggle big netCDFs on the fly? As I write this out, pre-clipping seems like a tall order, but maybe it saves us headaches later on... decided to just toss it on the table.
tagging @jdmaloney @jterstriep @yanliu-chn as FYI also.

Note that the script I linked does NOT create plot-level tiles. Basic logic for mosaic extractor is:

convert stereoTop .BIN files to GeoTIFF (this shouldn't be necessary since demosaic already does that - we should update to make it check for those first)
get list of all GeoTIFFs inside the field bounds for one day, and create a VRT (virtual TIF) that combines them all
feed VRT to gdal2tiles_parallel.py which prepares for Google Maps global tiles

The theoretical plot clipping extractor would chop the VRT into new TIFs based on each polygon in our plot definition.

@robkooper @jdmaloney @yanliu-chn @dlebauer @czender
Just spoke with Rob about moving this forward. I basically suggested:

we have extractors that convert raw sensor images into standard geocoded GeoTIFF format
we have a work script we run on Roger filesystem directly outside Clowder, probably a worker with some hefty memory.

loop across each sensor x day directory of GeoTIFFs
stitch the day's images together into one full field image (using VRT, or streaming or something)
clip full field image into images by plot
create aggregate metadata for the new plot image (e.g. timestamp down to the minute it was captured, even if multiple captures are blended into one plot)
organize data in Clowder by plot & date, once you get to single day collection you see a nice list of a single plot-level image for each sensor for that plot
we no longer store raw data/metadata in Clowder - the plot-level GeoTIFFs are "base level" data products for outside world

Main questions/concerns:

What does this mean for data usage projections i.e. our 10 PB estimate? we need to get a basic sample of the code working on one of the datasets to see how big the plot images are - the stereoTop existing mosaic code isn't a perfect comparison
How quickly can we be confident we have all the images from a day to be able to stitch & chop? 48 hours? 36?
do we need a buffer around plot boundaries/is there concern about the non-overlap between polys in the shapefile? this could be both a boon in terms of overall file size for a day, and a negative if there is anything of interest in between:

Mentioning @hmb1 since he is looking into efficient means of clipping netCDF to shapefiles...

@czender thanks. forgot to mention that I tagged you because I imagine the hyperspectral data might do this slightly differently than the others... e.g. maybe we bypass the GeoTIFF and modify your script to generate the .nc files per plot instead?

That's one possibility. If you have an ESRI shapefile or geoTIFF for a plot for which there's a corresponding hyperspectral image, please send it our way for experimentation...

@max-zilla To answer some questions

What does this mean for data usage projections i.e. our 10 PB estimate?

we are well below the 10PB estimate at this point, and if we do get closer, we can eliminate some of the intermediate data products and then if we still need to save space we can use lossy compression / downscaling

How quickly can we be confident we have all the images from a day to be able to stitch & chop? 48 hours? 36?

Isn't this information available as the difference between collection and transfer dates? (or maybe the more important time range is the beginning to end of transfer of one day's data)
(ignoring outliers caused by transfer downtime)

do we need a buffer around plot boundaries/is there concern about the non-overlap between polys in the shapefile? this could be both a boon in terms of overall file size for a day, and a negative if there is anything of interest in between:

You are correct that ~ 50% of the plot goes away if you just keep the plots. This is because of four rows, only the two in the middle are kept because the outside rows are 'border rows'. While the border rows do provide useful information prior to canopy closure, we have decided to start with these plot boundaries #187. Although the discussion of plot definitions is ongoing (https://github.com/terraref/reference-data/issues/60#issuecomment-275736954) I suggest we move forward with these plot definitions and exclude the alleys and border rows.
I am still confused about the use of a shapefile. I know we have discussed this before, but I was under the impression that our workflows would be based on OGC compliant data formats, and that we would import the plot boundaries into the PostGIS database where it can be queried in geojson or WKT format.

@czender @hmb1 are these files sufficiently aligned to the x,y grid that ncks -d x,xmin,xmax -d y,ymin,ymax in.nc out.nc would be sufficient and efficient?

@dlebauer and @max-zilla the 4-row plot arrangement and the issue of the 2 outer rows of each 4-row plot being 'border rows' only applies to Sorghum-season2.  Other seasons (Sorghum-season1, Wheat-season3, and the upcoming Sorghum-season4) are all planted as 2-row plots with both rows considered as data-collection rows.  The big differences between 2-row plot and 4-row plot arrangements is why Roman suggested making single rows the smallest units and building 'plots' by combining the single-rows of interest.

@dlebauer we have the plot boundaries in PostGIS and can build on that, I just used the shapefile as representative of the geometry we're working with so I could include that little picture.
@NewcombMaria that is helpful, thanks.

@dlebauer @hmb1 the rectangles above look rectangular to me so ncks will be efficient and fast on them. i am thinking of how best to use shapefiles at the "plant scale", e.g., how to mask the irregular regions of soil (or leaves) within a plot, as this seems like a useful capability for manipulating the hyperspectral data. we do not yet have a specific request like "can we get a netCDF file with valid values within this shapefile and _FillValues elsewhere", though it seems only a matter of time and we want to stay ahead of the curve...

@czender: @remotesensinglab is working on per-pixel classifications of sun/shade and plant/soil. That should be easier than irregular polygons (unlike plots, these irregular shapes will change in each image).
@NewcombMaria good point. We should re-visit Roman's proposal moving forward. For now we will keep these plot definitions for Sorghum Season 2.

OK. then the ncks -d x,xmin,xmax -d y,ymin,ymax in.nc out.nc method should suffice for HS imagery once @FlyingWithJerome commits the branch with the correct downsampled coordinates...

PS what I envision is that the classification algorithm could add new boolean variables like 'sun_leaf' 'shade_soil', etc to the reflectance index data product. @remotesensinglab does that make sense?

Hi David,
Yes, that is what I have in mind as well. Thanks,
Wasit

@czender that min/max arg looks perfect. FWIW you can grab the shapefile from my image above here:
https://github.com/terraref/extractors-metadata/tree/master/sensorposition/shp/sorghumexpfall2016v5
...we can (and soon should) use geostreams API to get boundaries of each plot like so:
GET https://terraref.ncsa.illinois.edu/clowder/api/geostreams/sensors?key=Pb3AUSqnUw&sensor_name=Range 10 Pass 2

RESPONSE
[
  {
    "id": 288,
    "name": "Range 10 Pass 2",
    "created": "2017-02-02T18:04:52Z",
    "type": "Feature",
    "properties": {
      "type": {
        "id": "MAC Field Scanner",
        "title": "MAC Field Scanner",
        "sensorType": 4
      },
      "name": "Range 10 Pass 2",
      "popupContent": "Range 10 Pass 2",
      "region": "Maricopa"
    },
    "geometry": {
      "type": "Point",
      "coordinates": [
        -111.974990376,
        33.0748710865,
        0
      ]
    },
    "min_start_time": "2016-06-30T18:11:40Z",
    "max_end_time": "2016-12-31T21:58:08Z",
    "parameters": [
      "centroid",
      "file_ids",
      "fov",
      "sources"
    ]
  }
]

Right now it just has the centroid point in geometry but I have a script to run soon that'll store the polygon of each plot in these. Range 10 Pass 2 is the plot in this case.

As a first target of this effort, I would propose:

[ ] select a sensor that has working converter to GeoTIFF (https://docs.google.com/spreadsheets/d/1LLiQSFHbEWoo_FkvG1nl-XoSq_lZ_o6VazfxmPfOH9Y/edit#gid=1888797382) - FlirIR would be a good choice as it also serves #207 directly
[ ] verify converter is embedding appropriate georeference information in GeoTIFF
[ ] target one recent day for that sensor and develop script to stitch them together & clip by plot (the main work in early going)
[ ] generate a set of clipped plots for that day and check 'em out (any QA concerns, file size, preserved georeference, etc)
[ ] verify other GeoTIFF converters also compatible with script
[ ] run on one day for other sensor GeoTIFFs

@yanliu-chn do you have suggestions for who could write the script in item 3? Also tagging @ZongyangLi @solmazhajmohammadi as they haven't been mentioned in this thread yet

@solmazhajmohammadi , @remotesensinglab @Paheding  @pless  @ZongyangLi - what is the standard way of handling data for overlapping images?  Is the data averaged?

@yanliu-chn suggests stitching together raw data as non-georeferenced image.

@ZongyangLi suggests using GeoTIFF and stereoTop for initial tests instead of FlirIR.

@max-zilla  Are we going to have stitching for hyperspectral data? It would be very helpful for developing plant level classification algorithms.

@Paheding The hyperspectral data are 0.75m x 20m non-overlapping swaths, so I do not think it would make as much sense to stitch these.

@yanliu-chn @ZongyangLi  the demosaic geotiffs are misaligned south of the plots:

also, the left/right of demosaic are offset up/down and not left/right. im thinking some coordinates/offsets/XY are mixed up in the extractor:

(from /ua-mac/Level_1/demosaic/2016-11-18/2016-11-18__11-15-00-680/)

@max-zilla I think the misaligned in the south might cause by lack of data in those region, because all the geo-information are base on the metadata in the json file.
For the left/right and up/down question, I do rotate a 90 degree to each image, that's base on my understanding of gantry position, if you could imagine that you are looking into the images from west to the east in the field, the left image is the 'up' image and the right image is the 'down' one. I may make some mistake, @dlebauer could you have a check if I was wrong.

Besides, since the 'Field of View' in the metadata is still not reasonable as mentioned in #126 , the script I uploaded is still using my assumption of the FOV.
To get a better full stitched map, it might be useful to only use one side images.

Not clear why https://github.com/terraref/computing-pipeline/issues/126 was closed but I reopened it.

@ZongyangLi I and @max-zilla checked the code to georef the geotiff files. I see that the code calculated the bbox latlon using a math formula that assumes the earth is a perfect sphere. On the other hand, the code assigns EPSG:4326 as the projection. This project's datum uses an ellipsoid to model earth.
The right way, IMHO, to do this is:

transform the center latlon to UTM12 (EPSG:32612) using either pyproj or gdal python
figure out the UTM12 coord of the two corner points of the bbox
transform the UTM12 coord of the bbox to latlon (EPSG:4326)

Besides, what is your current way to calculate the cell size? do you assume square cells, i.e., the cellsize along the lat direction is the same as that along the lon direction?

@yanliu-chn I think you are right, the earth model should be an ellipsoid.
Could you explain more about the idea of cell size as a square cell?

So suppose the camera outputs a (x pixels * y pixels) image. The first question is what is the resolution of each pixel? if it is cx mm *  cy mm, is cx == cy?
but anyway, this image would cover (x * cx)mm * (y * cy)mm spatial extent. cx and cy are two values you pass to geotiff's geotransform metadata.
the cellsize problem arises when we convert to latlon projection bc 1) 1 degree of lat, in meters, is not equal to 1 degree of lon ; 2) 1 degree of longitude, in meters, is different on different part of earth. therefore, if cx==cy in meters, they might be different in degrees. but at field view level, i think the difference is marginal. we only need to know how long in meters one degree of lat/lon is in Arizona. of course, the best way to know it is to let GDAL/Proj4 handle it. you can create a SRS object with UTM12 cell size, transform to EPSG:4326, then you know the cell size in degrees.

Then my assumption is a square cell.

sounds good. then you can calculate the cell size accordingly.

@ZongyangLi if you would make a branch with suggested changes to this: https://github.com/terraref/extractors-stereo-rgb/blob/master/demosaic/terra_demosaic.py
I can run it on a few test datasets and see how it looks in the map. I'm open to just stitching the left side or right side instead of both if you feel the results would be better, not sure the pros/cons for that approach.

@max-zilla I can try it. I was thinking @yanliu-chn is the expert in this subject, would it be much more straightforward to let @yanliu-chn finish this?

i might not get it done faster than you bc i'm constantly overloaded and has a very small percentage of my time in this project.

Clipping and storing data per plot would really helpful for someone who knows what is planted in each plot and interested just in several plots from plant research point. I also agree the pros of doing so mentioned above for data storage and management.
However, for me a mosaicked whole field image would be more useful for developing and testing methods. Plus the plots change from season to season. I run my algorithms on entire field and show variations of plant phenometrics over the field for pubs. For this same reason, we mosaic UAS image tiles as one.
Perhaps, there is possibility of developing options to download the image as one mosaic by defining area of interest?

I have been looking at the cropping 3D data to the plots

Pointclouds depend on the direction of the gantry is generated in different coordinate system. It seems to be a shift in the pointclouds depend on the direction. We need to find this shift, this issue depends on the issue#44

@remotesensinglab we've been discussing retaining the full mosaic, but we may not be able to store them all every day as a full-resolution image without doing some careful thinking about space usage and processing time for the overall pipeline, as those files will be significant in size. need to run a few tests first.

@max-zilla we make both first. There are many options for dealing with redundant  (stitched & subset) files but unless there is a compelling reason to test, first priority is to implement and identify use cases and patterns. Options for removing redundant files can wait a year so we can release both in Nov beta release.  One or more major commercial cloud providers are interested in hosting such files.

@ZongyangLi let me know how things are going re: the coordinate fix - if you're running into trouble, we can maybe get some other cyberGIS eyes on things aside from @yanliu-chn to help out

@max-zilla
I did some investigation on this last week. My understanding of Yan's suggestion is:

Use the lat/lon SE as (0, 0), and use gdal to convert it into UTM 12 in a flat coordinate system.
Use metadata in json file to get the meta image's center UTM 12 coordinate.
Use Field of View in json file to get the bbox of image, base on the assumption of square cells, also in UTM 12.
Transfer the bbox from UTM 12 back to lat/lon.

So here is my problem, we do not have a True 'Field of View' for stereo camera, how to figure out Step 3?

My comments were for fixing the problem in https://github.com/terraref/extractors-stereo-rgb/blob/master/demosaic/bin_to_geotiff.py in getting the bbox and setting the correct geotransform properties in geotiff.
if the coord info in the camera uses gantry coord, please refer to https://github.com/terraref/documentation/blob/master/spatial-reference-system-data.md to convert to USDA coord which if in meters is UTM12; if in degress, is in EPSG:4326.

@max-zilla
Added a function called 'get_bounding_box_fixed' here:
https://github.com/terraref/extractors-stereo-rgb/commit/39107d7e3e3423c789d229d45579d2c973e4efce
Please have a check and see if the stitching looks better.

@ZongyangLi we don't have the stitching code written yet - we need corrected geoTIFFs to write the code against, and I will probably not be the person writing that code.
https://terraref.ncsa.illinois.edu/clowder/datasets/58dbd6724f0c430e2bd7b246
I ran your updated code and the image shifted position and orientation:

Upper image is old version; lower image is new version. You can see by looking at the sliver of spectralon panel on the right side that the image was flipped vertically.
This image is from the upper right corner of field:

Pink plots in the image with range/pass:

The metadata reports:
gantry_system_variable_metadata:
position x [m]: 207
position y [m]: 4.501
position z [m]: 0.631

location in camera box x [m]: 0.877
location in camera box y [m]: 2.276
location in camera box z [m]: 0.578

So, I used the GANTRY -> MAC UTM that @yanliu-chn linked here:
https://github.com/terraref/documentation/blob/master/spatial-reference-system-data.md
...and compared that to using your get_bounding_box_fixed() formula:
YAN FORMULA
> formula_convert(-207, 4.501)
(409005.84550139995, 3659767.9647077997)
> utm.to_latlon(409005.84550139995, 3660182.0475077997, 12, 'S')
(33.076425455503376, -111.97489868337381)

Here's that location:

...but our actual image seems a bit off:

(see the lower, fixed image)
I'll try to dig into more examples because this particular image looks like it was in roughly the right spot to begin with, but we know that isn't the case everywhere. More after lunch.

@max-zilla
I didn't use Yan's formula to fix the bounding box, I used a script in https://github.com/terraref/extractors-metadata/blob/use-pyclowder-geostreams/sensorposition/terra_sensorposition.py#L58, and I realize now that might be wrong.
I will try to use those formula and update the script after lunch.

@ZongyangLi thanks - yeah the sensorposition code will also need to be fixed I believe, when we have a working formula for this extractor I'll use that in the other extractor(s) as well.

@max-zilla @ZongyangLi I need to point out that MAC itself has a shift of 2.5m due to its initial error on reference location (and inherited by all the data using MAC). So to get lat lon, we will need gantry->USDA.
you can either do gantry->MAC in UTM12; UTM12->latlong; MAC -> USDA documented in https://github.com/terraref/documentation/blob/master/spatial-reference-system-data.md, or check Solmaz's comment in https://github.com/terraref/documentation/issues/7#issuecomment-217183545 , which I don't know how the transformation matrix was created and which GPS system it referred to (MAC or USDA).

@yanliu-chn
The idea of 2.5m shifting means we need to add Latitude: Uy = My - 0.000015258894 Longitude: Ux = Mx + 0.000020308287 to get a lat/lon, is that right?

@ZongyangLi  Yes.

@ZongyangLi let me explain. gantry->mac will get you the coord in meters; then you convert the coord to latlon (UTM12->EPSG:4326); then use the mac->usda formula to get the correct lat lon.
you can directly adjust the 2.5m in UTM12 along x and y direction if you know how many meters 0.000015258894 degrees of lat and 0.000020308287 degrees of lon are.

@max-zilla I have just uploaded a bounding box method with formula in the repo.
https://github.com/terraref/extractors-stereo-rgb/tree/master/demosaic

@ZongyangLi this looks close to correct based on the formula I applied above (original is the lower image, fixed is upper):

Again, for:
gantry_system_variable_metadata:
position x [m]: 207
position y [m]: 4.501
position z [m]: 0.631

location in camera box x [m]: 0.877
location in camera box y [m]: 2.276
location in camera box z [m]: 0.578

This seems to agree with our basic plot model:
# GANTRY GEOM (GANTRY CRS) ############
#			      N(x)
#			      ^
#			      |
#			      |
#			      |
#      W(y)<------SE
#
# NW: (207.3,	22.135,	5.5)
# SE: (3.8,	0.0,	0.0)
#######################################

North corner X is 207.3 and our gantry position is 207; since in this CRS X is along the north/south axis and increases as we move north, we expect to be just below the northern boundary.
Similarly, East Y is 0 (aka the rightmost boundary) and West Y is 22.135; ours is 4.5 so we expect to be roughly 1/5 of the way across the field.
So this looks reasonable, although I notice the left/right images look identical - does that surprise you?
I think next step is to trigger this new version of extractor on a whole day of datasets and bring them all into a map to see how it looks and start to work on stitching. I'm also going to look at the metadata work @craig-willis has done to see if we can pull the parameters in this script from a standard location instead of having them hardcoded.

@ZongyangLi @yanliu-chn I am generating a full day of demosaic TIFFs here:
/sites/ua-mac/Level_1/demosaic/2017-02-07/

there are ~4,200 of these to generate, might be completed by the time of the TERRA meeting but not sure.

@pless and @ZongyangLi can focus on the stitching component of this. getting one big GeoTIFF for Feb 7.
@jterstriep has volunteered to handle the clipping to individual plot boundaries from that big GeoTIFF.

FOV issue is https://github.com/terraref/computing-pipeline/issues/126

@ZongyangLi @yanliu-chn @jterstriep  @smarshall-bmr looks better w/ new calculations, but we can see where we need the FOV correction:

Four images of same basic part of field with different images turned on/off:
Image A:

Image A+B (see split in left tire tread)

Image A+B+C (see another split in center dirt)

Image A+C (gap between images despite areas that should overlap)

In phone discussion with Zongyang, thinking was that FOV correction would correct for this.

What is a robust, simple approach that we can use at this point?
Should we differentiate between images before and after canopy closure?

when plants are small and lots of ground is visible, adjust plot boundary for scanner height so that ground footprint is consistent
when canopy is fully closed, use plot boundary  at canopy height?
Does it help if we stitch using both the image features and geospatial (field of view) information?

@solmazhajmohammadi could you please update this issue with a summary of your findings to date?

Feature based registration without considering the z height, this is working for registration in small area. When the image area gets bigger, number of the features increases and it is hard to register the images after that. (due to false match)
I did a little bit research on this: ASIFT algorithm seams to be a solution for this problem. This is a great opensource work, which forces the algorithm to only consider affine transformation. So we dont get false arbitrary matches.
http://www.ipol.im/pub/art/2011/my-asift/article_lr.pdf

@solmazhajmohammadi  SIFT based algorithms are very robust to image translation, rotation, and scaling, however the SIFT might be still patented. Alternatively, we can use SURF method ( article:
http://www.vision.ee.ethz.ch/~surf/eccv06.pdf). As for mismatched features, RANSAC http://www.cs.columbia.edu/~belhumeur/courses/compPhoto/ransac.pdf method can be used before feature extraction to eliminate outlier features, so that the false matches can be resolved.

@Paheding take a look at the figure 26 on the paper I posted. I have not tried ASIFT in the big plot size images yet, I think this approach warranty the affine transformation between the features.  When the number of the registered images get higher RANSAC didnot help on the false matches.

@solmazhajmohammadi Correct, there are some drawbacks of RANSAC.  Here is an optimized RANSAC algorithm https://pdfs.semanticscholar.org/6dc8/6d312ca1c5b18e53ecb98fa6b5fc1053e023.pdf, which tends to be better than the classic RANSAC.  Source code: http://www.cb.uu.se/~aht/code.html. Combination of the optimized RANSAC and ASIFT would be more robust.
Besides, another quick solution could be make a threshold to restrict the number of registered images.

Is this ready to be closed? #306 covers some remaining issues.

Yes, I believe so - current plan is for @jterstriep to develop service to query plots on-demand instead of mass conversion ahead of time. There are other issues where this is being discussed.

@jterstriep could you link to or create the issue that covers the on demand plot subsetting?


k8s unit tests on ppc64le panics with index out of range

What version of Go are you using (go version)?

$ go version
go version devel go1.18-6f74ed06c5 Thu Oct 7 03:41:40 2021 +0000 linux/ppc64le

Does this issue reproduce with the latest release?
No
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="ppc64le"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="ppc64le"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/usr/local/go/bin/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/go/bin:/root/bin"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_ppc64le"
GOVCS=""
GOVERSION="devel go1.18-6f74ed06c5 Thu Oct 7 03:41:40 2021 +0000"
GCCGO="gccgo"
GOPPC64="power8"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/kubernetes/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2441321377=/tmp/go-build -gno-record-gcc-switches"

What did you do?
Install upstream dev version of golang:
    Have a stable version of go already installed.
    git clone https://github.com/golang/go.git
    cd go/src
    ./make.bash
    cd ../..
    rm -rf /usr/local/go
    cp -r ./go /usr/local
    export PATH=/usr/local/go/bin:$PATH
    export PATH=$GOPATH/bin:$PATH
Clone k8s/k8s repo and run below steps: (git clone https://github.com/kubernetes/kubernetes)
    export KUBE_TIMEOUT='--timeout=600s' export KUBE_COVER="n" export KUBE_KEEP_VERBOSE_TEST_OUTPUT=y export LOG_LEVEL=4
    cd kubernetes
    pushd ./hack/tools
    GO111MODULE=on go install gotest.tools/gotestsum
    popd
    make test KUBE_RACE=-race WHAT=./cmd/genkubedocs

What did you expect to see?
Test passing successfully
What did you see instead?
+++ [1007 06:05:26] Running tests without code coverage and with -race
fatal error: index out of range

runtime stack:
runtime.throw({0x1748a03d, 0x1a85d840})
	/usr/local/go/src/runtime/panic.go:965 +0x58
runtime.panicCheck1(0x19c8bdc0, {0x1748a03d, 0x12})
	/usr/local/go/src/runtime/panic.go:35 +0xb0
runtime.goPanicIndex(0x26fd9, 0x26fd9)
	/usr/local/go/src/runtime/panic.go:88 +0x4c
runtime.findfunc(0x163bb150)
	/usr/local/go/src/runtime/symtab.go:831 +0x1d8
runtime.gentraceback(0x7ffff7490198, 0xffffffffffffffff, 0x17617ff6, 0xc0000021a0, 0x4, 0xc000f277a0, 0x20, 0x0, 0x17617ff6, 0x0)
	/usr/local/go/src/runtime/traceback.go:221 +0x564
runtime.callers.func1()
	/usr/local/go/src/runtime/traceback.go:857 +0x64
runtime.systemstack()
	/usr/local/go/src/runtime/asm_ppc64x.s:256 +0x68

goroutine 1 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_ppc64x.s:213 +0x10 fp=0xc000f276c0 sp=0xc000f276a0 pc=0x11deb360
runtime.callers(0xc00019a000, {0xc000f277a0, 0x11db2124, 0x11d8c318})
	/usr/local/go/src/runtime/traceback.go:856 +0xb8 fp=0xc000f27738 sp=0xc000f276c0 pc=0x11de1fe8
runtime.mProf_Malloc(0xc000d15000, 0x1000)
	/usr/local/go/src/runtime/mprof.go:343 +0x74 fp=0xc000f278b8 sp=0xc000f27738 pc=0x11dad804
runtime.profilealloc(0x163e13f0, 0x0, 0x0)
	/usr/local/go/src/runtime/malloc.go:1255 +0xac fp=0xc000f27900 sp=0xc000f278b8 pc=0x11d8c91c
runtime.mallocgc(0x1000, 0x167b7720, 0x1)
	/usr/local/go/src/runtime/malloc.go:1134 +0x7ac fp=0xc000f27990 sp=0xc000f27900 pc=0x11d8c40c
runtime.makeslice(0x1a916a60, 0x11defc90, 0x11f13d34)
	/usr/local/go/src/runtime/slice.go:100 +0xb4 fp=0xc000f279c8 sp=0xc000f27990 pc=0x11dd1f64
bufio.NewWriterSize(...)
	/usr/local/go/src/bufio/bufio.go:590
bufio.NewWriter(...)
	/usr/local/go/src/bufio/bufio.go:597
testing/internal/testdeps.TestDeps.StartTestLog({}, {0x183dc6e0, 0xc00019e6e8})
	/usr/local/go/src/testing/internal/testdeps/deps.go:110 +0xbc fp=0xc000f27a38 sp=0xc000f279c8 pc=0x11ff677c
testing/internal/testdeps.(*TestDeps).StartTestLog(0x1b371cf0, {0x183dc6e0, 0xc00019e6e8})
	<autogenerated>:1 +0x88 fp=0xc000f27a68 sp=0xc000f27a38 pc=0x11ff7b68
testing.(*M).before(0xc00080f4a0)
	/usr/local/go/src/testing/testing.go:1850 +0x82c fp=0xc000f27b68 sp=0xc000f27a68 pc=0x11f13e5c
fatal error: index out of range
panic during panic

runtime stack:
runtime.throw({0x1748a03d, 0x1a85d840})
	/usr/local/go/src/runtime/panic.go:965 +0x58
runtime.panicCheck1(0x183c44a0, {0x1748a03d, 0x12})
	/usr/local/go/src/runtime/panic.go:35 +0xb0
runtime.goPanicIndex(0x26fd9, 0x26fd9)
	/usr/local/go/src/runtime/panic.go:88 +0x4c
runtime.findfunc(0x163bb150)
	/usr/local/go/src/runtime/symtab.go:831 +0x1d8
runtime.gentraceback(0x8, 0x0, 0x11de1758, 0xc0000021a0, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0)
	/usr/local/go/src/runtime/traceback.go:221 +0x564
runtime.traceback1(0xc0000021a0, 0x0, 0x11db758c, 0xc0000021a0, 0x10)
	/usr/local/go/src/runtime/traceback.go:789 +0x1a4
runtime.traceback(...)
	/usr/local/go/src/runtime/traceback.go:743
runtime.tracebackothers(0x1a924aa0)
	/usr/local/go/src/runtime/traceback.go:981 +0xb4
runtime.dopanic_m(0x1a924aa0, 0x11db6e14, 0x11db6e18)
	/usr/local/go/src/runtime/panic.go:1165 +0x2b0
runtime.fatalthrow.func1()
	/usr/local/go/src/runtime/panic.go:1020 +0x4c
runtime.systemstack()
	/usr/local/go/src/runtime/asm_ppc64x.s:282 +0xa8
FAIL	k8s.io/kubernetes/cmd/genkubedocs	0.407s
FAIL
make: *** [Makefile:184: test] Error 1

Job History can be found here: https://prow.ppc64le-cloud.org/job-history/s3/prow-logs/logs/postsubmit-master-golang-kubernetes-unit-test-ppc64le
I suspect a recent commit https://github.com/golang/go/commit/8238f82bf1ddadd906585e3f11600968d95ce98c is causing the issue.
Verified that same test passed successfully on x86 machine

@archanaravindar verified for me earlier that it fails on e31c9ab but not on the commit prior.
Note that k8 is a test that generates very large binaries, requiring trampolines and maybe splitting text sections. It seems to me we've had issues with the functab with large programs like this in the past, but I'd like to see if I can find anything on it.

Yes, it is related to multiple text  sections. That CL is supposed to take that into consideration. But it is possible  something is not really correct.


Ship bailing due to out of memory 2.8

Ship bailing due to out of memory
~
urbit 2.8
boot: home is /root/poldec-tonteg
loom: mapped 2048MB
lite: arvo formula 2a2274c9
lite: core 4bb376f0
lite: final state 4bb376f0
loom: mapped 4096MB
boot: protected loom
live: mapped: GB/3.601.547.264
live: loaded: KB/16.384
boot: installed 657 jets
---------------- playback starting ----------------
pier: replaying events 814260067-814263500
pier: (814260566): play: done
pier: (814261066): play: done
pier: (814261566): play: done
loom: fault mprotect (10619): Out of memory
Assertion '0' failed in pkg/noun/manage.c:1796
home: bailing out
pier: EOF
pier: work exit: status 0, signal 6

root@poldec-tonteg:~# free -m
              total        used        free      shared  buff/cache   available
Mem:           7983         134        2896          15        4951        7465
Swap:          4095         776        3319

@matthew-levan @barter-simsum @belisarius222 @joemfb

I doubled the ram of the box and it's still failing

I'm also seeing this on 2.9 (aarch64 with --loom 32). Booting with --no-demand appears to be stable.

Closing in favor of #461.


WIFIDIAG Cluster Attributes not able to be read

Problem
Some of the attributes on the Wifi Diagnostics cluster are failing for the following common reason: CHIP:DMG: Error retrieving data from clusterId: 0x0000_0036, err = ../../examples/all-clusters-app/linux/third_party/connectedhomeip/src/platform/Linux/ConnectivityUtils.cpp:487: CHIP Error 0x000000AE: Read from file failed
Attributes are the same for TC-WIFIDIAG-1.1 and TC-WIFIDIAG-1.2 , each test case just looks at the chip-tool request vs the chip-all-clusters-app response to the same chip-tool command.
Affected Attributes:

channel-number
rssi
beacon-lost-count
current-max-rate

Test Cases
TC-WIFIDIAG-1.1 | Attributes with server as DUT

https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/cluster/wifi_diagnostics.adoc#tc-wifidiag-1-1-attributes-with-server-as-dut

TC-WIFIDIAG-1.2 | Attributes observation at client as DUT

https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/cluster/wifi_diagnostics.adoc#tc-wifidiag-1-2-attributes-observation-at-client-as-dut

Testing Setup
chip-tool and chip-all-clusters-app: commit f51eeb39de1e58c8f55a13cfc59782e8e11c6948 (HEAD -> test_event_7, origin/test_event_7)
Logs
WIFIDIAG Cluster Attributes Logs.txt

Update for TE 7.5: Same Failures
commit: b0af6ba42b6ce494748f18ecf3e054a400b8f45b
Same issue persists across the entire issue filed above.
Lines of Interest from Logs
(Only from consolidated_mac_accessory_TC-WIFIDIAG-1-1.txt, as both logs are the same commands and same failures)

channel-number

[1642731832650] [90506:3876295] CHIP: [TOO] Run command failure: ../../commands/common/CHIPCommand.cpp:232: CHIP Error 0x00000032: Timeout

rssi

[1642731877931] [90509:3876831] CHIP: [TOO] Run command failure: ../../commands/common/CHIPCommand.cpp:232: CHIP Error 0x00000032: Timeout

beacon-lost-count

[1642731923340] [90512:3877318] CHIP: [TOO] Run command failure: ../../commands/common/CHIPCommand.cpp:232: CHIP Error 0x00000032: Timeout

current-max-rate

[1642732146056] [90529:3879224] CHIP: [TOO] Run command failure: ../../commands/common/CHIPCommand.cpp:232: CHIP Error 0x00000032: Timeout

Logs
consolidated_mac_accessory_TC-WIFIDIAG-1-1.txt
consolidated_mac_accessory_TC-WIFIDIAG-1-2.txt
.

Closing this issue
Moving to duplicate issue to track the problem from now on: https://github.com/project-chip/connectedhomeip/issues/13418#issuecomment-1018837380


Startup Flow: Linux

Currently, if no valid scheme is given from the command line, we prompt the user for a scheme before loading the main dialog.  In Linux this causes an issue where we have 3 icons in the Ubuntu dock, but one visible window.
Most likely there is weird behavior where the dialogs created before the main frame are being counted as top level windows and get their own icon, and for whatever reason, the icons remain, but only in Linux.  We are destroying the windows, so I'm not sure what is going on.
With all of that said, we can probably handle this situation by creating the main frame first and have the user prompts be children. Or maybe we need to destroy these dialogs differently.  Either way, the issue needs some kind of resolution before the next release.

https://github.com/facelessuser/subclrschm/commit/4ca35124418d6c9bb89757cc14c8086b07b14f9e
https://github.com/facelessuser/subclrschm/commit/72309f843c96fb6e3025382d6db0f394533cea3f
https://github.com/facelessuser/subclrschm/commit/449abbc597ac3b497b41892328633488e2f0ea58
Ugh.  This was more annoying than it had to be, but it is fixed now.  Probably could use some reworking in the future, but I'm going to get this out hopefully today, so reworking will come another day.


Add ssl offloading middleware

@gabrielarrais Do you have anything specific about SSL offloading you'd like to recommend/suggest? Like, are you suggesting that Caddy become an SSL offloader for other backend systems? That could perhaps be built into the proxy middleware...

I think he want Caddy to work like nginx ssl directive...
(Example from: http://nginx.com/blog/nginx-ssl/)
And now you want to add HTTPS support, with NGINX decrypting the traffic using the certificate and private key and communicating with the backend servers over HTTP:
upstream backends {
server <IP_ADDRESS>:80;
server <IP_ADDRESS>:80;
}
server {
listen              80;
listen              443 ssl; # The ssl directive tells NGINX to decrypt
# the traffic
server_name         www.example.com;
ssl_certificate     www.example.com.crt; # This is the certificate file
ssl_certificate_key www.example.com.key; # This is the private key file
location / {
proxy_pass http://backends;
}
}

I see. If that's the case, that could definitely be a good enhancement for the existing proxy middleware. I think. Someone is welcome to look into this since I may not get around to it for a while.

Ok I can look at this =]

@gabrielarrais You can try this caddyfile?
<IP_ADDRESS> {
   tls web/server.crt web/server.key
   proxy / http://<IP_ADDRESS>:8080/
}

Using proxy + tls works for me, please try this and see if works for you =]


WinUI Design Toolkit Bug: Status Bar component is missing

WinUI Toolkit Bug Report

Name of component or part of Toolkit:

Describe the bug: Status bar is a vital part of every app which is missing.

Version Info: 2.13.0

Additional context:

I've created this community file for all. Please check status bar via Figma Link.

The Status bar is not a Windows UI control, so we haven't included it in the Windows UI toolkit.


Source SDK Base 2013 Multiplayer

When i try to open Source SDK Base 2013 Multiplayer or Source SDK Base 2013 singleplayer it always opens onto a Video Stress Test for Valve. Can you help with this problem i'm trying to make maps for Gmod and such. Please help me fix it.

We're not valve.


refactor(config): use cspell-lib to handle findConfig function

This PR closes #13.
By analyzing the source code, the actual findConfig responsability is to find a path to the specified config file from CLI arguments.  So we can just reimplement this logic part from the original cspell-libs readConfig function.
By adding cspell-lib as a direct dependency, package managers with caching store strategies like pnpm now can interact correctly with the CLI.

Done, removed cspell from tsup.config.ts

Great, thanks! I added json to the editorconfig, could you reformat your changes?

Great, thanks! I added json to the editorconfig, could you reformat your changes?

Done :)

tsconfig.json and tsconfig.eslint.json too please <3. Also, did you test this? It looks like it will work great, but would be good to make sure.

I ran yarn test after changes and the tests passed correctly, but good to notice:
I found that (at least in Windows) after installing rspell and running it for the first time, the code won't find cspell.json on default configuration path and it will throw an error. I'll write some tests to check this, but I'll be interesting to work on this specific problem on a different issue!

Oh, really? You bring up a good point, that the cli should probably create a config for the default path if none is found there, otherwise the whole default path thing would be useless. But I thought I caught all the potential errors if the file doesn't exist haha. If you want to look into that, that would be great! thank you so so so much again, I'm really happy someone is interested in this project <3
As for the testing thing, the tests are kind of outdated, so they only cover the actual typo fixing functionality and don't interact with config stuff. So I think just creating a cspell.json in the project directory, running rspell, using "ignore all" on a word and seeing if it appears there would be a good manual test.

So, what do you think about just leaving this PR by adding a single passing test and working on this other on a different PR/Issue? So we can separate the topics better
describe('findConfig', () => {
	test("should return the input if it's a valid path", async () => {
		const input = 'local_cspell.json';

		const config = await findConfig(input);

		expect(config).toBe(input);
	});
});

New test added and .json files formatted

I don't think that test is very productive because the function just returns the path if there's one specifically inputted. I think the best thing to do would be to mock whatever fs api that cspell-lib uses to look at files, make it think there's a cspell.json in the cwd, and see if findConfig returns that path. But I'm fine with merging this with no tests as long as you can quickly test that yourself manually, and then we can revisit it in a different pr or something like that.

Thanks for the contribution!!


Add doc about how to pull encrypted Nydus image with image-rs

Fixed the bug 'read only filesystem' when start container with Nydus image rootfs.
Add doc about how to pull encrypted Nydus image with image-rs to accelerate container startup process.

/retest

All CI passed, thanks @taoohong


Duplicate webviews may exist after refreshing parent's tree item

Subscription1
|-- Cache1 <=> webview1
|-- Cache2 <=> webview2

Currently, webviews are associated with a particular instance of a Cache tree item. But when you refresh the parent node of a Cache tree item (i.e. Subscription tree item), it discards all of its current child items and re-creates new ones.
Say for instance the user has the cache properties webview open for Cache1 and then refreshes the subscription tree node. If they click on the cache tree node again, instead of focusing the existing webview it opens another webview, resulting in two cache properties webviews for the same cache.
Demo:

This also happens if you do something like this:

Click on the tree item for list1. This should open up the webview.
Change the key filter to list1.
Click on the tree item for list1. This opens another webview. Now there are two webviews for list1 open.


Add tip on how to add DEPLOYMENT_BRANCH in Portal

I'm unable to get the Azure CLI working on my computer so I like when the guides show how to accomplish something in the Portal as another option. This PR adds instructions on how to change DEPLOYMENT_BRANCH in the Portal.

@pamelafox : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@cephalin,
Can you review the proposed changes?
IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge. Thanks.

#sign-off


Reuse existed containers upon scale.

With docker-compose up, usually we can reuse the existed container.
Question
How can we achieve the same thing with docker service scale ...?
So, I want to reuse some data that was modified under that existed containers.
Example
$ docker service create --replicas=4 --name=test_loop ...
$ docker ps -a
... STATUS				PORTS	NAMES
... Up 19 seconds				test_loop.1.dk4rs8eqbow3wbbjch34y287z
... Up 19 seconds				test_loop.2.0t7p2ab0wabrsmt6ljne3bh91
... Up 19 seconds				test_loop.3.57n0vhfcdfwbwbmlo0f1hghsd
... Up 19 seconds				test_loop.4.7wt2wittys6m37vehwffbb8ln

$ docker service scale test_loop=2
$ docker ps -a
... STATUS				PORTS	NAMES
... Up 3 minutes				test_loop.1.dk4rs8eqbow3wbbjch34y287z
... Up 3 minutes				test_loop.2.0t7p2ab0wabrsmt6ljne3bh91
... Exited (137) 6 seconds ago	test_loop.3.57n0vhfcdfwbwbmlo0f1hghsd
... Exited (137) 6 seconds ago	test_loop.4.7wt2wittys6m37vehwffbb8ln

$ docker service scale test_loop=4
$ docker ps -a
... STATUS				PORTS	NAMES
... Up Less than a second		test_loop.4.eplof06ud9860udsyenjckqi5  <--- Instead of creating this
... Up Less than a second		test_loop.3.4l8dvn361q1lba1dqakx74hvn  <--- Instead of creating this
... Up 4 minutes				test_loop.1.dk4rs8eqbow3wbbjch34y287z
... Up 4 minutes				test_loop.2.0t7p2ab0wabrsmt6ljne3bh91
... Exited (137) 1 minutes ago	test_loop.3.57n0vhfcdfwbwbmlo0f1hghsd  <--- I want to reuse this
... Exited (137) 1 minutes ago	test_loop.4.7wt2wittys6m37vehwffbb8ln  <--- I want to reuse this

With docker-compose up  previous instance [exited] is deleted and new instance is created, thats why exited containers are not visible in docker ps -a
docker run  has option of --rm to auto-remove container on exit, but such option is not available in docker service command.
Also docker service tries to maintain the state and instances, that is defined during docker service [create|update] i.e. auto-recreate new instance for failing containers.
If you wish to reuse data, you should store such data outside container. May be this will be helpful https://docs.docker.com/engine/tutorials/dockervolumes/

@kunalkushwaha Thank you very much for the quick answer.
However, it seems running docker-compose actually reuses the exited containers if available.
For example,

I run a simple compose, then it creates container id, e.g. 6ac238e7323e
I create a file test.txt inside this container.
I stop this container. The docker ps -a shows 6ac238e7323e has exited.
I run the same compose up command. The docker ps -a shows 6ac238e7323e is up. And I found my test.txt file is still there.

BTW, the docker service update --replicas= ... is interesting to me, but it seemed updating the --replicas acts quite the same as docker service scale ... (not reuse the exited containers).
I'm not sure if we can have this kind of docker-compose behavior with Docker swarm 1.12 when scaling the replicas.
Also, for the volume solution, if needed, I may need to do some wrapper solution. Since the service I'm running (Hyperledger) requires a separated storage. See. https://github.com/hyperledger/fabric/issues/2316

Yes, you are right.
Stopped containers can be started again using docker start

I'm not sure if we can have this kind of docker-compose behavior with
Docker swarm 1.12 when scaling the replicas.

Not sure about docker swarm, as scheduling logic creates new instance.
Could be added as option in scheduling logic, but not sure, about other
complexities.
On Fri, Aug 5, 2016 at 3:47 PM, Siriwat K<EMAIL_ADDRESS>wrote:

@kunalkushwaha https://github.com/kunalkushwaha Thank you very much for
the quick answer.
However, it seems running docker-compose actually reuses the exited
containers if available.
For example,

I run a simple compose, then it creates container id, e.g. 6ac238e7323e
I create a file test.txt inside this container.
I stop this container. The docker ps -a shows 6ac238e7323e has exited.
I run the same compose up command. The docker ps -a shows 6ac238e7323e
is up. And I found my test.txt file is still there.

BTW, the docker service update --replicas= ... is interesting to me, but
it seemed updating the --replicas acts quite the same as docker service
scale ... (not reuse the exited containers).
I'm not sure if we can have this kind of docker-compose behavior with
Docker swarm 1.12 when scaling the replicas.
Also, for the volume solution, if needed, I may need to do some wrapper
solution. Since the service I'm running (Hyperledger) requires a separated
storage. See. hyperledger/fabric#2316
https://github.com/hyperledger/fabric/issues/2316
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/docker/swarmkit/issues/1316#issuecomment-237769813,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACoXIW1FyATfoeAjKaLvo5LJOJLH_5mDks5qctyRgaJpZM4JdUkP
.

--
Regards,
Kunal Kushwaha

Also docker swarm mode is not yet supported by docker-compose 1.8 https://github.com/docker/compose/issues/3656

Thanks for the info @kunalkushwaha
I'm not sure if we can reuse the container with swarm right now.
If not available, I may +1 for those who need this function :)

Reusing containers is sort of at odds with the idea of swarmkit, which is that individual tasks are ephemeral. There are reasons why you wouldn't want to reuse containers. For instance, perhaps something about the task spec has changed between scale downs and ups? You would have to check the dead container against the task spec.
That's pretty much all there is to say on this, so I'm gonna go ahead and close it. You're definitely welcome to reopen this issue if you feel it hasn't been adequately addressed.


A compile error while using the C++ API

I tried to use the function searchSequence in CompactedDBG, and there was error showed:
   include/bifrost/CompactedDBG.tcc:5593:62: error: passing ‘const CompactedDBG<DataAccessor<MyBool>, DataStorage<MyBool> >’ as ‘this’ argument discards qualifiers [-fpermissive]

   const UnitigMap<U, G> um = findUnitig(seqs.c_str(), pos_seq, seqs.length(), mhi);

And I found that searchSequence is a const member function but findUnitig is not. So I modified searchSequence to a non-const member function and no compile error this time.

Good catch! I'm actually going to propose findUnitig() and searchSequence() in const and non-const member functions to be able to work with both ;)
As I have some new code for searchSequence() (should be slightly faster than the current one), I will also update that at the same time. Should be ready for tomorrow evening but in the meantime, I keep this issue open.
Thanks!

findUnitig() and searchSequence() are now available as const and non-const member functions.

findUnitig() and searchSequence() are now available as const and non-const member functions.

Thank you!


Fix data race errors

Hi @saoussen , please, can you rebase this PR. The communication layer has changed a lot. I just want to make sure your changes are compatible with what exists on main. Thanks :)

@adecaro I made the requested changes


Backward compatibility issue

In https://github.com/open-telemetry/opentelemetry-collector-builder/commit/c9fe6966eba55a28368caccb589b9c913adba7e5 , the template file internal/scaffold/components.go has been changed to use the the new API, and hence the builder does not work for earlier versions

@fangyi-zhou should we close this one?


verifying my tokens

Could you please verify my slp tokens:
Digital Fils : ID   a7af0ffe54a50b7777a27d7ad3c9d077f710a3bec89a6b5eb4425023a546d5b0
Bitcoin Fils : ID   cf8e566d2bf49ab0f52d83c1db6e0f7785ee625d6cb9891f77f4401a5b3816f2

@BTCfils hello. We do not accept pull requests, please open an issue and we will do the icons for you.


Get localStorage token from nuxtServerInit

So my problems is how would I get the localStorage jwt token that i've been using to pass on the header for my separate apollo graphql server
Below is how I try to setup the token
store/index.js
import Vuex from "vuex";
import { Auth, Login } from "~/apollo/gql/Auth.gql";
import Cookie from "js-cookie";
export const state = () => ({
  auth: null,
  token: ""
});

export const mutations = {
  SET_TOKEN(state, data) {
    state.token = data;
  }
};
export const actions = {
  async nuxtServerInit(store, context) {
    let client = context.app.apolloProvider.defaultClient;
    let { data } = await client.query({
      query: Auth,
      fetchPolicy: "network-only"
    });
    const token = Cookie.get("token");
    console.log(token);
    console.log(context, store, data);
  },
  async login({ commit }, data) {
    const client = this.app.apolloProvider.defaultClient;

    const { data: val } = await client.mutate({
      mutation: Login,
      variables: {
        data
      }
    });

    localStorage.setItem("token", val.result.token);
    Cookie.set("token", val.result.token);
    const token = Cookie.get("token");
    console.log(token);
    commit("SET_TOKEN", val.result.token);
  }
};

I'll use my login function to send a mutation request on my api server then get that token back
It works when i Login but when on the case of reloading the page
the token is not set on nuxtServerInit so what I wanted to do is how do I store the token somehow on the request object of the nuxt server so whenever I try to check the user if he/she is authenticated I will just set the token on startup on the Token state
Also tried the cookie library
but seems I don't get the idea how cookie works maybe give me a simple point of view of how cookie session work?
I tried to find the session property on the request object but it's not there
this some part of the config for apollo-client
// auth token
  let getToken = () =>
    process.server ? ctx.req.session : localStorage.getItem("token");

  // middleware
  const middlewareLink = new ApolloLink((operation, forward) => {
    if (process.server) console.log(ctx);
    operation.setContext({
      headers: {
        authorization: `Bearer ${getToken()}`
      }
    });
    return forward(operation);
  });

  const wsLink = process.client
    ? new WebSocketLink({
        uri: process.env.SUBSCRIPTIONS_ENDPOINT,
        options: {
          reconnect: true,
          connectionParams: () => {
            const token = getToken();
            return {
              Authorization: token ? `Bearer ${token}` : null
            };
          }
        }
      })
    : "";

You cannot get localstorage from server side, localstorage is a client only API, your approach with cookies is the right one.


Issue 1799

Please complete the following checklist:

[x] I have read the guidelines in CONTRIBUTING.md.

[x] I have documented all public-facing changes in CHANGELOG.md.

[x] This pull request includes code, documentation, or other content derived from external source(s). If this is the case, ensure the external source's license is compatible with scikit-bio's license. Include the license in the licenses directory and add a comment in the code giving proper attribution. Ensure any other requirements set forth by the license and/or author are satisfied. It is your responsibility to disclose code, documentation, or other content derived from external source(s). If you have questions about whether something can be included in the project or how to give proper attribution, include those questions in your pull request and a reviewer will assist you.

[x] This pull request does not include code, documentation, or other content derived from external source(s).

Note: REVIEWING.md may also be helpful to see some of the things code reviewers will be verifying when reviewing your pull request.

The biggest Problem I have with this pull request, is that it is effectively copying  Biopython's implementation.  The only change here is to explicitly define available parameters instead of the simpler kwargs, with perhaps inclusion of more error messages.
I am also not a fan of global variables controlling the api_key for this module. It seems more appropriate for the api_key to be an instance attribute here.

Well it really isn't a copy of anything, although it does take some inspiration from biopython's implementation of course. I prototyped a bunch of different implementarions at the time and this was the one that felt simpler to use and more in line with skbio's API, which to me is an indication that it simply happens to be the best way to implement these features. It would be no good doing things differently just for the sake of doing them differently. This implementation is also a big improvement over biopython's implementation in terms of reliability (the error messages you mentioned, while biopython usually fails silently), convenience (converting the resulting querry to skbio objects such as alignments etc instead of ) and documentation. Anyone that has used biopython for entrez querying knows how horribly awkward it feels as a consequence of the issues i mentioned (returning the objects as streams to be parsed is specially clunky).
Im not opposed to encapsulating the functions inside a EntrezQuery class to have the API key as an instance attr if you think this is a supperior option. I personally agree that module variables are a bit effy, although to me in this case they are doing more good than bad by avoiding clutter and keeping the API more functional. Another option would be adding the API key as an argument for all functions.
Cheers


feat(Button): Make content-projection for icon possible for any other icon definitions like fa-icon or

Feature Description
Actually the button HTML is implemented like this:

<ng-content select=".material-icons:not([iconPositionEnd]), mat-icon:not([iconPositionEnd]), [matButtonIcon]:not([iconPositionEnd])">
</ng-content>

<ng-content></ng-content>

<ng-content select=".material-icons[iconPositionEnd], mat-icon[iconPositionEnd], [matButtonIcon][iconPositionEnd]">
</ng-content>

We can see that the mat-icon is projected into the button by using css classes. This has the limitation to use the button only with mat-icon if the button should habe a icon. But whats about fa-icon or ?
It would be great to add a button like:
  <button mat-icon-button (click)="click()">
    <fa-icon [icon]="faIcon"></fa-icon>
  </button>

In this case the fa-icon contains a svg like mat-icon can do. It would be cool to project fa-icon into the button like mat-icon. This can be done with a directive name like icon or any other name.
Use Case
Use mat-button also with other icon components like  or fa-icon.

You can project a different icon by setting the matButtonIcon attribute.

Okay, I see. I think we can close this feature request.


README file "Upcoming: local notifications, background-state Rx queue (iOS equivalent)" yet commit says you support android local notifications..

And thanks for this great library, starting integration with it now...

Local notifications are supported, so you're right - the readme is outdated. Best of luck, we'll try our best to help out!

Hi! Thanks for adding this support.
I was wondering if there is a way to set a custom sound and fire date for these Android local notifications.

Great thanks for the quick response, I've assigned a pull request with the README file update, chears.


isContextSensitive in the checker returns false for literals

For example, it should probably return true for 1 and { x: 1 }.
We'd like to use this in the language service to make intelligent decisions about which extracted constants need type annotations.  The test extractConstant_ContextualType shows a case where the current behavior causes an incorrect decision to omit a type annotation.

FYI @sandersn @weswigham

Quick note about the problem we hit when changing it experimentally:
Arrow functions that are only context-sensitive in their return value (such as () => 12 or (_n: string) => "foo") are treated incorrectly in a number of ways. The worst was adding a this-parameter from the contextual type. I'm not sure if isContextSensitive should return false for these functions or if various code paths that use isContextSensitive need to be understand that they might now be operating on an arrow function.
@weswigham I think there might have been another issue with the change. Do you remember anything else?

The way inference works, we use fixed non-contextual types from the first pass to apply contextual types to things in the second pass. When literals become context-sensitive, we no longer get any useful literal types to apply during the second pass, resulting in spurious inference failures. It feels like we need a special kind of context-sensitive for literals that behaves more like a 'contextually refined' type, where it can still contribute contextual information during the first pass, but can also be refined by contextual information in the second pass to handle this correctly without degrading existing inferences.

The isContextSensitive function really has nothing to do with literal types. As the comment on the function states: "Returns true if the given expression contains (at any level of nesting) a function or arrow expression that is subject to contextual typing." It's basically a helper function that is used to determine if we should assign contextual types to parameters without type annotations.

@amcasey I can't seem to find a test called extractConstant_ContextualType anywhere. Can you post the code here?


"Dead kernel" after installing and runing "IHaskell notebook"

There's problem with starting notebook after installing.
➜   IHaskell notebook
...
2015-03-06 11:28:44.750 [NotebookApp] Kernel started: 9cbc74df-e4e5-4a57-a947-3c6985797860
IHaskell: Failed to load interface for ‘IHaskell.Display’
Use -v to see a list of the files searched for.
Failed to load interface for ‘IHaskell.Display’
Use -v to see a list of the files searched for.
Failed to load interface for ‘IHaskell.IPython.Stdin’
Use -v to see a list of the files searched for.

2015-03-06 11:28:47.749 [NotebookApp] KernelRestarter: restarting kernel (1/5)
WARNING:root:kernel 9cbc74df-e4e5-4a57-a947-3c6985797860 restarted
IHaskell: Failed to load interface for ‘IHaskell.Display’
Use -v to see a list of the files searched for.
Failed to load interface for ‘IHaskell.Display’
Use -v to see a list of the files searched for.
Failed to load interface for ‘IHaskell.IPython.Stdin’
Use -v to see a list of the files searched for.

I use OS X 10.9.4
➜ ghc --numeric-version
7.8.3
➜ cabal --version
cabal-install version <IP_ADDRESS>

There's the same problem with console
IHaskell console
Using system-wide IPython.
IHaskell: Failed to load interface for ‘IHaskell.Display’
Use -v to see a list of the files searched for.
Failed to load interface for ‘IHaskell.Display’
Use -v to see a list of the files searched for.
Failed to load interface for ‘IHaskell.IPython.Stdin’
Use -v to see a list of the files searched for.

Resolving dependencies...
Configuring ipython-kernel-<IP_ADDRESS>...
Building ipython-kernel-<IP_ADDRESS>...
Installed ipython-kernel-<IP_ADDRESS>
Configuring ihaskell-<IP_ADDRESS>...
Building ihaskell-<IP_ADDRESS>...
Installed ihaskell-<IP_ADDRESS>

Can you advice me how to debug it?

I've seen this error occasionally and often solved it by cabal install-ing from the ihaskell directory again; I'm not totally sure what causes it.
Try doing import IHaskell.Display and :m + IHaskell.IPython.Stdin in ghci, does that work?

I'm not sure what the problem was. I removed all dependent packages and reinstalled... it works :).
Thanks @gibiansky, I already :heart: :heart: :heart: IHaskell.

@slon1024 Good to hear that reinstalling worked; I've had the same experience. I think it has to do with *.hi files becoming somehow outdated, or perhaps not matching the versions between different installs or something.

I have encountered a similar problem when running iHaskell in Jupyter.
The "dead kernel" message which emerges on a startup seems to be related to
the fact that the pattern matching in the function 'replyTo' defined in main/Main.hs
is not exhaustive. The latter can be fixed by appending to the definition the following:
replyTo _ _ _ ks = liftIO $ do
return (ks, SendNothing)
The compiler is going to issue a warning about the overlapping instances,
so these lines should be placed after all other partial definitions of 'replyTo'.

Which version (of ipython/jupyter, ihaskell etc.) are you using? This issue seems to have been fixed a long time ago.

The latter can be fixed by appending to the definition the following:
replyTo _ _ _ ks = liftIO $ do
return (ks, SendNothing)

This also has some problems of it's own. A catch-all case removes errors about incomplete pattern matches, which are very helpful when, e.g. a new constructor is added to an argument. In the current state, any changes from the IPython side usually become "kernel died" errors, which might be intentional. @gibiansky would have more to say about this.

You are right, I just haven't noticed the commit ceb3abdd5187c749374e69f3c56f8f5a480bc9ce (#569).
Sorry, it's clearly my mistake! It is great if the problem is not there.
I have actually encountered the "dead kernel" message in the middle of August and had to fix it (I should have reported then).
You can reproduce the effect taking cab4cfdf4b31f3ef7111089c75a9c3236734bd19 which corresponds to ihaskell-<IP_ADDRESS>.
Unfortunately, I haven't built the latest gibiansky/IHaskell commit yet (I run out of 6GB RAM compiling ihaskell-widgets).
Anyway, assume that you have:
Arch Linux, kernel 4.2.1-1-ARCH
ncurses 6.0-3
zeromq 4.1.3-1
cairo-infinality-utimate 1.14.2-1
pango 1.36.8-1
blas 3.5.0-1
lapack 3.5.0-1
python 3.4.3
ipython 4.0.0
ghc 7.10.1
firefox 41.0
cabal <IP_ADDRESS>
I run the following sequence:
clone https://github.com/gibiansky/IHaskell.git,
checkout cab4cfdf4b31f3ef7111089c75a9c3236734bd19,
append 'hmatrix >= 0.10 && <0.17' and 'diagrams-lib <= <IP_ADDRESS>' to library/build-depends in ihaskell.cabal.
After that I install everything in a cabal sandbox, execute 'cabal exec bash',
run 'jupyter notebook', launch firefox, and open a new Haskell notebook in Jupyter.
This leads to a dead kernel.
The shortest way to get rid of it is to make 'replyTo' in main/Main.hs total by appending
replyTo _ _ _ ks = liftIO $ do {return (ks, SendNothing)}

Yup. ihaskell-widgets does give me a lot of pain when compiling (I run out of 8gb of ram). I added a workaround to the build script, which promptly skips building ihaskell-widgets :smile:.
git clone https://github.com/gibiansky/IHaskell.git
cd IHaskell
./build.sh all no-widgets

Surprisingly enough, ghc-7.8.4 on travis manages to build this under 3gb of ram. Just after a system upgrade (ghc-7.10.2 became available today, hurray !!), I did not need to do anything other that just run those commands.

Thanks a lot for pointing this out.
I have now upgraded to ghc 7.10.2 and it does a much better job than 7.10.1!
I am able to compile the iHaskell commit 8104d98f7b733df6f993c4bb6138def874e2f79e in one shot.
My machine has 6Gb RAM and the compilation of ihaskell-widgets consumes around 80% of memory in its highest point. The variant with no-widgets option and then compiling the widgets separately works fine as well, of course.
There is a small typo in ihaskell-display/ihaskell-widgets/ihaskell-widgets.cabal.
In the library/build-depends section the upper bound for singletons is <IP_ADDRESS>, while it should apparently be just 2.0:
singletons >= 0.9.0 && <2.0
For some reason cabal (upgraded to <IP_ADDRESS>) ignores the inequality "<<IP_ADDRESS>" and proceeds with an installation of the most recent version into my sandbox causing a long sequence of compilation errors for the widgets.The inequality "singletons <2.0" results in an installation of <IP_ADDRESS> what is just fine.

@artuuge I suggest that you treat that as a bug in IHaskell and file an issue, and a PR if you're willing :wink:

I had the same problem. Running cabal install resolved it for me.


Jaeger rename

We will need to change links inside of the .csproj file as well as copyright info once the move has happened.

LGTM! 👍


Click at position not on screen

Assume we move window half in the screen and half out of the screen.
import pyautogui as g
g.moveTo(-100, 100)
g.click()

The click seems not working

well ,maybe pyautogui can't click something that isn't in main screen just like it don't support mutimonitor


GPTQ Support Dict-type Calibration Data

Type of Change
enhance
Description
support gptq function when calibration dataset consists of dict
{
    'input_ids': xxx
    'attention_mask': xxx
}

etc.
Expected Behavior & Potential Risk
Dict input
How has this PR been tested?
PreCI
Dependency Change?
None

Hey every one I am  new to open-source contribution and willing to contribute to intel projects , can some provide me with some good first issues ,  also provide the community channel.....


Native AOT support

I opened https://github.com/IdentityModel/IdentityModel.OidcClient/pull/451 in IdentityModel.
If this is where ongoing maintenance is and the PR is useful, should I open it against this repo too?

Yes, migrating all the open source work we sponsor into this mono repo is in flight, so migrating your changes here would help us out a lot. Thanks!


[HAMMER] [WIP] Add timezone for report cols

It's a hack to get the timezones on reports on hammer to be correct.
needed for hammer version of https://github.com/ManageIQ/manageiq/pull/19285
@lpichler what do you think about this mess? 😆
Edit, maybe actually this should've been done by
https://github.com/ManageIQ/manageiq/pull/18438
... which never got backported.

Closing because https://github.com/ManageIQ/manageiq/pull/18438 fixed this but never got backported to hammer. See https://bugzilla.redhat.com/show_bug.cgi?id=1759776#c10.


HDMap: add GetVersion interface

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Please sign CLA.

This pull request will be temporarily closed. Please reopen after signing the CLA.


create global service has no response

Output of docker version:
$ docker version
Client:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 23:54:00 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 23:54:00 2016
 OS/Arch:      linux/amd64

Output of docker info:
core@worker-10-50-1-101 ~ $ docker info
Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 23
Server Version: 1.12.0
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null overlay
Swarm: active
 NodeID: 5xig0dyp26ebrxp1g4043fk9g
 Is Manager: true
 ClusterID: 5y9yaqelshau40nm55awdk41h
 Managers: 1
 Nodes: 7
 Orchestration:
  Task History Retention Limit: 10
 Raft:
  Snapshot interval: 10000
  Heartbeat tick: 1
  Election tick: 3
 Dispatcher:
  Heartbeat period: 5 seconds
 CA configuration:
  Expiry duration: 3 months
 Node Address: <IP_ADDRESS>
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.6.3-coreos
Operating System: CoreOS 1068.6.0 (MoreOS)
OSType: linux
Architecture: x86_64
CPUs: 40
Total Memory: 125.9 GiB
Name: worker-10-50-1-101
ID: ATRO:LG4Q:YR6Z:SOJM:KIFV:Q5PM:5QGK:OWYD:FWQA:OENT:Y5VT:4YJD
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
 IP=<IP_ADDRESS>
 CPU_NUMBER=40
 MEMORY=131990668kB
 DISK=hdd
 VERSION=1068.6.0
 NAME=CoreOS
Insecure Registries:
 <IP_ADDRESS>/8

Additional environment details (AWS, VirtualBox, physical, etc.):
CoreOS on physical :
$ cat /etc/os-release
NAME=CoreOS
ID=coreos
VERSION=1068.6.0
VERSION_ID=1068.6.0
BUILD_ID=2016-07-12-1826
PRETTY_NAME="CoreOS 1068.6.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

cluster:
core@worker-10-50-1-101 ~ $ docker node ls
ID                           HOSTNAME            STATUS  AVAILABILITY  MANAGER STATUS
46w7emjaro9n5wqsdvlb2axeo    worker-10-50-1-102  Ready   Active
5bvwh7kzlxxe2srss0samjsyt    worker-10-50-1-106  Ready   Active
5xig0dyp26ebrxp1g4043fk9g *  worker-10-50-1-101  Ready   Active        Leader
6ftk92pv02y85vpfg70yeqan6    worker-10-50-1-107  Ready   Active
7l65yv0zi3qbtojm01jqad6b9    worker-10-50-1-103  Ready   Active
b2v8dld2zuqjgfqx3p1p6oxy1    worker-10-50-1-104  Ready   Active
ezqa3ggqrg5e2w5d3mogmelxo    worker-10-50-1-105  Ready   Active

ipvs info:
core@worker-10-50-1-101 ~ $ cat /proc/net/ip_vs
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port Forward Weight ActiveConn InActConn

Steps to reproduce the issue:
core@worker-10-50-1-101 ~ $ docker service create --name test-nginx --network mynet --network ingress --mode global -p 80:80  nginx
0zcqmh5e7y6el81sz7ayn6ame
core@worker-10-50-1-101 ~ $ docker service ls
ID            NAME              REPLICAS  IMAGE                           COMMAND
0zcqmh5e7y6e  test-nginx        global    nginx
5qld98b1o02z  echo              1/1       dhub.yunpro.cn/shenshouer/echo
btgpv1p1tu5k  z7a7e4ec7386b738  5/5       dhub.yunpro.cn/shenshouer/echo
core@worker-10-50-1-101 ~ $ docker service ps test-nginx
ID                         NAME            IMAGE  NODE                DESIRED STATE  CURRENT STATE           ERROR
5uvokw95ykqq9slfl6t4dhw9h  test-nginx      nginx  worker-10-50-1-105  Running        Assigned 8 seconds ago
03a23l6yifb1msbhmstwafzgj   \_ test-nginx  nginx  worker-10-50-1-104  Running        Assigned 8 seconds ago
dn35soqpfloxtp211j69w4d54   \_ test-nginx  nginx  worker-10-50-1-102  Running        Assigned 8 seconds ago
7nqlxms8fytdw50tg2d3qqfiw   \_ test-nginx  nginx  worker-10-50-1-101  Running        Assigned 8 seconds ago
cxj18mtwciu551jcujhipboh7   \_ test-nginx  nginx  worker-10-50-1-103  Running        Assigned 8 seconds ago
5k26le5prc908pwwxapggz94i   \_ test-nginx  nginx  worker-10-50-1-107  Running        Assigned 8 seconds ago
7jfubym41jffq6zlk79503fdk   \_ test-nginx  nginx  worker-10-50-1-106  Running        Assigned 8 seconds ago

Describe the results you received:
no nginx container running on any host in cluster after wait long time.
Describe the results you expected:
run ngninx container on all host in cluster
Additional information you deem important (e.g. issue happens only occasionally):
have no output log for create service

It worked for me after update the version of docker to 1.12.1


Use System.Text.Json instead of Newton.Json

Fixes #515

@microsoft-github-policy-service agree

/azp run

Thanks for your contribution.  This is an excellent improvement I think.
I added a simple benchmark to test the speed of just the export time. It generates a trivial report with N issues for setup and then exports the same report to JSON repeatedly. The benchmark shows about a 2x speedup, and about 10x less memory allocations.
Current main branch

Method
N
Mean
Error
StdDev
Ratio
Gen0
Allocated
Alloc Ratio

ExportRecordsToJson
1000
3.680 ms
0.0736 ms
0.1833 ms
1.00
23.4375
387.88 KB
1.00

ExportRecordsToJson
10000
30.532 ms
0.6079 ms
1.5138 ms
1.00
218.7500
3762.89 KB
1.00

This PR

Method
N
Mean
Error
StdDev
Ratio
Gen0
Allocated
Alloc Ratio

ExportRecordsToJson
1000
2.052 ms
0.0494 ms
0.1442 ms
1.00
-
52.17 KB
1.00

ExportRecordsToJson
10000
13.035 ms
0.2579 ms
0.4310 ms
1.00
15.6250
480.6 KB
1.00

/azp run

/azp run

This is relatively large in terms of files touched so I'm going to give it another look tomorrow to double check nothing got missed but I expect to merge it sometime tomorrow.

Thanks folks, I really appreciate your responsiveness. I will check this PR if I need to modify anything.

@dbalikhin I did a quick check of the json, sarif and html outputs compared to the latest release from main and everything looks good, as well as double checking that custom rules can still be specified and loaded properly.
Thanks again for the contribution, great work!

@gfs

Thanks again for the contribution, great work!

My pleasure.


apps: fix OpenWrt spelling

OpenWrt is spelled 'OpenWrt' and not 'OpenWRT'. Fix that.

Good catch, I never noticed rt was in lower case.


fix: debug print the full bytes

Either this one, or https://github.com/dashpay/dash/pull/6227 but, please, for the love of all that is right in this world, DO NOT print out THE WRONG NUMBER OF BYTES!!!!
Better yet, why are there different forms of serialization for the same object?

This should just consistently serialize to the same object because it's the same output script.

P.S. I realize that the byte discrepancy is probably just legacy nonsense from the BitCore devs that you inherited, and that other than accidentally calling the wrong serialize function by mistake when adding the AssetLockTx / ExtraPayload functions to the code, it's probably not your fault. I'm just frustrated because this codebase is always a nightmare to work in and it's red herrings all the way down. EVERY. TIME.

And I did take a quick look to see if I could fix the serialization but I couldn't find it. Probably in some UniValue macro 6 layers down...

Please follow the PR template when opening PRs. I've pasted in the template again for you. Please fill it in. Marking PR as a draft for now, please mark ready for review once done.

superseded by #6229


space key no longer triggers commit of selected mention item

This behavior existed to preserve parity with the pre-Lexical-editor behavior, but I think we should remove it because it will interfere with more complex @-mentions where the "query" does need to include spaces. (For example, think about @-mentioning a title of a doc with spaces in it.)
Test plan
CI

Confirmed Tab still works.


Instruction Dataset

Hi, great paper and congratulation for being accepted in EMLNP 2024. And thanks for publishing the collected dataset. Btw, would you also publish the data that was generated by GPT4? That would be great, thanks.

Thank you for your support of our work! You can find the processing reviews here: SEA_data

Yes I have checked the zip files, it contains review_json directory with all reviews in json format from crawled raw reviews. But as I think it doesn't contain the consolidated review that is generated by GPT4, does it?

Hello, the ZIP file contains the review comments integrated through SEA-S, which was distilled from GPT-4's integration capabilities. If you would like to access the original review content integrated by GPT-4, you can click here to visit.

Thanks for adding the original review content integrated by GPT-4


Avoid outputting Rosetta 2 caveats on Intel machines

Verification

[X] This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

Provide a detailed description of the proposed feature
Check for system type (Intel or Silicon) to filter which Caveats are shown. I'm on an Intel mac and I get the following Caveat:
"{cask} is built for Intel macOS and so requires Rosetta 2 to be installed.
You can install Rosetta 2 with:
softwareupdate --install-rosetta --agree-to-license
Note that it is very difficult to remove Rosetta 2 once it is installed."
This particular time it was for the "session" cask, but I have gotten it for other installs too.
What is the motivation for the feature?
This caveat is a warning that is simply unnecessary for Intel systems.
How will the feature be relevant to at least 90% of Homebrew users?
Less visual clutter and cleaner user experience. The first time I got it I had to investigate a little bit as to why I was getting this warning since I am on an Intel system and I had no clue what "Rosetta 2" was, nor did it imply anything about the warning being only for Silicon systems. My first though was that something in my system was misconfigured since the warning showed up.
What alternatives to the feature have been considered?
An alternative is to modify the caveat text to:
"For Apple Silicon systems:
{cask} is built for Intel macOS and so requires Rosetta 2 to be installed.
You can install Rosetta 2 with:
softwareupdate --install-rosetta --agree-to-license
Note that it is very difficult to remove Rosetta 2 once it is installed."

It looks like there is already a check that should ensure that this only gets displayed on arm machines.
https://github.com/Homebrew/brew/blob/29c22e0ab3fd9826cefd4b14c143266dea3bc391/Library/Homebrew/cask/dsl/caveats.rb#L143-L152
Could you run the following diagnostic commands locally and post the output here?
brew doctor

brew config

brew ruby -e 'p Homebrew::SimulateSystem.current_arch'

Actually, I figured this out. It's based on pre-computing the caveats when serializing formula for the API.
From the JSON API:
$ brew info Sleipnir
==> sleipnir: 4.6.6
https://www.fenrir-inc.com/jp/sleipnir/
Not installed
From: https://github.com/Homebrew/homebrew-cask/blob/HEAD/Casks/s/sleipnir.rb
==> Name
Sleipnir
==> Description
Web browser
==> Artifacts
Sleipnir.app (App)
==> Caveats
sleipnir is built for Intel macOS and so requires Rosetta 2 to be installed.
You can install Rosetta 2 with:
  softwareupdate --install-rosetta --agree-to-license
Note that it is very difficult to remove Rosetta 2 once it is installed.

From the Ruby formula:
$ HOMEBREW_NO_INSTALL_FROM_API=true brew info Sleipnir
==> sleipnir: 4.6.6
https://www.fenrir-inc.com/jp/sleipnir/
Not installed
From: https://github.com/Homebrew/homebrew-cask/blob/HEAD/Casks/s/sleipnir.rb
==> Name
Sleipnir
==> Description
Web browser
==> Artifacts
Sleipnir.app (App)

It seems like this issue is already covered by https://github.com/Homebrew/brew/issues/17288 so maybe we can close this one.

Yeah dynamic caveats are not supported, and even if it theoretically were we would still avoid them wherever possible for speed reasons.
The proper fix here would be for us to get rid of the Rosetta caveat in favour of a requires_rosetta true DSL (like auto_updates). Brew could then add the template text to the caveats dynamically.

Above point still applies but reading this again:

It's based on pre-computing the caveats when serializing formula for the API.

Why does it not differ in variations? That would be the quick fix here (though it would notably bloat the JSON a bit).

Above point still applies but reading this again:

It's based on pre-computing the caveats when serializing formula for the API.

Why does it not differ in variations? That would be the quick fix here (though could bloat the JSON a little bit).

My bad I wrote formula above but this is a cask-only issue.
https://github.com/Homebrew/brew/blob/916044581862c32fc2365e8e9ff0b1507a98925e/Library/Homebrew/cask/cask.rb#L458
What's happening here is that we only calculate variations if the cask has on system blocks which isn't the case here.
It looks like there are 1227 casks in the main cask repo that have requires_rosetta set in the caveats section so I'd be a bit worried about it bloating the JSON size but I haven't tested the size locally.

Making Rosetta a requirement for installing these casks on arm machines makes sense to me too assuming it's a straightforward change.

It looks like there is already a check that should ensure that this only gets displayed on arm machines.
https://github.com/Homebrew/brew/blob/29c22e0ab3fd9826cefd4b14c143266dea3bc391/Library/Homebrew/cask/dsl/caveats.rb#L143-L152
Could you run the following diagnostic commands locally and post the output here?
brew doctor

brew config

brew ruby -e 'p Homebrew::SimulateSystem.current_arch'

Thanks everyone for taking the time to check this. I'm guessing from the discusssion that I don't need to run the diagnostic commands anymore.
What you're discussing is absolutely out of my knowledge, but don't hesitate to tag me if in need of testers for this.
Thanks a lot again. (Hopefuly I didn't open a pandora's box :P)

@garvamel all good, thanks for the helpful issue!

It looks like there are 1227 casks in the main cask repo that have requires_rosetta set in the caveats section so I'd be a bit worried about it bloating the JSON size but I haven't tested the size locally.

We should prioritise correctness over JSON size. I think having something like caveats_rosetta: true or another minimal addition rather than duplicating the text would make the most sense here.

Formulae already have a similar notion of depends_on arch: :x86_64

Casks do as well, for example amd-power-gadget.
One solution might be to add :rosetta or :x86_64_or_rosetta as a possible value for depends_on arch: to mean "runs on x86_64 or on ARM with translation installed", which leaves depends_on arch: :x86_64 to mean "requires x86_64 hardware", as is the case above.

Are there any instances of depends_on arch: :x86_64 that won't work on Rosetta? If not, then we could probably just infer the Rosetta requirement from depends_on arch: :x86_64.

Probably only casks that tap directly into the hardware like amd-power-gadget, cuda-z, turbo-boost-switcher, and their ilk.

If depends_on arch: :x86_64 were to mean "runs on x86_64 or on ARM with Rosetta installed", then the Rosetta message would never be shown on x86_64 machines and always be shown on Rosetta-free ARM machines, as it should. The only risk there is with the few cases where the message implies that having Rosetta installed would allow the app to run, but in fact it still wouldn't run because it needs actual x86_64 hardware, rather than just machine translation.

If depends_on arch: :x86_64 were to mean "runs on x86_64 or on ARM with Rosetta installed", then the Rosetta message would never be shown on x86_64 machines and always be shown on Rosetta-free ARM machines, as it should. The only risk there is with the few cases where the message implies that having Rosetta installed would allow the app to run, but in fact it still wouldn't run because it needs actual x86_64 hardware, rather than just machine translation.

I think checking for specific hardware for apps is outside of the scope of what brew needs to check and warn for. This is because as a user you don't come to know of a formula or cask you are choosing to install, by searching in brew first instead of a search engine. If there are any doubts users brew info it to make sure. It's because of this "workflow" this that I was confused about the Rosetta warning, since the cask I was installing was supposed to be a clean install with no caveats.
The only edge case I can imagine is that one of these apps that need actual hardware were a dependency not obvious to the user.


Fix 'update_project_provisioning' to set PROVISIONING_PROFILE_SPECIFIER

Update the 'update_project_provisioning' action to set PROVISIONING_PROFILE_SPECIFIER (Name) rather than just the PROVISIONING_PROFILE (UUID)

Checklist

[x] I've run bundle exec rspec from the root directory to see all new and existing tests pass
[x] I've followed the fastlane code style and run bundle exec rubocop -a to ensure the code style is valid
[x] I've read the Contribution Guidelines
[x] I've updated the documentation if necessary.

Motivation and Context

Xcode 8 uses PROVISIONING_PROFILE_SPECIFIER in preference over PROVISIONING_PROFILE but this action is only setting PROVISIONING_PROFILE. This change adds better support for the Xcode 8+

https://github.com/fastlane/fastlane/issues/8973

Manually tested using both a small test app and a full scale production app.
I looked into writing a unit test for this change but the existing behavior (setting the PROVISIONING_PROFILE) is not unit tested and it was difficult to add a specific unit test for this change.
Description

Set PROVISIONING_PROFILE_SPECIFIER so that the correct provisioning profile is used when building with Xcode 8+.

I signed it!

I signed it!

I signed it!

Thanks for merging that so fast.
Would you mind also giving me your thoughts on the new action I suggested here. See: update_code_signing_options
If you think that would be a viable change I'm happy to put another PR together for it.


Use Occluders for performance improvement?

I think this is the only thing func_Godot does better than TBLoader, automagically creates occluder meshes to cull geometry. It would be useful to add if anyone would be interested (unfortunately I can't, I'm much too stupid).

Looks like this can be done with OccluderInstance3D. I haven't done anything with it yet, but potentially this might be pretty straightforward.


Bad z-buffer on visualization_msgs::Marker for CUBE_LIST or SPHERE_LIST

If you create a CUBE or SPHERE list as Marker type and do a:
int N = rand()%10000;
marker.points.resize(N);
marker.colors.resize(N);
for(int i = 0; i < N/2; i++){
 marker.points[i].x = (rand()%1000)/1000.0;
 marker.points[i].y = (rand()%1000)/1000.0;
 marker.points[i].z = (rand()%1000)/1000.0;
 marker.colors[i].r = (rand()%1000)/1000.0;
 marker.colors[i].g = (rand()%1000)/1000.0;
 marker.colors[i].b = (rand()%1000)/1000.0;
 marker.colors[i].a = 1;
}

Cubes (or Spheres) now has bad z-buffer. Farthest elements are drawn as first. See attached image, on the left part. (It is also noteworthy that entire voxel map is drawn behind rviz standard grid regardless of its real position that is surely on top of it: z > 0)

Briefly the problem seems to be that when you iterate on a lesser number of points/color lists size something strange happens in z-buffer. The same behavior when you add some overlapping CUBE or SPHERE (two or more elements with same position and same color).
Problem disappears if simple "push_back" is used to fill a void point/color list iteratively.
The bug was tested on several machines with several Video card on: RViz version 1.11.15 (indigo).

Thanks for the detailed report. I'll have a look as soon as possible. It would be helpful for when someone finds time to look into it to have a simple .bag file and .rviz file (or instructions on how to setup the issue in rviz) so that reproducing the problem is easier.

These issues may be related:

https://github.com/ros-visualization/rviz/issues/1006
https://github.com/ros-visualization/rviz/issues/925

I would like to add that LINE_LIST markers are problematic too:
https://www.youtube.com/watch?v=VVgRpsvlv0Y&feature=youtu.be
This is on Ubuntu 17.04, ROS Lunar.

@wjwwod here's also a bag and an rviz file. https://www.dropbox.com/sh/ne95xuq11xv1ess/AABIPKOZXnxgKOqeNvlEj4H6a?dl=0

A small update which could be an useful clue: I changed the following line to be setDepthWriteEnabled( true ) and this helped prevent the bug: https://github.com/ros-visualization/rviz/blob/kinetic-devel/src/rviz/ogre_helpers/billboard_line.cpp#L262

Thanks @ojura, I'll try to have a look at that soon.

Thanks! By the way, have you reproduced the other issue (>16384 points crash)? Should be fairly simple to reproduce, just fill up a marker with more than 16384 points. But if you need a bag, let me know.


UUID of primary service cannot be changed

Hello.
I'm using bleno in Raspberry Pi 2, and I connect it from android.
The UUID of primary service cannot be changed.
For example, I tried echo example of bleno, the UUID of primary service cannot be changed from 0000ec00-0000-1000-8000-00805f9b34fb (ec00), even if I changed the UUID part (ex. ec01).
The advertising UUID can be changed, so onScanResult() of BluetoothLeScanner returns the changed UUID (ec01).
But getServices() returns the former UUID (ec00).
This problem is not resolved even if I re-up the hci0, restart OS, re-plug the Bluetooth adapter (USB), reset from hcitool.
If a new Bluetooth adapter is plugged, the UUID can be changed for the first time only.
My Bluez version is 4.99, but same problem in 5.37. Node version is 5.5.0.
The Bluetooth adapter and the android phone can use Bluetooth 4.0.
Maybe I'm making an elementary mistake.
Thank you for responding.

@azukileaf make sure you've change both the advertising UUID and service UUID:

https://github.com/sandeepmistry/bleno/blob/master/examples/echo/main.js#L13
https://github.com/sandeepmistry/bleno/blob/master/examples/echo/main.js#L25


Introduce API_URL environment variable to set Saleor's domain

Problem
For Saleor to properly generate URLs pointing to itself, it needs to know the actual domain at which it is hosted. @patrys proposed the solution composed of:

Introducing the API_URL environment variable, which could be set, for example, to https://api.example.com/
Deprecating the shopDomainUpdate mutation

This should resolve the issues listed below and mitigate similar ones in the future:

https://github.com/saleor/saleor/issues/11267
https://github.com/saleor/saleor/issues/12288
https://github.com/saleor/saleor/discussions/11117

I'd consider PUBLIC_URL, which is more explicit, especially since API sounds a little like "external API" while its about "own service"
More context from Apps perspective:
Apps during installation require Saleor API URL to be provided (in the request header). The app is using URL and token to check several things related mainly to security. Saleor Cloud has this URL set properly, but self hosted is using some default/fallback value (localhost) instead public URL. This results in failed app installation

Agreed, PUBLIC_URL sounds better. It's how the rest of the world can reach Saleor. It's not necessarily the hostname of the machine running Saleor, as you can have load balancers and reverse proxies in front of the actual service.


install old version of docker

File: docker-for-mac/install.md
I need to install an old version of docker-- 1.13.
I've tried the following to no avail:

homebrew and cask don't have older versions of docker-- or none that i could find
i went to https://docs.docker.com/v1.13/docker-for-mac/install/ and it installed v1.18 !!!!! not what i wanted!
i google searched for 10 minutes trying to figure out how to download 1.13 instead of the latest and am still searching...

Did you check the information that's on the release notes? https://docs.docker.com/docker-for-mac/release-notes/#docker-for-mac-1130-2017-01-19
I'm closing this issue given that this is documented, but feel free to comment.


Change robot.txt on creativecommons.net or scrap the site

Creativecommons.net now redirects to creativecommons.org. however its content is still being indexed by search engines and available if you have a direct link like: https://creativecommons.net/r/paul/works/
I suggest removing the site and redirect all traffic to .org or change the robot.txt to stop it indexing the site.

Ensure redirects handle edge cases like https://creativecommons.net/join


[Feature] Modify file before filepond. e.g compress file before filepond add it

Is there an existing issue for this?

[X] I have searched the existing issues

Is your feature request related to a problem? Please describe.
Hi @rikschennink
First of all, thank you so much for the great package. Always appreciate the hard work that you have put into it.
Is there a way I can like before filepond starts adding a file I take the file and do something with it e.g compress it and give that compress File object to filepond etc.
Describe the solution you'd like
None
Describe alternatives you've considered
i tried the before add file hook but still filepond upload the original File object
beforeAddFile(item) {
      console.log('beforeAddFile', item);
      // before adding a file we compress the image using compressorjs by fengyuanchen library
      // and then we return the compressed image to filepond
      return new Promise((resolve, reject) => {
        // we check if the file is an image
        if(item.file.type.includes('image'))
        {
            new Compressor(item.file, {
            maxWidth: 2500,
            quality: 0.6,
            resize: 'cover',

            // The compression process is asynchronous,
            // which means you have to access the `result` in the `success` hook function.
            success(result) {
                // we return the compressed image to filepond
                console.log('compressed image', result);
                // item.file = result;
                resolve(result);

            },
            error(err) {
              console.log(err.message);
              return;
            },
          });
        } else {
          // if the file is not an image we resolve the promise with the file src
          resolve(item.file);
        }
      });

Thanks! This can be with a plugin, I'd advise to look in to a couple plugins and see how you can intercept a file (for example the file encode plugin). I believe the original image edit plugin

Thanks! This can be with a plugin, I'd advise to look in to a couple plugins and see how you can intercept a file (for example the file encode plugin). https://github.com/pqina/filepond-plugin-file-encode/blob/master/src/js/index.js

thanks for the reply.. i checked the code but I still can't wrap my around it.. can I use the addFilter in my filepond? what which filter I need to intercept it?

@rikschennink is there any guide on how to write filepond plugins?

@aneesdev at the moment there is not, your best bet is to look at existing plugins.

Hello. Did you find any solution to handle/compress image before adding to FilePond?

https://github.com/june07/filepond-plugin-zip