Program is interrupted at stage 1

I used a pair of large remote sensing images to generate DEM,
the version of ASP is 3.0.1, the error of "too many open files" has been fixed,
and the dense matching method is SGM.
When running in stage 1, the program is interrupted and an error is reported The information is as follows:
I would like to ask what might be the reason for this error and how to solve it? Thanks!
[2022-Feb-21 01:46:23] : CORRELATION FINISHED
Traceback (most recent call last):
File "/home/amax/anaconda3/envs/asp301/bin/parallel_stereo", line 891, in
spawn_to_nodes(step, settings, self_args)
File "/home/amax/anaconda3/envs/asp301/bin/parallel_stereo", line 491, in spawn_to_nodes
generic_run(cmd, opt.verbose)
File "/home/amax/anaconda3/envs/asp301/libexec/stereo_utils.py", line 96, in generic_run
raise Exception('Failed to run: ' + cmd_str)
Exception: Failed to run: parallel --will-cite --env ASP_DEPS_DIR --env PATH --env LD_LIBRARY_PATH
--env PYTHONHOME -u -P 4 -a /ssdnvme/block-image/tmp59dk3l80
"/home/amax/anaconda3/envs/asp301/bin/python /home/amax/anaconda3/envs/asp301/bin/parallel_stereo block-fwd.tif block-nad.tif try-1-sgm/1
--threads-singleprocess 8 --job-size-h 2048 --job-size-w 2048 -t rpc -s SGM-stereo.default --skip-low-res-disparity-comp --processes 4
--threads-multiprocess 8 --entry-point 1 --stop-point 2 --work-dir /ssdnvme/block-image --tile-id {}"

You are using the conda build, which is already six months old. Maybe you
can try the latest build. See the instructions here:
https://github.com/NeoGeographyToolkit/StereoPipeline/releases
If is also not clear what the error is. Are there more error messages? Does
it work with a different data set? Can you try to run plain stereo, so not
parallel stereo, and without  -s SGM-stereo.default. Maybe it will print a
fuller error message.
On Monday, February 21, 2022, 赵林博 @.***> wrote:

I used a pair of large remote sensing images to generate DEM,
the version of ASP is 3.0.1, the error of "too many open files" has been
fixed,
and the dense matching method is SGM.
When running in stage 1, the program is interrupted and an error is
reported The information is as follows:
I would like to ask what might be the reason for this error and how to
solve it? Thanks!
[2022-Feb-21 01:46:23] : CORRELATION FINISHED
Traceback (most recent call last):
File "/home/amax/anaconda3/envs/asp301/bin/parallel_stereo", line 891, in
spawn_to_nodes(step, settings, self_args)
File "/home/amax/anaconda3/envs/asp301/bin/parallel_stereo", line 491, in
spawn_to_nodes
generic_run(cmd, opt.verbose)
File "/home/amax/anaconda3/envs/asp301/libexec/stereo_utils.py", line 96,
in generic_run
raise Exception('Failed to run: ' + cmd_str)
Exception: Failed to run: parallel --will-cite --env ASP_DEPS_DIR --env
PATH --env LD_LIBRARY_PATH
--env PYTHONHOME -u -P 4 -a /ssdnvme/block-image/tmp59dk3l80
"/home/amax/anaconda3/envs/asp301/bin/python /home/amax/anaconda3/envs/asp301/bin/parallel_stereo
block-fwd.tif block-nad.tif try-1-sgm/1
--threads-singleprocess 8 --job-size-h 2048 --job-size-w 2048 -t rpc -s
SGM-stereo.default --skip-low-res-disparity-comp --processes 4
--threads-multiprocess 8 --entry-point 1 --stop-point 2 --work-dir
/ssdnvme/block-image --tile-id {}"
—
Reply to this email directly, view it on GitHub
https://github.com/NeoGeographyToolkit/StereoPipeline/issues/360, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKDU3E6TFSIGZ5JSZYUFYDU4JKENANCNFSM5O64WRMQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

My ASP version is a newer version installed in February 2022.
My two satellite images overlap an area of about 150 km in the east-west direction and about 60 km in the north-south direction.
The geometric model of the satellite image is the RPC model.
Compared with traditional satellite images, the amount of data is larger,
which also poses new challenges for 3D reconstruction of satellite images.
These days I have done further experiments with my images and the program can successfully generate DEM when I use a smaller range of images,
but when the range of images is larger, the program often fails,
which I currently believe may be due to poor accuracy of the RPC model or poor alignment of the epipolar image.
In response to some ideas I have, I would like to consult some features of ASP and hope to get your answer.
Some of my questions are shown below.
1.Whether epipolar images generated after the end of the Preprocessing stage generate new RPC parameters for use in subsequent triangulation stages.

About the use of the bundle_adjust tool

I used the asp_bm and asp_sgm algorithms to generate the DEM and found that the edge area of the dem is warped,
as "taco-shaped" described in Chapter 9 Bundle Adjustment in aspbook.
So I think I need to use the bundle_adjust tool to optimize the RPC parameters.
In the previous experiment I used the stereo_gui tool to visualize A.tif, B.tif and reference.tif,
manually selected GCP on the three images and saved for subsequent Bundle Adjustment use.
However, manual selection is time-consuming, and the manual selection of points is too little compared to the larger satellite image area.
I have also used the ipfind and ipmatch tools to get the .match file between the experimental image and the reference.tif,
but the iPmatch tool cannot get the common interest point of the three images
I would like to ask if it is possible to use the ipfind and ipmatch tools to match two experimental images to a reference.tif respectively.
Open the obtained two .match files with stereo_gui and generate .GCP files respectively, and use the bundle_adjust tool twice.
Finally, use bundle-adjust-prefix in parallel_stereo to pass in two camera adjustments

Cannot get enough matches to align images

When I use the whole image, the program often breaks, so I want to try to use the partial image,
and finally use the pc_merge tool to stitch the partial point cloud into a complete one.
I use stereo_gui to visualize two satellite images, hold ctrl and use the left mouse button to select the same area, then run parallel_stereo.
A large number of interest points are extracted in the preprocessing stage, but no matching points can be obtained.
But I used ipfind and ipmatch tools to process L-cropped.tif and R-cropoped.tif and got a lot of matching points.
I would like to ask if the matching points were eliminated because of the poor accuracy of my RPC model.
The error output from the terminal is as follows:
Matching forward
--->Obtained 14119 matches
Matching backward
--->Obtained 16405 matches
Matched  0 points
VW Error :unable to  match left and  right  images.

Meaning and use of corr-search parameter

Due to the larger coverage of satellite imagery,
image alignment may be less effective than traditional satellite imagery, which may also lead to erroneous results.
I opened L-cropped.tif and R-cropoped.tif in ENVI, looked for the same points and compared their pixel coordinates.
It is found that the same point is not in the same line in the two epipolar images and has a large difference.
Therefore I want to modify the "corr-search" parameter to optimize the generated dem.
When I was reading the latest aspbook, the second paragraph of Section 8.2.2 gave the definition of the four numbers for "corr-search".
But "horizontal maximum boundary" is defined twice, which I think is a bug.
Taking "corr-search -80 -2 20 2 " as an example, if I find that the vertical disparity of the same point on the two epipolar images is not well eliminated,
the error is about 20 pixels, and then 20 Expanded by 50% to 30.
At this time, the four parameters should be modified "-80 -30 20 30". I wonder if I understand this correctly.
What I understand is that "-80 -2 20 2" constitutes a rectangular search area,
The program searches the area for the same pixel.
Assuming that the alignment quality of the image is good most of the time, vertical parallax has been largely eliminated, so "maximun vertical" is set to a small number like 2 or 4.
But why is "horizontal disparity" not symmetrical like "vertical disparity"?

Scope of use and precautions for local_epipolar

I found that the latest asp adds a new method "local_epipolar" for image alignment.
I guess if the larger images don't align well, it would be a lot better to split the image and realign it.
but I noticed that "epipolar" only works with pinhole camera models, so I want to ask if "local_epipolar" supports it RPC model
I also tried the "local_epipolar" method, from the generated "GoodPixelMap.tif", the effect is poor, there is a large area of red.
The above is my question, maybe the description in some places is not very clear, please forgive me.
Looking forward to your answer, thank you very much

1.Whether epipolar images generated after the end of the Preprocessing
stage generate new RPC parameters for use in subsequent triangulation
stages.

The RPC parameters are not changed. But the "epipolar" aligned images are
used from the preprocessing stage all the way to the triangulation stage.
Until that stage it is only important to match the pixels, not the exact
cameras. Only at the triangulation stage, after all the pixels are matched
between left and right image, the software goes back to the orginal camera
pixels, before "epipolar" alignment, and then uses your cameras to
triangulate.

About the use of the bundle_adjust tool

I used the asp_bm and asp_sgm algorithms to generate the DEM and found
that the edge area of the dem is warped,
as "taco-shaped" described in Chapter 9 Bundle Adjustment in aspbook.
So I think I need to use the bundle_adjust tool to optimize the RPC
parameters.
In the previous experiment I used the stereo_gui tool to visualize A.tif,
B.tif and reference.tif,
manually selected GCP on the three images and saved for subsequent Bundle
Adjustment use.
However, manual selection is time-consuming, and the manual selection of
points is too little compared to the larger satellite image area.
I have also used the ipfind and ipmatch tools to get the .match file
between the experimental image and the reference.tif,
but the iPmatch tool cannot get the common interest point of the three
images
I would like to ask if it is possible to use the ipfind and ipmatch tools
to match two experimental images to a reference.tif respectively.
Open the obtained two .match files with stereo_gui and generate .GCP files
respectively, and use the bundle_adjust tool twice.

Finally, use bundle-adjust-prefix in parallel_stereo to pass in two camera

adjustments

There is no need to generate GCP files, that is indeed rather
time-consuming. The bundle-adjust-prefix by itself should be enough. Then
you can do alignment later, if you have a reference DEM. You can try to do:
bundle_adjust left.tif right.tif left.xml right.xml -o run_ba/run
stereo left.tif right.tif left.xml right.xml  run_stereo/run
--bundle-adjust-prefix run_ba/run
This will bundle-adjust the cameras, and use the result in stereo. It will
create match files for you, and run stereo. It won't optimize the RPC
parameter, rather, it will solve for a rotation + translation adjustment
for each camera.
Here it is assumed that the cameras are in .xml files. Otherwise if they
are in the images, one can run the commands without them.
You should check after this if the edge of the DEM is still warped. If it
is, let me know. Note that the DEM may be in the wrong place, that is
expected, because bundle_adjust may move both cameras. For now it is
important that the shape of the DEM is good, rather, where it is.
When you create a DEM, you can also compute the error image, such as
point2dem --errorimage run_stereo/run-PC.tif.
I will suggest examining the error image (you can use the "colormap" tool
to make a color image out of it) and see if there is any tilting in it,
such as errors in one area much bigger than another. Hopefully not, as then
that will mean bundle adjustment failed.
If you have a reference DEM, such as Copernicus 30 m DEM (which is freely
available), you can compare your obtained DEM to that one. The pc_align
tool can be uesd to align your DEM to a reference DEM.

Cannot get enough matches to align images

When I use the whole image, the program often breaks, so I want to try to
use the partial image,

That should not be the case.  Otherwise stereo may not work.
If it still breaks, you can tell me the error image. I think I pushed a fix
to stereo failing with the "too many open files" error. If you run into
that one, you may want to get the latest build.

and finally use the pc_merge tool to stitch the partial point cloud into a
complete one.
I use stereo_gui to visualize two satellite images, hold ctrl and use the
left mouse button to select the same area, then run parallel_stereo.
A large number of interest points are extracted in the preprocessing
stage, but no matching points can be obtained.
But I used ipfind and ipmatch tools to process L-cropped.tif and
R-cropoped.tif and got a lot of matching points.
I would like to ask if the matching points were eliminated because of the
poor accuracy of my RPC model.

The error output from the terminal is as follows:

Matching forward
--->Obtained 14119 matches
Matching backward
--->Obtained 16405 matches
Matched 0 points
VW Error :unable to match left and right images.

That is interesting. Maybe you have more luck after you use bundle
adjustment, as described earlier (without manual GCP or ipfind or ipmatch).
I hope you can send the full log here, I want to understand what happens
before and after that text above. Maybe indeed your RPC are not accurate.
Not sure.

Meaning and use of corr-search parameter

Due to the larger coverage of satellite imagery,
image alignment may be less effective than traditional satellite imagery,
which may also lead to erroneous results.
I opened L-cropped.tif and R-cropoped.tif in ENVI, looked for the same
points and compared their pixel coordinates.
It is found that the same point is not in the same line in the two
epipolar images and has a large difference.
Therefore I want to modify the "corr-search" parameter to optimize the
generated dem.
When I was reading the latest aspbook, the second paragraph of Section
8.2.2 gave the definition of the four numbers for "corr-search".
But "horizontal maximum boundary" is defined twice, which I think is a bug.
Taking "corr-search -80 -2 20 2 " as an example, if I find that the
vertical disparity of the same point on the two epipolar images is not well
eliminated,
the error is about 20 pixels, and then 20 Expanded by 50% to 30.
At this time, the four parameters should be modified "-80 -30 20 30". I
wonder if I understand this correctly.
What I understand is that "-80 -2 20 2" constitutes a rectangular search
area,
The program searches the area for the same pixel.
Assuming that the alignment quality of the image is good most of the time,
vertical parallax has been largely eliminated, so "maximun vertical" is set
to a small number like 2 or 4.

The L-cropped.tif and R-cropped.tif are before alignment, just with a crop.
You should open L.tif and R.tif and compare them. You can also open them in
stereo_gui, side by side, and if you click on a pixel, its coordinates will
be printed on screen.
Yes, corr-search is a rectangular search window, of the form minx, miny
widthx widthy.

But why is "horizontal disparity" not symmetrical like "vertical
disparity"?

After alignment, the horizontal disparity has most of the information,
while the vertical disparity should have little information, since,
hopefully the alignment worked.

Scope of use and precautions for local_epipolar

I found that the latest asp adds a new method "local_epipolar" for image
alignment.
I guess if the larger images don't align well, it would be a lot better to
split the image and realign it.
but I noticed that "epipolar" only works with pinhole camera models, so I
want to ask if "local_epipolar" supports it RPC model
I also tried the "local_epipolar" method, from the generated
"GoodPixelMap.tif", the effect is poor, there is a large area of red.

Normally local_epipolar should work. I used it myself recently and it
creates good results. But I don't know what your data is.
 I don't know if you can share pictures with your attempts.
That may help. It would be interesting to see a screenshot of stereo_gui
with left and right regions selected with Control-mouse, then the result of
running parallel_stereo from the menu and point2dem, so the DEM itself. It
would be interesting to also see the error image, obtained as above with
point2dem --errorimage, and also what you mean by "warped DEM".

The above is my question, maybe the description in some places is not very
clear, please forgive me.
Looking forward to your answer, thank you very much

It takes a lot of effort to get comfortable with the tools, and using them
with big images can be difficult. Your questions are very welcome.
I am not sure how clear my own answers are. You can try to see if anything
helps. Then let me know how it goes and we'll see what to do next.
Message ID: @.***

.com>

First of all, thank you very much for your answer and guidance last time. According to your suggestion, I did some further experiments to answer your question.
The current conjecture is that with the increase of the image width, the ability of the rational function model to fit the rigorous sensor model gradually decreases, which may lead to ASP processing errors and the generation of DEMs with poor quality and accuracy.
I checked the L.tif and R.tif generated by ASP, and extracted the second band of F.tif, the vertical disparity band. I found the vertical disparity to be large and not well removed.

Reply to last question

I do bundle adjustment on two images and output the error image. The red area has the largest triangulation error, about 1000 meters. It looks like bundle adjustment has failed

The warping of dem can be considered as the elevation or descent of the terrain. I displayed my terrain in 3D, and it can be found that the ground on the right side has risen up, but his real terrain is a plain, so there should be no big fluctuations.

Question about point feature matching

When the preprocessing stage is over, I use the GUI to load the .vwip and .match files to see the matching points. I found that although the image has an evenly distributed and sufficient number of interest points, there are no matching points in the upper right corner of the image.
When I read the asp documentation, I found the parameter "ip-triangulation-max-error","epipolar-threshold". Is it because the error is too large or the epipolar line is poor and the matching point in the upper right corner of the picture is deleted?
The images are similar enough in illumination. It can be found from the figure below that in the area with matching points, the number of matching points is sufficient and evenly distributed. This also shows that the similarity of the images is better. The image on the left is the .match situation loaded in the GUI, and the image on the right is an enlargement of the upper right corner of the image.

Weird GoodPixelMap.tif

When I use the local_epipolar alignment method and the dense matching algorithm is SGM, asp sometimes fails to found interest points, and then fails to match, and the terminal log is as follows:

_Using algorithm: 1
--> Reading global interest points.
* Loading match file: try-8-sgm-part3-local-epipolar/1-31200_43680_86_1427/31200_43680_86_1427-L-cropped__R-cropped.match
--> Writing: try-8-sgm-part3-local-epipolar/1-16640_43680_2080_1427/16640_43680_2080_1427-left-aligned-tile.tif
Left:  [] 100%Removed 0 interest point outliers based on their differences.
Left:  [] Complete!
--> Writing: try-8-sgm-part3-local-epipolar/1-10400_43680_2080_1427/10400_43680_2080_1427-right-aligned-tile.tif
Right:  [.......................................................] 0%	--> Individually normalize images
Removed 0 interest point outliers based on their differences.
Left:  [] Complete!
Min and max disparities: -98 161.
--> Writing: try-8-sgm-part3-local-epipolar/1-6240_43680_2080_1427/6240_43680_2080_1427-right-aligned-tile.tif
Right:  [.......................................................] 0%	--> Writing: try-8-sgm-part3-local-epipolar/1-18720_43680_2080_1427/18720_43680_2080_1427-left-aligned-tile.tif
Left:  [........................................................] 0%	--> Writing: try-8-sgm-part3-local-epipolar/1-8320_43680_2080_1427/8320_43680_2080_1427-left-aligned-tile.tif
Right:  [] Complete!
Left:  [] 100%IP-based search range: (Origin:<PHONE_NUMBER>,<PHONE_NUMBER>) width: 0 height: 0)
Looking for IP in left image...
Using 300 interest points per tile (1024^2 px).
Detecting IP
Min and max disparities: -87 352.
Removed 0 interest point outliers based on their differences.
Right:  [] Complete!
Left:  [**] Complete!
Removing IP near nodata with radius 4
Found interest points: 2072
Looking for IP in right image...
Using 300 interest points per tile (1024^2 px).
Detecting IP
Removing IP near nodata with radius 4
Found interest points: 0
--> Adjusted uniqueness threshold: 0.8
Matching: [] Complete!
    Matched points: 0

Computing the epipolar rectification using RANSAC with 1000 iterations and inlier threshold 2.
RANSAC Error.  Insufficient data._

But when choosing some small regions to run local_epipolar, sometimes the effect is not very good, as can be seen from GoodPixelMap.tif. The bottom left part of each tile succeeds, but everything else fails. The situation is shown in the figure below.

This situation of GoodPixelMap.tif also sometimes happens when my alignment method is affineepipolar, the periphery of the image can be matched successfully, but the center area fails, as shown in the figure below.

pc_align tool

I use the pc_align tool to align the dem results of relative bundle adjustment to the open source dem, but the generated results have three bands. Normally, DEMs have one band to store elevation values, but why are there three here? Below is the command I entered
pc_align   open-dem.tif  sgm-DEM.tif
-o align/1   --alignment-method similarity-least-squares     --save-transformed-source-point

processes and threads

When processing images, we need to set the processes and threads values used. But sometimes I'm not very clear about the difference between the two, so sometimes I just set the value of threads. Below is some of my understanding, please correct me if I am wrong.
I often use the SGM algorithm, so I set the ''corr-memory-limit-mb'' parameter. If my threads value is 10 and corr-memory-limit-mb is 4000, does it mean that the maximum memory usage is 104=40G.
But if I set processes to 2, threads to 10, and corr-memory-limit-mb to 4000, the maximum memory usage at this time is 210*4=80G.
When I use SGM algorithm, threads is set to 24, processes are not set. When I use the MGM algorithm, because it uses up to 8 threads, I set processes to 3 and threads to 8. I would like to ask if the processing speed in these two cases is roughly the same?

too many open files

The asp version I installed was the version released on March 8, 2022, but the problem of "too many open files" still occurred during the filtering phase. I noticed that this bug has been fixed in the latest version, but I have this problem again.
      Error:GdalIO: getdem/1-49600_36800_1600_1600-RD.tif:  Too many open files (code = 4)

The above are my thoughts and experimental results in the recent period of time, and I hope to continue to receive your guidance. Thank you very much~

The current conjecture is that with the increase of the image width, the ability of the rational function model to fit the rigorous sensor model gradually decreases, which may lead to ASP processing errors and the generation of DEMs with poor quality and accuracy.

That makes sense.

Question about point feature matching

When the preprocessing stage is over, I use the GUI to load the .vwip and .match files to see the matching points. I found that although the image has an evenly distributed and sufficient number of interest points, there are no matching points in the upper right corner of the image.
When I read the asp documentation, I found the parameter "ip-triangulation-max-error","epipolar-threshold". Is it because the error is too large or the epipolar line is poor and the matching point in the upper right corner of the picture is deleted?

Could be. You can try to run stereo with --skip-rough-homography and --no-datum, and see if that brings back those missing points. With these options the camera information is not used, which in your case appear not so accurate.

Weird GoodPixelMap.tif

pc_align tool

I use the pc_align tool to align the dem results of relative bundle adjustment to the open source dem, but the generated results have three bands. Normally, DEMs have one band to store elevation values, but why are there three here?

The result of pc_align is a point cloud. If you want a DEM, you should use point2dem to create it. (https://stereopipeline.readthedocs.io/en/latest/tools/pc_align.html#output-point-clouds-and-convergence-history)

processes and threads

When processing images, we need to set the processes and threads values used. But sometimes I'm not very clear about the difference between the two, so sometimes I just set the value of threads. Below is some of my understanding, please correct me if I am wrong.
I often use the SGM algorithm, so I set the ''corr-memory-limit-mb'' parameter. If my threads value is 10 and corr-memory-limit-mb is 4000, does it mean that the maximum memory usage is 10_4=40G. But if I set processes to 2, threads to 10, and corr-memory-limit-mb to 4000, the maximum memory usage at this time is 2_10*4=80G.

I think the memory usage does not depend on the number of threads, only on the number of processes. So, if you have 16 cores, it is better to use 2 processes with 8 threads each (8 threads is at most what MGM supports).

too many open files

The asp version I installed was the version released on March 8, 2022, but the problem of "too many open files" still occurred during the filtering phase. I noticed that this bug has been fixed in the latest version, but I have this problem again.
      Error:GdalIO: getdem/1-49600_36800_1600_1600-RD.tif:  Too many open files (code = 4)

The above are my thoughts and experimental results in the recent period of time, and I hope to continue to receive your guidance. Thank you very much~

The file above is a little tile. Your output directory should have an RD.tif file which is just a text file, which, inside, should have a list of such little tiles. I will suggest opening it and taking a look. It should start like this:
VRTDataset rasterXSize="688" rasterYSize="1204"
Now, I hope you can test if it has further down lines like this:
SourceProperties RasterXSize="688" RasterYSize="180"
This is what my fix put in. If you don't have such lines, that means your build is without the fix. If you have such text, maybe there's a bug somewhere which I'd have to figure out how to reproduce. Other users told me that after my fix the problem no longer shows up.

I made a sample example showing how to run our software with a dataset having RPC cameras. It is at https://github.com/NeoGeographyToolkit/StereoPipelineSolvedExamples/releases. Maybe experimenting with that one a little may offer some insights into what is going on with your own dataset.

I will also suggest when running point2dem to use the option --errorimage, and then to colorize and examine that image. Normally the errors should be no more than your image resolution in meters/pixel, or comparable. If they are big or the errors are concentrated in one area, that means your RPC model is wrong. Not sure if bundle adjustment can help.
A reference DEM can be found here, https://portal.opentopography.org/raster?opentopoID=OTSDEM.032021.4326.3, which can also help understand if the issue is your data, or how you use the tools.

Since ASP worked for you on the ASTER example I provided, and since your own dataset, as you say, has a large intersection error, that means that your camera model is indeed not accurate enough.

In the preprocessing stage, I added the --skip-rough-homography and --no-datum parameters, but the resultant matching points became worse, and the matching points only exist in some areas, as shown in the following figure.

I am running out of ideas. ASP really doesn't like something about your dataset. If you can share the data with me privately I could take a closer look (I know data sharing may not be possible for various reasons).

Yesterday I deleted the old version of ASP on computer A and installed version 20220308. After the installation is complete, I package the new environment and migrate it to computer B. Then I run it again on computer B, "too many open files" still appears, and there is still no "SourceProperties RasterXSize RasterYSize" in RD.tif.

Can you do parallel_stereo --version? When I do on mine, I get:
NASA Ames Stereo Pipeline 3.0.1-alpha
Build ID: c9f68ad7
Build date: 2022-03-10
Also, you can go to the directory where ASP is, and look up the file libexec/parallel_stereo (not bin/parallel_stereo). Open it, and  search for this text:
https://github.com/NeoGeographyToolkit/StereoPipeline/blob/master/src/asp/Tools/parallel_stereo#L340

Does this happen because of the influence of the old version of ASP on computer B? I am confused as to why the new version cannot be installed correctly.

I don't know.  I would not think so.

I also compared with the public DEM, the elevation errors in some areas range from tens of meters to hundreds of meters, which is obviously wrong. But without taking the elevation error into account, the plane position of my DEM doesn't deviate too much, and the terrain shape is the same as the public DEM. Is this because in the dense matching stage, the SGM algorithm performed better and found most of the correct matching points?

I guess that is good. It is still not good when that error is too high.

The triangulation error is described in the ASP documentation as "the distance between two rays, is not the true accuracy of the DEM. It is only another indirect measure of quality" I would like to know how ASP calculates the distance between two rays.

Two rays are traced, and if they don't intersect, the closest segment between the rays are found. The intersection error is the length of that segment. If the intersection error is high it means that your rays which are meant to intersect at the ground point are really off, which is a bad thing.

I learned the example of how to run asp with a dataset having RPC cameras. And I think this kind of picture is simple, intuitive and beautiful. Can I ask what software is used to open the DEM.

Try stereo_gui --hillshade run/run-DEM.tif. Also see https://stereopipeline.readthedocs.io/en/latest/next_steps.html#generating-color-hillshade-maps

Create my own camera model

Since the RFM model is no longer applicable to a large area, the original rigorous sensor model is a good choice. Is it possible to call my own camera model by modifying the code, for example -t mymodel. It's difficult for me to understand and modify the source code at the moment, but I want to know if the idea is possible?

Creating your own camera model is a giant pain. If your sensor is at least a frame camera, rather than linescan (also called pushbroom) it may be easier. But for linescan there's a good amount of code to write.

Personally, I really want to share data with you to overcome these doubts, but due to some regulations I can't do so, I'm very sorry, please forgive me.

parallel_stereo –version

When I enter "parallel_stereo --version" in the terminal, the displayed information is as follows
NASA Ames Stereo Pipeline 3.0.0
Build ID: 2d554034
Built against:
NASA Vision Workbench 3.0.0
Build ID: 90486189
USGS ISIS 5.0.1
Boost C++ Libraries 106800
GDAL 2.4.1 | 20190315
I find the installation path of ASP "/home/admin/anaconda3/envs/aspnew3/libexec"  and open libexec folder, but I can not find the parallel_stereo .

I also used coloramp to create some beautiful images before, but my legends don't have labels like yours, they only have colors, are these labels added by myself later? It would be better if the coloramp tool could also add the map scale directly.

Finally, I wish you a good mood

I do know data sharing is impractical for anything except public datasets.
Your build is old, from Jul 27, 2021. You may want to set carefully the PATH variable to point to the new build or even wipe or hide the old one.
ASP's tools cannot create labels, map scales, etc. The images in the doc I shared with you were likely done with some regular viewer. One can use QGIS for that which is free.
The bundle_adjust tool can reduce the intersection errors, or at least make them better distributed. (I don't recall if you used it before.) See https://stereopipeline.readthedocs.io/en/latest/tools/bundle_adjust.html. Then stereo needs to be called with --bundle-adjust-prefix as specified there, and one can do point2dem --errorimage on new PC cloud to see the intersection error.
Note that bundle_adjust creates its own match files, and you can examine if those are any better. Bundle adjustment also filters those matches for outliers. (Stereo later can also use the bundle_adjust match files if they are copied over to the stereo directory before starting.)

To add to the previous one:

I find the installation path of ASP "/home/admin/anaconda3/envs/aspnew3/libexec" and open libexec folder, but I can not find the parallel_stereo .

I see. You installed ASP with conda, so parallel_stereo should be in the bin directory. But in either case, you have the old build. The daily build is not shipped with conda. You will need to get the tarball from https://github.com/NeoGeographyToolkit/StereoPipeline/releases and extract it somewhere in your user dir and set the path to it.

I have used bundle_adjust tool before. Then use --errorimage to generate intersection-Err-Map. And compare it with intersection-Err-Map generated without bundle_adjust tool. The error in some areas is indeed smaller, but the effect is very limited, and the area to be optimized is very small. In addition, I also compared two DEMs, and the improvement in accuracy is also small.

For smaller areas of my satellite imagery data, I also selected images with a width of about 70km to generate a DEM. This time the triangulation error has become smaller. Compared with the public DEM, the elevation is similar and the accuracy is better.
For the case where the matching points could not be generated before, I think it is also due to the problem of RFM accuracy. As I said before, the parameter "ip-triangulation-max-error" eliminates those matching points, resulting in no matching points in some areas.
Therefore, for satellite images with large width, I think the accuracy of my RFM model is too poor, maybe the first thing I should do is to improve the precision of RPC parameters, and then pass it into asp to get the DEM.

My asp was installed using conda, I  found parallel_stereo in my bin folder, I opened it, but I didn't find "Bugfix for the "too many open files" problem." I think I understand you eans, daily build cannot be updated using conda.

But I'm not quite sure about the update method you're talking about.
Maybe you mean to unzip "StereoPipeline-3.0.1-alpha-2022-03-18-x86_64-Linux.tar.bz2" and put it in the ASP installation folder such as "/home/zhao/anaconda3/envs /asp". Replace the old files with the new unzipped installation package files.
You say "set the path to it". I don't know how to proceed with this, can you give me more detailed guidance?
Thank you~

You can unzip the ASP .tar.bz2 file anywhere in your home directory and
then run /home/your/location/asp/bin/parallel_stereo --version to confirm
the version, then you can continue using the full path to this and other
programs instead of your other installation.
On Sun, Mar 20, 2022, 8:02 AM 赵林博 @.***> wrote:

I have used bundle_adjust tool before. Then use --errorimage to
generate intersection-Err-Map. And compare it with intersection-Err-Map
generated without bundle_adjust tool. The error in some areas is indeed
smaller, but the effect is very limited, and the area to be optimized is
very small. In addition, I also compared two DEMs, and the improvement in
accuracy is also small.

For smaller areas of my satellite imagery data, I also selected images
with a width of about 70km to generate a DEM. This time the triangulation
error has become smaller. Compared with the public DEM, the elevation is
similar and the accuracy is better.
For the case where the matching points could not be generated before, I
think it is also due to the problem of RFM accuracy. As I said before, the
parameter "ip-triangulation-max-error" eliminates those matching points,
resulting in no matching points in some areas.
Therefore, for satellite images with large width, I think the accuracy of
my RFM model is too poor, maybe the first thing I should do is to improve
the precision of RPC parameters, and then pass it into asp to get the DEM.

My asp was installed using conda, I found parallel_stereo in my bin
folder, I opened it, but I didn't find "Bugfix for the "too many open
files" problem." I think I understand you eans, daily build cannot be
updated using conda.

But I'm not quite sure about the update method you're talking about.
Maybe you mean to unzip
"StereoPipeline-3.0.1-alpha-2022-03-18-x86_64-Linux.tar.bz2" and put it in
the ASP installation folder such as "/home/zhao/anaconda3/envs /asp".
Replace the old files with the new unzipped installation package files.
You say "set the path to it". I don't know how to proceed with this, can
you give me more detailed guidance?
Thank you~
—
Reply to this email directly, view it on GitHub
https://github.com/NeoGeographyToolkit/StereoPipeline/issues/360#issuecomment-1073269786,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKDU3HUO3EEMJA7UAELYLLVA44Y7ANCNFSM5O64WRMQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you commented.Message ID:
@.***>

To add to the previous message, if your solution is indeed that the RPC model is only locally accurate, but results in warping if used globally, one may try to process things in tiles. In each tile, which could be maybe 5 km by 5 km, with 1 km overlap among the tiles, one could replace their RPC model with ours for that tile, with the tool cam2rpc. Then, the left and right image crops for that tile with their local rpc camera files could be bundle adjusted individually (the param --camera-weight can be increased if the resulting DEM goes too far), see if the intersection error at least for a tile goes down, then run stereo for that tile, get a DEM, and align that one to a third party global DEM using our pc_align tool. Then, if those look consistent (which can be verified with the geodiff tool),  those could be merged with dem_mosaic.
This would be a very desperate measure though, which would take a lot of time, and I am not sure about the chances of success. The ideal approach is to use the exact model. We support that only for DigitalGlobe, PeruSat, and SPOT5, and as before, adding support for a new model is tricky.

Glad there is progress, and I understand that things take time and it is hard to figure out the best way of handling issues.
Not sure you did the alignment step (with our pc_align) before comparing your little DEMs to the pubic one and before mosaicking, but if you say things are acceptable, then maybe it is ok without alignment.

I used other tools to align the DEM with the public DEM. The following figure is the error distribution of the two DEMs, the ordinate is the number of pixels, and the abscissa is the elevation difference. It can be seen that the error is mainly concentrated in 16 meters.

I also used the pc_align tool to align the DEM, and only performed the "translation" transformation. The calculated transfrom matrix is the identity matrix. Can it be considered that the plane accuracy is good?
Considering that my dem resolution is 3 meters and the public dem resolution is 30m, I don't know if the error can be considered acceptable.
Thank you~

If the public DEM grid resolution is 30 m that does not mean the vertical
resolution is 30 m, it is likely a few factors better than that and may
depend on slope. You may want to consult that public DEM's info for more
details.
The mostly 16 m vertical accuracy seems to be a little high to me. But I am
not sure. Our 'geodiff' tool can be used to colorize the absolute
difference, and the colormap and DEMs can be overlaid and hillshaded in the
GUI to see where the errors are and if there are any obvious horizontal or
vertical shifts or other blunders. Or, if you have more than one image set
at 3 m resolution, one could try to create DEMs from those too and see how
they all compare.
On Tue, Mar 22, 2022 at 7:48 AM 赵林博 @.***> wrote:

I used other tools to align the DEM with the public DEM. The following
figure is the error distribution of the two DEMs, the ordinate is the
number of pixels, and the abscissa is the elevation difference. It can be
seen that the error is mainly concentrated in 16 meters.
[image: 配准后高程误差图]
https://user-images.githubusercontent.com/67795058/159509344-89a9dd71-9e3c-45a6-b7a9-b1501499f54e.jpg
I also used the pc_align tool to align the DEM, and only performed the
"translation" transformation. The calculated transfrom matrix is the
identity matrix. Can it be considered that the plane accuracy is good?
Considering that my dem resolution is 3 meters and the public dem
resolution is 30m, I don't know if the error can be considered acceptable.
Thank you~
—
Reply to this email directly, view it on GitHub
https://github.com/NeoGeographyToolkit/StereoPipeline/issues/360#issuecomment-1075277803,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKDU3FYIMEMNQIWHU72EV3VBHMT3ANCNFSM5O64WRMQ
.
You are receiving this because you commented.Message ID:
@.***>

The image coverage area I chose was a mountainous area with an average elevation of 1500 meters. First, use the geodiff tool to calculate the absolute elevation error between the DEM and the public DEM. The error value is limited to the range of 0-50 meters.The error distribution is as follows.

The error in the red area in the lower right corner of the image is large, which may be caused by the low fitting accuracy of RPC in this area.
The following are some local detail pictures, put the error map together with the public DEM to make an image swipe. It can be found that the line with larger error is consistent with the valley line. The upper half of the image is the elevation error map, and the lower half is the public DEM.

Considering that the test area is mountainous, and most of the lines with large errors are consistent with the valley lines, is this accuracy within an acceptable range?
Thank you very much~

To me this looks like there's a problem somewhere. The error is too big and it has a clear pattern.
Normally, if you have one low-resolution DEM and another high resolution DEM, and take their difference, the error is mostly  small except at corners of steeper areas.
But it is hard to say for sure just by looking at pictures. And if there's a big error, it is hard to say if it comes from lack of alignment or from poor cameras.
Note that point2dem can be used to make a DEM at 30 m too, not just at your native 3 m resolution, by changing the --tr parameter. Then one could compare datasets at same resolution. Then, as before, if you have another image pair, then another DEM can be created and used for comparison.
So I don't know what to say for sure.


[UI Plugin] support multiple publishes and pages in one plugin

Is your feature request related to a problem? Please describe.

Currently in our definition https://github.com/microsoft/BotFramework-Composer/blob/main/Composer/plugins/sample-ui-plugin/package.json, publish and page are keys which means we could only add one for each in one plugin.
Describe the solution you'd like

Change it to array so we could add multiple publishes and pages in one plugin
Describe alternatives you've considered

Additional context

@xieofxie what are you trying to do?

Hi @cwhitten I may implement a test page plugin as what I did before (without any schedule).
But this does not block me. I just found this design limitation and bring it up.

Changed to array now.


[lint] Fix shellcheck errors across the repo

This PR adds shellcheck to CI and fixes all the warnings it throws. Note that we are only checking issues with severity "warning" or higher.
Addresses #13343.

Sorry I should backtrack - I was under the impression that the shellcheck errors block CI as well, which does not seem to be the case. Let me make a separate fix for the Clang issues.

It is quite hard to install shellcheck on Google Airgapped environment. It needs Haskell compiler and libraries.. That means, whenever we decide to block the CI for shellcheck, Turn-around Time will be longer.

Are you saying that installing shellcheck is slow because it pulls in the Haskell platform? AFAICT, this is not true on my Debian machine.
% apt-cache show shellcheck | grep Depends
Depends: libc6 (>= 2.29), libffi8 (>= 3.4), libgmp10 (>= 2:6.2.1+dfsg)

I think the trick is that the Haskell compiler bakes the runtime into the binary, so the package doesn't even need to install the compiler.

I think @eunchan is talking about the fact that if the shellcheck doesn't run on the airgapped setup it will be a little more of a hassle to make changes on an airgapped setup and then test them on the non-airgapped setup as opposed to just having the necessary tools on both. Needless to say, we have solved this issue on our airgapped setup.


Human Brain Cell Atlas - revise cell_type annotation

It's come to our attention that many of the cell_type terms in this Collection are not as specific as they could be - for example, most of the "All neurons" Dataset is annotated as "neuron", but the supercluster_term field has some more specific terms
@dosumis may have some more specific terms as a starter
CC @saracarsanaro

CC @AvolaAmg
Also see https://github.com/brain-bican/human-brain-cell-atlas_v1_neurons/blob/main/curation_tables/annotation.tsv
However, it might be worth reviewing some of these to see if they align or clash with brain region.
Also note - the table includes Neuropeptide annotation which, combined with brain region => a potential source of more annotations to well-known cell types.


Avatars stop working after upgrade from 1.20.2 to 1.20.3

Description
I did an upgrade from 1.20.2  to 1.20.3 and i can't see any avatars now. (IDK if i can replicate it @ demo)
Logs:
...
2023/08/20 23:52:47 routers/web/base.go:80:2() [W] Unable to find avatars 72fc5c6965c587f2258f0019f40024e8
2023/08/20 23:52:47 routers/web/base.go:80:2() [W] Unable to find avatars d1efad72dc5b17dc66a46767c32fff40
2023/08/20 23:52:47 routers/web/base.go:80:2() [W] Unable to find avatars 953f53d361fbdf2326267ee0db866b06
2023/08/20 23:52:47 routers/web/base.go:80:2() [W] Unable to find avatars c33c49fef93f8d5dfc6b1cf987b0501b
2023/08/20 23:52:47 routers/web/base.go:80:2() [W] Unable to find avatars d41d8cd98f00b204e9800998ecf8427e
2023/08/20 23:52:47 routers/web/base.go:80:2() [W] Unable to find avatars c5a64653d6ccf4fe4b4f615ba09b67ca
2023/08/20 23:52:47 routers/web/base.go:80:2() [W] Unable to find avatars 9e1db24f83040e3c09de395179a8a427
2023/08/20 23:52:47 routers/web/base.go:80:2() [W] Unable to find avatars e1a865414e73ffdba833222a1e0397e6
2023/08/20 23:52:47 routers/web/base.go:80:2() [W] Unable to find avatars c5a64653d6ccf4fe4b4f615ba09b67ca
2023/08/20 23:52:57 routers/web/base.go:80:2() [W] Unable to find avatars c33c49fef93f8d5dfc6b1cf987b0501b
2023/08/20 23:52:57 routers/web/base.go:80:2() [W] Unable to find avatars c5a64653d6ccf4fe4b4f615ba09b67ca
2023/08/20 23:52:57 routers/web/base.go:80:2() [W] Unable to find avatars c5a64653d6ccf4fe4b4f615ba09b67ca
2023/08/20 23:52:57 routers/web/base.go:80:2() [W] Unable to find avatars 953f53d361fbdf2326267ee0db866b06
2023/08/20 23:52:57 routers/web/base.go:80:2() [W] Unable to find avatars e1a865414e73ffdba833222a1e0397e6
2023/08/20 23:52:57 routers/web/base.go:80:2() [W] Unable to find avatars d41d8cd98f00b204e9800998ecf8427e
2023/08/20 23:52:57 routers/web/base.go:80:2() [W] Unable to find avatars 72fc5c6965c587f2258f0019f40024e8
2023/08/20 23:52:57 routers/web/base.go:80:2() [W] Unable to find avatars 9e1db24f83040e3c09de395179a8a427
2023/08/20 23:52:57 routers/web/base.go:80:2() [W] Unable to find avatars d1efad72dc5b17dc66a46767c32fff40
2023/08/20 23:53:12 routers/web/base.go:80:2() [W] Unable to find avatars c5a64653d6ccf4fe4b4f615ba09b67ca
...

Gitea Version
1.20.3
Can you reproduce the bug on the Gitea demo site?
Yes
Log Gist
No response
Screenshots
No response
Git Version
?
Operating System
Debian 11
How are you running Gitea?
source code + systemd
Database
PostgreSQL

Could you please check if you are affected by https://blog.gitea.com/release-of-1.20.3/#-breaking-change and the PR fixing it?
We had to change the logic here.

Oh, I had relative path in AVATAR_UPLOAD_PATH, thank you @delvh


Remove Nth Node From End of List

Is your feature request related to a problem? Please describe.
Want to add Remove Nth Node From End of List Python Data structures Folder
Describe the solution you'd like
Problem Statement :
Given the head of a linked list, remove the nth node from the end of the list and return its head.

Example 1:
Input: head = [1,2,3,4,5], n = 2
Output: [1,2,3,5]
Example 2:
Input: head = [1], n = 1
Output: []
Example 3:
Input: head = [1,2], n = 1
Output: [1]
Describe alternatives you've considered
Leet Code question no. 19
Additional context
I would like to contribute and work on this issue as a participant of LGM SOC'21 will you please assign it to me
Programming language

[ ] C
[ ] C++
[ ] Java
[x] Python

@Ayush7614 @ravikr126 @Amit366
I am interested to work on this issue in C++.
Kindly assign this issue to me under LGM SOC 21

@Ayush7614 @ravikr126 @Saurabh4626 I'd like to work on this issue in C language under HCSSoC. Please assign me this issue.


Why response.download downloads file in default name even though I add the second parameter?

I used repl.it in mobile forking a project that served an HTML file using sendFile. I modified the call to download using .download adding the second parameter for a custom file name. But client browser is downloading the file as 'download' without any extension which seems to be default behavior.

We'll need more details to see that this is an express issue, not an incorrect API invocation.

I believe this is not express related as it was due to request/response caching which was returning 304 status that I clearly overlooked. I basically found 2 work-arounds : Adding 'no-cache' in response header, pressing Shift + F5 in browser to send in new request to express app. Sorry for the inconvenience.


WASM.SIMD: extra frontend checks

Fix build of when wasm is enabled, but wasm.simd is disabled.
Make sure we don't inadvertently allow Simd type to flow in the engine if it is disabled.

This change is

@arunetm Could you take a look please ?

lib/Runtime/Library/WebAssemblyModule.cpp, line 40 at r1 (raw file):

{
    m_alloc = HeapNew(ArenaAllocator, _u("WebAssemblyModule"), scriptContext->GetThreadContext()->GetPageAllocator(), Js::Throw::OutOfMemory);
    // the first elem is the number of Vars in front of I32; makes for a nicer offset computation

nit leading whitespace before 'the'

Comments from Reviewable

Review status: 0 of 8 files reviewed at latest revision, 2 unresolved discussions, some commit checks failed.

lib/WasmReader/WasmBinaryReader.cpp, line 23 at r1 (raw file):

    if (!Wasm::Simd::IsEnabled())
    {
        throw WasmCompilationException(_u("Wasm.Simd is not supported"));

Would it make sense to say "Wasm.Simd support is not enabled?" That way the user get a hint that it can be enabled for experiments.

Comments from Reviewable

Review status: 0 of 8 files reviewed at latest revision, 2 unresolved discussions.

lib/Runtime/Library/WebAssemblyModule.cpp, line 40 at r1 (raw file):
Previously, arunetm (Arun Etm) wrote…
nit leading whitespace before 'the'

The leading space is intentional, it is my coding style

Comments from Reviewable

Review status: 0 of 8 files reviewed at latest revision, 2 unresolved discussions, some commit checks failed.

Comments from Reviewable


Silence portman

Is it possible to silence portman? Its kind of annoying for ci runs to have random console.log statemens, i.e. in jest; Also it's generally not encouraged to ship code with console.log but some more standard logging system.
Is there some flag or something I can pass to silence it? In the code it seems like there is not option for it and I could not find anything.

Great feature request.
@nicklloyd and myself have also been discussing about this.
We have 2 types of "output".

CLI output
Postman output

Would you want control over what to silence?
And do you have a proposal for the CLI parameter: --silence?

Think --quiet is better. Bit friendlier 😅

Portman CLI outputs automatically set "postman variables" that could used by the developer.
The CLI output would safe him the trouble of opening Postman and reviewing each which variables have been provided by Postman.
example:
- Set variable for "post-push-broadcast" - use {{post-push-broadcast.id}} as variable for "response.id"

While the Postman output, is mainly for error / missing values output: handy for debugging
Example:
INFO - Unable to assign variable {{get-email-messages.messageId}}, as jsonData.value[0].messageId is undefined.

So my take on it with CI in mind is:
I want to be able to configure portman in such a way that it keeps quiet when everything is fine, but on error I get all the information needed to track the cause down.

@fellnerse If you look at your current CI output, could you share screenshots of what you want silenced?

We are not actually running it in the CI yet; but it's basically the same as you would do a normal run:
❯ npm run test:api:remote

><EMAIL_ADDRESS>test:api:remote
> portman --cliOptionsFile ./portman-cli-options.remote.json

======================================================================================================================================================================================================================
 Remote Url:        https://XXX.net/docs/openapi.yml
 Output Path:       ./crm.postman.json
 Portman CLI Config:    ./portman-cli-options.remote.json
 Portman Config:    portman/portman-config.crm.json
 Postman Config:    portman/postman-config.crm.json
 Environment:       .env
 Inject Tests:      true
 Bundle Tests:      true
 Run Newman:        true
 Newman Iteration Data: false
 Upload to Postman:     false
======================================================================================================================================================================================================================
  ✔ Conversion successful
======================================================================================================================================================================================================================
 Run Newman against:
======================================================================================================================================================================================================================
newman

Core API

❏ Contract Tests / customers
↳ Create Customer
  POST https://XXX/core/v1/customers [200 OK, 373B, 5.3s]
  ✓  [POST]::/customers - Status code is 2xx
  ✓  [POST]::/customers - Content-Type is application/json
  ✓  [POST]::/customers - Response has JSON Body
  ✓  [POST]::/customers - Schema is valid
  ✓  [POST]::/customers - Response header Location is present

→ Validate IBAN
  POST https://XXX/core/v1/iban/validation [200 OK, 445B, 404ms]

❏ customers
↳ Add Contract
  POST https://XXX/core/v1/customers/elit ea/contracts [404 Not Found, 443B, 63ms]

❏ Variation Tests / customers Variations
↳ Create Customer[All][required customerType]
  POST https://XXX/core/v1/customers [400 Bad Request, 411B, 63ms]
  ✓  [POST]::/customers - Response status code is 400
  ✓  [POST]::/customers - Content-Type is application/json
  ✓  [POST]::/customers - Response has JSON Body
  1. [POST]::/customers - Schema is valid
  2⠄ AssertionFailure in test-script

┌─────────────────────────┬────────────────────┬───────────────────┐
│                         │           executed │            failed │
├─────────────────────────┼────────────────────┼───────────────────┤
│              iterations │                  0 │                 0 │
├─────────────────────────┼────────────────────┼───────────────────┤
│                requests │                  4 │                 0 │
├─────────────────────────┼────────────────────┼───────────────────┤
│            test-scripts │                  2 │                 1 │
├─────────────────────────┼────────────────────┼───────────────────┤
│      prerequest-scripts │                  0 │                 0 │
├─────────────────────────┼────────────────────┼───────────────────┤
│              assertions │                  9 │                 1 │
├─────────────────────────┴────────────────────┴───────────────────┤
│ total run duration: 0ms                                          │
├──────────────────────────────────────────────────────────────────┤
│ total data received: 396B (approx)                               │
├──────────────────────────────────────────────────────────────────┤
│ average response time: 1465ms [min: 63ms, max: 5.3s, s.d.: 2.2s] │
└──────────────────────────────────────────────────────────────────┘

  #  failure                                   detail

 1.  AssertionError                            [POST]::/customers - Schema is valid
                                               expected data to satisfy schema but found following errors:
                                               data should have required property 'code', data.message should be string
                                               at assertion:3 in test-script
                                               inside "Variation Tests / customers Variations / Create Customer[All][required customerType]"

 2.  AssertionFailure                          [POST]::/customers - Schema is valid
                                               at test-script:38:17
                                               inside "Variation Tests / customers Variations / Create Customer[All][required customerType]"

======================================================================================================================================================================================================================
Newman run failed with:
[POST]::/customers - Schema is valid

on the output above, we can find 3 types of Output:

Portman conversion output

Newman execution output

Newman failure output

Which output do you want reduce?

So with the current available options I was only able to remove some of 2 and 3 (was not able to remove the table); In my opinion you should not use the console.log to print 1 because in the CI tests the only way to silence this is to overwrite console.log with a mock :D
Ideally nothing is printed on success; but on failure its detailed enough to reproduce locally.
That said I think 1 is easy for you because you can just have a logger of some sort; 2 and 3 are hard I guess because it is not really your domain, maybe I should open another issue at newman


Refactored Item and Block init

contains merge conflicts that must be resolved?

@Nick-77ab2 You're welcome to do so ;)
@tterrag1098 Tell me if you want to pull this, I'll merge it once then.

oh, i was just confused on why there's conflicts, and i'm not really good at programming at all as i'm pretty much just finishing VB.Net at my school xD and haven't taken java and other classes yet.

I think I'll be doing this differently when I attempt the modularization later down the road


sphinx: pad out the starting packet with random bytes

By padding out the starting packet with random bytes rather than leaving
the zeroes in tact, we patch a privacy leak that may reveal a lower
bound on the true route length to an adversarial exit node.
In order to reconcile this with our existing set of test vectors, we've
introduced a new abstraction that allows the caller to specify how they
want the starting bytes of the packet to be filled out. By default, all
callers will use the randPacketFiller, but may also pass in the
blankPacketFiller, if they packet construction to be deterministic.
Fixes #42.

Latest push should now pass the build. We now require the caller specify a packet filler. In lnd and everywhere else, the randomPacketFiller should be used. For our test vectors, we're able to pass in a blankPacketFiller which doesn't actually do anything a (noop).


ttdl: init at 3.5.0

Description of changes
TTDL (Terminal ToDo List) is a CLI tool to manage todo lists in todo.txt format
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 209066 run on aarch64-darwin 1

1 package built:

ttdl


Small fix to python agent-config option

Found the correct string in the code here
Currently the user will get:
RuntimeError: Unsupported OTLP protocol 'http' is configured

@open-telemetry/python-approvers PTAL!


Refine.

Small refactoring.

Thanks for the PR (was out for a couple of weeks). Generally looks good.


Drop functions to read sidecar JSONs

Closes #759.
Changes proposed in this pull request

Remove qsiprep.interfaces.bids.ReadSidecarJSON (this was actually unused, as the interface that was used in init_phdiff_wf was the one imported from niworkflows).
Remove qsiprep.interfaces.bids.get_metadata_for_nifti in favor of config.execution.layout.get_metadata.
Remove niworkflows's ReadSidecarJSON calls in favor of config.execution.layout.get_metadata.
Add phase_meta input to init_phdiff_wf. This actually matches the SDCFlows workflow a bit better as an added bonus.

I haven't removed read_nifti_sidecar, which is a big barrier to using the inheritance principle in DWI datasets.

The readthedocs error is very strange, I don't think this PR changes anything in extras_require

No, but I'm seeing the same failure in #766. I think it might be an issue with DiPy.

Codecov Report
Attention: Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.

Please upload report for BASE (master@87f0206). Learn more about missing BASE report.

Files
Patch %
Lines

qsiprep/workflows/fieldmap/base.py
0.00%
2 Missing :warning:

qsiprep/workflows/fieldmap/phdiff.py
0.00%
1 Missing :warning:

Additional details and impacted files
@@            Coverage Diff            @@
##             master     #760   +/-   ##
=========================================
  Coverage          ?   29.79%
=========================================
  Files             ?       97
  Lines             ?    14504
  Branches          ?     1870
=========================================
  Hits              ?     4322
  Misses            ?    10056
  Partials          ?      126

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

I think this will work, but I'm wondering if we should hold off on major refactors until we add a couple more CI tests (like for GRE fieldmaps). Wdyt?

I'm happy to wait on it.


chore(nextjs): Reuse ClerkProviderProps

Type of change

[ ] 🐛 Bug fix
[x] 🌟 New feature (nextjs/app-beta)
[ ] 🔨 Breaking change
[x] 📖 Refactoring / dependency upgrade / documentation
[ ] other:

Packages affected

[ ] @clerk/clerk-js
[ ] @clerk/clerk-react
[ ] @clerk/nextjs
[x] @clerk/nextjs/app-beta
[ ] @clerk/remix
[ ] @clerk/types
[ ] @clerk/themes
[ ] @clerk/localizations
[ ] @clerk/clerk-expo
[ ] @clerk/backend
[ ] @clerk/backend-core
[ ] @clerk/clerk-sdk-node
[ ] @clerk/edge
[ ] @clerk/shared
[ ] build/tooling/chore

Description

[x] npm test runs as expected.
[x] npm run build runs as expected.

~This PR try to re-use ClerkProviderProps from @clerk/clerk-react instead of narrower (specific) type to allow feature parity, for example the ability to specify localizations.~
This PR enable localization opt-in in nextjs app folder by re-using ClerkProviderProps from @clerk/clerk-react instead of narrower (specific) type. Only tested props are added to a NextAppBetaClerkProviderProps to ensure DX.

I think it's better to close this PR, since this has been superseded by #786.


⚠ AVS (kty) has degraded performance

In 50d3bd6, AVS (kty) (https://avus.com.br) experienced degraded performance:

HTTP code: 200
Response time: 8079 ms

Resolved: AVS (kty) performance has improved in 19ca921.


Not all options parameters are validated for CSVData

General Information:

Library version: 0.3.4

Describe the bug:
The options dict is not validated other than header. We need to validate all possible parameters (delimiter, etc.)
Tests also need to be added to address this

Submitted a PR for this issue: https://github.com/capitalone/DataProfiler/pull/89


Installation error

Trying all the different approaches to python wrapper installation but they all seem to boil down to this error:
incubator-datasketches-cpp/python/src/fi_wrapper.cpp:90:6: error: no matching member function for call to 'def'     .def("merge", &frequent_items_sketch<T>::merge)
along with a few other errors in pybind of candidate template ignored
What am I missing?

We added a moving merge method earlier but we don't have a CI pipeline to run the full set of tests and missed that.
Pybind is confused about which merge method to use, so need to specify one in that line. I believe that will fix it.  I dunno how fast I can get an approval on the patch but I think i can get a new branch in a few minutes.

Yup, a one-line fix. Change that line to:
.def("merge", (void (frequent_items_sketch<T>::*)(const frequent_items_sketch<T>&)) &frequent_items_sketch<T>::merge)
and it should work. Alternatively, check out the python_fi_fix branch. Or wait until the PR is approved, which is the slowest option.

Thanks for the speedy response and fix! I really imagined that I had something misconfigured on my end because... that's usually what the trouble is.
While I'm here I also wanted to point out that in the recursive clone the pybind repo is linked with ssh which can cause a sad permission error for anyone not set up for ssh. Https links for that dependency would make it simpler, since it's a public package anyway.

@steveatwork thanks for that suggestion.
I have a branch pybind_via_https that supposedly changes the include to https. It works if I look at the .gitmodules file, anyway. But I made it a clean branch -- which means the earlier fix isn't in it since that hasn't been merged yet (pending review). I think/hope it'll all be tidied up sometime Friday, US Pacific time.

@steveatwork The fix is in the master. Could you give it a go please? Thank you.

Tested and everything looks fine to me. Thanks!


Issue #1308 - Remove analytics link for Sei

Add optional carbonAnalytics config param and add it to the Ethereum config only
Remove "Analytics" from menu if carbonAnalytics URL is undefined

fix #1308

update on which URLs to remove

[ ] analytics
[ ] dune dashboard
both should be removed from the sei.carbondefi.xyz only.


Team member in BC Cloud Saas not behaving as Limited user

Describe the bug
A clear and concise description of what the bug is.
Naveksa is an ISV who has a Shopfloor solution. Both CfMD and Appsource approved.
As part of this application, we have a so-called client which run against the NAV/BC database using a limited user on the dynamics NAV version, and a team member user on the BC On-premise versions.
Everything works fine on the on premise version. But it does not work in the BC Cloud SaaS version (BC 15.2).
This means we are stuck with more file access denials. F.ex. do we get an update denial on trying to update the field "routing status" in table 5409 Production Order routing line
Where is the difference between a limited user on-premise and the team member user in the cloud SaaS version, if any?
It is our firm belief that this is not in accordance with the rules for the Team member user?
This problem is a big one for our customers who have been forced to run a temporary full user solution.
Regards
Bent Korsgaard
To Reproduce
Steps and to reproduce the behavior:

Go to '...'

AL Code to reproduce the issue
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
5. Versions:

AL Language:

Business Central:

Apparently a Team member license does not allow permission rights to any tables in manufacturing.
It looks like the restriction is on the objects belonging to the Premium Edition.
Can anyone confirm?

Hi! Could you please open a support case for this? This repository only handles issues related to the compiler and AL development tools.


Add KIC in Konnect with KGO example

Description
Updated KGO KIC guide to include Konnect
Testing instructions
Preview link: /gateway-operator/1.2.x/get-started/kic/create-gateway/
Checklist

[x] Review label added
[x] Conditional version tags added, if applicable.

I've made small changes to the guide (adapting to UI changes and such).
1 thing that is making me pause is the fact that we're adding a Konnect focused guide in  KIC focused docs subtree (not sure about nomenclature here).

While this might be slightly confusing to readers I'm not sure if we have a better option for this.
One thing I'd consider is to split the 2 pages into 3:

Konnect DataPlanes
Konnect KIC
On Prem KIC

That might be done separately and this guide is 👍 to me.


Adding translation for Brazilian Portuguese language (pt-BR)

Added translation for the pt-BR locale, simply copied the files from the en locale and translated them.
If anything is wrong or incorrect, please, contact me.
Thanks!

Thanks!


[ENH]: backoff parameter?

Sometimes, if you hit APIs for too many tiles, the API will ratelimit.
It would be nice if there were a rudimentary support for a wait time within bounds2* functions, so that you could specify a number of seconds to wait between tile queries & a max_retries amount.

This sounds good. Would there be any disadvantage? would defaults make it work in the same way as right now?

Somewhat related (as it can also help with rate limits), would be an automatic caching of downloaded tiles.

Related to #46

If I read it correctly, this was included in #35. Closing as implemented, but feel free to reopen otherwise.


FIX-2109 Add icon to open self-closing tags in QueryView

Fixes
This pull request fixes #2109
Description
Added a icon on lines with a self-closing tag. Clicking the icon expands the tag and places the cursor between the opening and closing tag.
Type of change

[ ] Bugfix
[ ] New feature (non-breaking change which adds functionality)
[x] Enhancement of existing functionality
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Impacted Areas in Application

[x] Frontend
[ ] API
[ ] WITSML
[ ] Other (please describe)

Checklist:
Communication

[ ] I have made corresponding changes to the documentation
[ ] PR affects application security

Code quality

[x] I have self-reviewed my code
[x] No new warnings are generated

Test coverage

[ ] New code is covered by passing tests

Further comments

I have tried to find the types instead of using any, but I could not find the correct types to use. The types that I thought would work were missing several methods that I am using. So for now we'll leave the types out. If the type libraries are updated with the methods later, we could use it at a later point.


Upgrade to version 0.1.3

In this upgrade, two features are introduced.

An abstraction of LLM models, AutoGenLLM class (which shares the same API as autogen OpenAIWrapper). User can subclass this class for their backend. AutoGenLLM is a subclass of AbstractModel which is a abstract class to support auto-refreshing model instances and serialization. As a result, AutoGenLLM can be deep copied and optimizers in trace.optimizers which is useful for reproducing experiments. Originally OpenAIWrapper is rlocked and cannot be serialized.
trace.bundle is refactored to support python asyncio. Now async functions can be bundled. The bundled function is still an async function. Running the function will return a coroutine. Await the coroutine to get the result as a MessageNode. Error handling is implemented too.

@microsoft-github-policy-service agree company="Microsoft"


[GESTURE] Gesture script is finished. [GESTURE] Python script is terminated. It will restart soon. Keyboard Interrupted

Starting MagicMirror: v2.7.1
Loading config ...
Loading module helpers ...
No helper found for module: alert.
Initializing new module helper ...
Module helper loaded: updatenotification
No helper found for module: clock.
Initializing new module helper ...
Module helper loaded: calendar
No helper found for module: weatherforecast.
Initializing new module helper ...
Module helper loaded: mvgmunich
Initializing new module helper ...
Module helper loaded: newsfeed
Initializing new module helper ...
Module helper loaded: MMM-NowPlayingOnSpotify
Initializing new module helper ...
Module helper loaded: MMM-GroveGestures
All module helpers loaded.
Starting server on port 8080 ...
Server started ...
Connecting socket for: updatenotification
Connecting socket for: calendar
Starting node helper for: calendar
Connecting socket for: mvgmunich
Connecting socket for: newsfeed
Starting module: newsfeed
Connecting socket for: MMM-NowPlayingOnSpotify
Connecting socket for: MMM-GroveGestures
Sockets connected & modules started ...
Launching application.
Create new news fetcher for url: http://rss.sueddeutsche.de/rss/Topthemen - Interval: 300000
(node:2602) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Refreshed access token because it has expired. Expired at: 21:39:26 now is: 21:39:27
Keyboard Interrupted
[GESTURE] Gesture script is finished.
Refreshed access token because it has expired. Expired at: 22:39:27 now is: 21:39:27
[GESTURE] Python script is terminated. It will restart soon.
Keyboard Interrupted
[GESTURE] Gesture script is finished.
[GESTURE] Python script is terminated. It will restart soon.
Keyboard Interrupted
[GESTURE] Gesture script is finished.
[GESTURE] Python script is terminated. It will restart soon.
Keyboard Interrupted
[GESTURE] Gesture script is finished.
[GESTURE] Python script is terminated. It will restart soon.
Keyboard Interrupted
[GESTURE] Gesture script is finished.
[GESTURE] Python script is terminated. It will restart soon.
Keyboard Interrupted
[GESTURE] Gesture script is finished.
[GESTURE] Python script is terminated. It will restart soon.
Keyboard Interrupted
[GESTURE] Gesture script is finished.
[GESTURE] Python script is terminated. It will restart soon.
Keyboard Interrupted
[GESTURE] Gesture script is finished.
[GESTURE] Python script is terminated. It will restart soon.
Keyboard Interrupted
[GESTURE] Gesture script is finished.
[GESTURE] Python script is terminated. It will restart soon.
Keyboard Interrupted
[GESTURE] Gesture script is finished.
[GESTURE] Python script is terminated. It will restart soon.
Keyboard Interrupted
[GESTURE] Gesture script is finished.
[GESTURE] Python script is terminated. It will restart soon.
Keyboard Interrupted
[GESTURE] Gesture script is finished.

Directly after start. Did I miss something? The module never shows up. The python test script does only work with SUDO (I can see the gestures in the terminal) sudo python gesture_print.py, but if I try python gesture_print.py I get the following error:
python gesture_print.py
Traceback (most recent call last):
  File "gesture_print.py", line 37, in <module>
    import grove_gesture_sensor
  File "/home/pi/MagicMirror/modules/MMM-GroveGestures/py/grove_gesture_sensor.py", line 26, in <module>
    bus = smbus.SMBus(1)
IOError: [Errno 13] Permission denied

It means you haven't installed MagicMirror or module with right permission. Usually ppl made a mistake to use sudo on git clone... or npm install. That makes your MM or module should be run with superuser permission.
I don't know which step you've made a mistake. Anyway remove module and try to reinstall carefully without sudo.

Thanks for you help. Did a clean install of MM. No sudo. MM starts up fine. Just no config created yet.
When I try to test the sensor without sudo, I got the same error. Any idea what went wrong?
><EMAIL_ADDRESS>start /home/pi/MagicMirror
> sh run-start.sh

Starting MagicMirror: v2.7.1
Loading config ...
WARNING! Could not find config file. Please create one. Starting with default configuration.
Loading module helpers ...
Initializing new module helper ...
Module helper loaded: updatenotification
No helper found for module: helloworld.
All module helpers loaded.
Starting server on port 8080 ...
Server started ...
Connecting socket for: updatenotification
Sockets connected & modules started ...
 You should commit this file.
added 1 package in 3.876s
pi@RPI3_Mirror:~/MagicMirror/modules/MMM-GroveGestures $ cd scripts
pi@RPI3_Mirror:~/MagicMirror/modules/MMM-GroveGestures/scripts $ chmod +x *.sh
pi@RPI3_Mirror:~/MagicMirror/modules/MMM-GroveGestures/scripts $ cd ~/MagicMirror/modules/MMM-GroveGestures/py
pi@RPI3_Mirror:~/MagicMirror/modules/MMM-GroveGestures/py $ cp grove_gesture_sensor.py.RPI grove_gesture_sensor.py
pi@RPI3_Mirror:~/MagicMirror/modules/MMM-GroveGestures/py $ cd ~/MagicMirror/modules/MMM-GroveGestures/py
pi@RPI3_Mirror:~/MagicMirror/modules/MMM-GroveGestures/py $ python gesture_print.py
Traceback (most recent call last):
  File "gesture_print.py", line 37, in <module>
    import grove_gesture_sensor
  File "/home/pi/MagicMirror/modules/MMM-GroveGestures/py/grove_gesture_sensor.py", line 26, in <module>
    bus = smbus.SMBus(1)
IOError: [Errno 13] Permission denied
pi@RPI3_Mirror:~/MagicMirror/modules/MMM-GroveGestures/py $

Go to Raspberry config program(raspi-config), Then check which I2C Bus is enabled. I assumed Bus 1 (I2C_1) be enabled. if you have to use other bus, modify python sources to other bus (SMBUS(n))

Strange... i made a clean start and now everything works. Thanks for help!


Problems with dynamic schemas and writing with optional fields

I am new to this package, and have a requirement for building dynamic schema's at runtime, but am having problems getting the whole solution to work. I have seen many other issues regarding this, but have been unable to piece together a full solution. Most the other issues are either unanswered, or don't have a full code snippet that works. As part of this, I am also needing Optional fields.
These are all the similar issues I have read and have not been able to figure it out.

https://github.com/parquet-go/parquet-go/issues/30
https://github.com/segmentio/parquet-go/issues/311
https://github.com/parquet-go/parquet-go/issues/25
https://github.com/segmentio/parquet-go/issues/392
https://github.com/segmentio/parquet-go/issues/310

I would like to create a schema at runtime. Both of these two options appear to work correctly for me when I create the schema.
schema := parquet.NewSchema("runtime_schema", parquet.Group{
	"id":  parquet.Optional(parquet.Int(32)),
	"age": parquet.Optional(parquet.Int(32)),
})

structFields := []reflect.StructField{}

tp := reflect.TypeOf(int32(0))

structFields = append(structFields, reflect.StructField{
	Name: strings.ToUpper("age"),
	Type: tp,
	Tag:  reflect.StructTag(fmt.Sprintf(`parquet:"%v,optional"`, "age")),
})
structFields = append(structFields, reflect.StructField{
	Name: strings.ToUpper("id"),
	Type: tp,
	Tag:  reflect.StructTag(fmt.Sprintf(`parquet:"%v,optional"`, "id")),
})

structType := reflect.StructOf(structFields)
structElem := reflect.New(structType)

schema := parquet.NewSchema("runtime_schema", parquet.SchemaOf(structElem.Interface()))

Both of those options create the same schema as expected:
message runtime_schema {
        optional int32 age (INT(32,true));
        optional int32 id (INT(32,true));
}

I am not sure on which option is better? I have seen both ways suggested in different issues: https://github.com/parquet-go/parquet-go/issues/30#issuecomment-1649189264, https://github.com/segmentio/parquet-go/issues/311#issuecomment-1219031236.
Now, the problem I am running in to, is using the the GenericWriter with this schema. I am unsure of what to put for the type parameter, so I have been using any. It compiles, and effectively "works", but nothing I have tries works with Optional fields.
When trying to use WriteRows with optional fields, in any way the schema is defined(typed struct, Group, or reflection), the optional fields are always null. This bug was noted here:https://github.com/parquet-go/parquet-go/issues/25#issuecomment-1642459872, although the comment made it seem like it was only on the deprecated Writer api, however it appears to also apply to the GenericWriter as well. Example of the bug with GenericWriter: https://github.com/parquet-go/parquet-go/issues/30#issuecomment-1656347829.
Now, I don't have a specific need for WriteRows right now, but it's the only way I could get it to compile, or not panic when writing. When trying to use Write instead, I run into panics. Usually of some reflection call. I'm sure I'm just making a simple mistake.
I am able to use Write like this:
func main() {
	schema := parquet.NewSchema("runtime_schema", parquet.Group{
		"id":  parquet.Optional(parquet.Int(32)),
		"age": parquet.Optional(parquet.Int(32)),
	})

	var f bytes.Buffer
	pw := parquet.NewGenericWriter[any](&f, schema)

	a := []any{
		map[string]int32{
			"id":  20,
			"age": 22,
		},
	}

	_, err := pw.Write(a)
	if err != nil {
		log.Fatal(err)
	}
	if err := pw.Close(); err != nil {
		log.Fatal(err)
	}

	reader := parquet.NewGenericReader[any](bytes.NewReader(f.Bytes()), schema)
	out := make([]parquet.Row, 1)
	if _, err := reader.ReadRows(out); err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%+v", out)
}

Prints: [[C:0 D:1 R:0 V:22 C:1 D:1 R:0 V:20]]
however this very simplified schema will not work for my use case. Because the schema isn't just of one type. So I would need to use a map[string]any, but it panics.
Example:
func main() {
	schema := parquet.NewSchema("runtime_schema", parquet.Group{
		"id":   parquet.Optional(parquet.Int(32)),
		"age":  parquet.Optional(parquet.Int(32)),
		"name": parquet.Optional(parquet.String()),
	})

	var f bytes.Buffer
	pw := parquet.NewGenericWriter[any](&f, schema)

	a := []any{
		map[string]any{
			"id":   20,
			"age":  22,
			"name": "bob",
		},
	}

	_, err := pw.Write(a)
	if err != nil {
		log.Fatal(err)
	}
	if err := pw.Close(); err != nil {
		log.Fatal(err)
	}

	reader := parquet.NewGenericReader[any](bytes.NewReader(f.Bytes()), schema)
	out := make([]parquet.Row, 1)
	if _, err := reader.ReadRows(out); err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%+v", out)
}

This panics with:
panic: cannot create parquet value of type INT32 from go value of type interface {}

goroutine 1 [running]:
github.com/parquet-go/parquet-go.makeValue(0x1, 0xc00018a930, {0x796100?, 0xc0000254a0?, 0xc000193a60?})
        /home/kason/go/pkg/mod/github.com/parquet-go/parquet-go@v0.17.0/value.go:339 +0xd89
github.com/parquet-go/parquet-go.deconstructFuncOfLeaf.func1({0xc00009c3c0, 0x3, 0xc0000a7410?}, {0xd0?, 0x71?, 0x41?}, {0x796100?, 0xc0000254a0?, 0x198?})
        /home/kason/go/pkg/mod/github.com/parquet-go/parquet-go@v0.17.0/row.go:538 +0x99
github.com/parquet-go/parquet-go.deconstructFuncOfOptional.func1({0xc00009c3c0?, 0x3?, 0x3?}, {0x0?, 0x0?, 0x0?}, {0x796100?, 0xc0000254a0?, 0xc000193bd0?})
        /home/kason/go/pkg/mod/github.com/parquet-go/parquet-go@v0.17.0/row.go:439 +0x103
github.com/parquet-go/parquet-go.deconstructFuncOfGroup.func1({0xc00009c3c0, 0x3, 0x3}, {0xc0?, 0x3b?, 0x19?}, {0x79d840?, 0xc0000a7410?, 0x7fb0e0?})
        /home/kason/go/pkg/mod/github.com/parquet-go/parquet-go@v0.17.0/row.go:515 +0x19c
github.com/parquet-go/parquet-go.(*Schema).deconstructValueToColumns(0xc000092300, {0xc00009c3c0, 0x3, 0x3}, {0x79d840?, 0xc0000a7410?, 0x7a8400?})
        /home/kason/go/pkg/mod/github.com/parquet-go/parquet-go@v0.17.0/schema.go:236 +0x114
github.com/parquet-go/parquet-go.(*Schema).Deconstruct(0xc000092300, {0x0, 0x0, 0x0}, {0x79d840?, 0xc0000a7410})
        /home/kason/go/pkg/mod/github.com/parquet-go/parquet-go@v0.17.0/schema.go:224 +0x1f4
github.com/parquet-go/parquet-go.(*Writer).Write(0xc0000925a0, {0x79d840, 0xc0000a7410})
        /home/kason/go/pkg/mod/github.com/parquet-go/parquet-go@v0.17.0/writer.go:379 +0x1d6
github.com/parquet-go/parquet-go.(*GenericWriter[...]).writeAny(0x0, {0xc000025490?, 0x1, 0x10})
        /home/kason/go/pkg/mod/github.com/parquet-go/parquet-go@v0.17.0/writer.go:233 +0x65
github.com/parquet-go/parquet-go.(*GenericWriter[...]).Write.func1(0xc000193e30?)
        /home/kason/go/pkg/mod/github.com/parquet-go/parquet-go@v0.17.0/writer.go:169 +0x62
github.com/parquet-go/parquet-go.(*writer).writeRows(0xc000188160, 0x1, 0xc000193e28)
        /home/kason/go/pkg/mod/github.com/parquet-go/parquet-go@v0.17.0/writer.go:993 +0xbb
github.com/parquet-go/parquet-go.(*GenericWriter[...]).Write(0xc0000a7410?, {0xc000025490?, 0x4?, 0xc0000686b0?})
        /home/kason/go/pkg/mod/github.com/parquet-go/parquet-go@v0.17.0/writer.go:168 +0x65
main.main()
        /home/kason/nuspire/nusiem-log-mover/cmd/parquet2/main.go:36 +0x4cd
exit status 2

Any help would be greatly appreciated! Thank you.

So I've been able to finally get the Write method to work correctly, even with Optional fields in the schema. Using the schema struct built with reflection, and then building up the struct instance allowed it work.
func main() {
	structFields := []reflect.StructField{}

	structFields = append(structFields, reflect.StructField{
		Name: strings.ToUpper("name"),
		Type: reflect.TypeOf(""),
		Tag:  reflect.StructTag(fmt.Sprintf(`parquet:"%v,optional"`, "name")),
	})
	structFields = append(structFields, reflect.StructField{
		Name: strings.ToUpper("id"),
		Type: reflect.TypeOf(int32(0)),
		Tag:  reflect.StructTag(fmt.Sprintf(`parquet:"%v,optional"`, "id")),
	})
	structFields = append(structFields, reflect.StructField{
		Name: strings.ToUpper("age"),
		Type: reflect.TypeOf(int32(0)),
		Tag:  reflect.StructTag(fmt.Sprintf(`parquet:"%v,optional"`, "age")),
	})

	structType := reflect.StructOf(structFields)
	structElem := reflect.New(structType)

	schema := parquet.NewSchema("runtime_schema", parquet.SchemaOf(structElem.Interface()))

	structValue := structElem.Elem()
	structValue.FieldByName("NAME").Set(reflect.ValueOf("bob"))
	structValue.FieldByName("ID").Set(reflect.ValueOf(int32(7)))
	structValue.FieldByName("AGE").Set(reflect.ValueOf(int32(30)))

	var f bytes.Buffer
	pw := parquet.NewGenericWriter[any](&f, schema)

	a := []any{
		structValue.Interface(),
	}

	_, err := pw.Write(a)
	if err != nil {
		log.Fatal(err)
	}
	if err := pw.Close(); err != nil {
		log.Fatal(err)
	}

	reader := parquet.NewGenericReader[any](bytes.NewReader(f.Bytes()), schema)
	out := make([]parquet.Row, 1)
	if _, err := reader.ReadRows(out); err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%+v", out)
}

This correctly prints: [[C:0 D:1 R:0 V:bob C:1 D:1 R:0 V:7 C:2 D:1 R:0 V:30]].
And then if I comment out the ID field for example, it prints that field as null as I would expect: [[C:0 D:1 R:0 V:bob C:1 D:0 R:0 V:<null> C:2 D:1 R:0 V:30]]
So I may have answered my own problems. I am just wondering if there is possibly a better/more efficient way to achieve this? In practice l will be ranging over column names gathered from elsewhere to build up this struct, so it'll be more involved then this simple example. Also the bug in WriteRows still exists, although I don't think I need it now.

There's another issue that currently stumps me.  That problem is that when writing 0-values to optional columns, they become null-values.
This can be reproduced by setting ID to 0 instead of 7 in the above example.  This also happens with other column types, e.g., float64.


Top-level-await, new WASM modules and async modules based on import await and import async

[x] adds top-level-await support
[x] adds import await support
[x] adds import async support
[x] adds exports await from support
[x] adds exports from async support
[x] adds new async wasm support based on that
[x] make wasm and top-level-await experiements

see example: https://github.com/webpack/webpack/tree/feature/top-level-await/examples/top-level-await
cc @littledan @jhnns @lukastaegert @TheLarkInn
based on https://gist.github.com/sokra/33f3db1b2714e9a6720f890842c47ae6

What kind of change does this PR introduce?
feature

Did you add tests for your changes?
yes

Does this PR introduce a breaking change?
yes, wasm is now opt-in by default

What needs to be documented once your changes are merged?
New experiments:

experiments.importAwait -> import/export await is enabled
experiments.importAsync -> import/export allows to use async module too
experiments.topLevelAwait -> allows to use await on top-level
experiments.asyncWebAssembly -> WebAssembly Modules are async modules
experiments.syncWebAssembly -> The webpack 4 WebAssembly Modules implementation (outdated)

So I added WASM-ESM-integration based on WASM being async modules according to the proposal here: https://github.com/WebAssembly/esm-integration/tree/master/proposals/esm-integration
Here are a few examples:

https://github.com/webpack/webpack/tree/feature/top-level-await/examples/wasm-simple
(only wasm without dependencies)
https://github.com/webpack/webpack/tree/feature/top-level-await/examples/wasm-complex
(wasm with dependencies, wasm depending on other async module with top-level-await)

There are a few functional changes compared to the old wasm implementation:

Importing memory, tables and i64-siguratures from JS is now allowed. Now there are no additional restrictions compared to the native WASM API. Actually this is used without change.
As circular dependencies are not allowed in async modules, circular dependencies including a WASM module are not supported. It was allowed before because trampolines are used for functions.
WASM need to be imported with import await or import(). This was not required before.
WASM can be in the initial chunk.
require() any async module gives a Promise, so does require-ing WASM now. This was the exports object before.

Implementation-wise:

No function trampolines are generated for wasm (Better runtime performance)
WASM does no longer be rewritten. (Better build performance at least in dev mode)
The new wasm implementation is default, but the old can still be enabled with type: "webassembly/experimental". But I would consider it deprecated and it will be removed in next major.

A note on wasm-bindgen: It will break for two reasons:

It doesn't use import await to import the wasm part. This can easily changed.
The wasm part imports the js part in a circular way. As circular dependencies are not supported, it need to be changed to generate 3 files instead. 1 WASM part and 2 JS parts.

Wow @sokra that's an awesome update! Thanks for the work.
From a Wasm viewpoint it's mostly lgtm, apart from:

i64-siguratures from JS is now allowed
This should still be disallowed, as it will throw when called from JS. A proposal exists already for that: https://github.com/WebAssembly/JS-BigInt-integration.

Do you think this can still land this week?

This should still be disallowed, as it will throw when called from JS. A proposal exists already for that: https://github.com/WebAssembly/JS-BigInt-integration.

Before this change we throw a build error when using i64-signatures, but as this will be soonish supported, I feel a build error is to much. Using an i64-signature might not even be an error, as somebody could have a browser with BigInt support.

Do you think this can still land this week?

Not sure. I actually wait for some feedback from @littledan about the import vs import await topic in the top-level-await discussion.
Unrelated from this: I want to move this and other things behind experiments flags, so you need to opt-in to use experimental features:
experiments: {
  mjs: true,
  topLevelAwait: true,
  asyncWasm: true,
  // ...
}

Opt in sounds important to do before landing this. I am excited to see the Webpack leading with experimentation here. However, the current TC39 proposal is based on import, not import await.

However, the current TC39 proposal is based on import, not import await.

I could also implement this, but it's a bit more involved since the async-or-not status of a module no longer only depends on itself, but on all dependencies too. So async status need to be inferred in a separate step after building the module graph. (That's also what I'm criticizing on the proposal. This is confusing for the developer too)

Before this change we throw a build error when using i64-signatures, but as this will be soonish supported, I feel a build error is to much. Using an i64-signature might not even be an error, as somebody could have a browser with BigInt support.

That's very true, I agree.

Unrelated from this: I want to move this and other things behind experiments flags

Would that allow Webpack to iterate faster, aka break code?

Would that allow Webpack to iterate faster, aka break code?

I would use relaxed SemVer for experiments so breaking changes in experiments could occur in minor versions.

Hey All,
We put together a small doc as an explainer for why the champions of Top-Level Await are not planning to move forward with the import await syntax for today. FWIW this is in no way implying that this PR shouldn't land, that tooling shouldn't experiment with new ideas, or that this proposal couldn't come to the committee in the future. I've included a link to the gist and the important bits copy pasted below.
Thank you for thanking the time to work on this alternative approach, it really helped us improve the algorithm for Top-Level Await.
https://gist.github.com/MylesBorins/ae97abab4a4144411c12240bb09dc7dd

How the current specification works

No more microticks unless there is top-level await

If no top-level await in graph then it is executed synchronously

Optmizes async execution

Only the modules that include a Top-Level Await are run asyncronously
Subsquent sub graphs are run synchronously
No micro ticks for remaining graph

Concerns with import await

Viral Syntax

no way to introduce async module into graph without import await to root
use of import await  defensively likely to happen

wasm modules will be async

will force introduction of wasm modules to be Semver-Major (and viral)

Removes ability for engine to optimize async execution

import await would force graph up to root to execute asynchronously
running import async modules synchronously would break developer contract

In Summary

It is my belief that import await is not a replacement for Top-Level Await
This does not mean that others, e.g. webpack, should not experiment with the API
this does not preclude import await from being presented to the committee as an additional proposal in the future

@MylesBorins Here is my answer to this: https://gist.github.com/sokra/34d2afe7555eb9a687e2c7566e9a9646
I'm currently in progress of implementing the original top-level-await proposal in this PR in addition to import await. You will be able to experiment with both.
I'm currently pushing towards import await because I favor it over import, but if the final spec will have a different shape I'm happy to use this instead. Both proposals solve a lot of problems with wasm modules we currently have. Having any of it is great.
One thing I have a bit trouble with is understanding the exact behavior of async cycles. I would love to hear some plaintext explanation about that.

To clarify, cycles of modules which have static import statements do not deadlock. The cycle is broken just as today with sync modules.

// entry.js
import x from "a";
import y from "b";
console.log(x);
console.log(y);

// a.js
import { hello, world } from "b";
export default hello + world;
await 1;
export function f() {
  return "hello";
}

// b.js
import value, { f } from "a";
export const hello = f();
await 1;
export const world = "world";
export default (() => { try { value } catch(e) { return e; } })();

So am I correct that this example prints:

helloworld
ReferenceError

So both proposals are now supported on opt-in:

experiments.importAwait -> import/export await is enabled
experiments.importAsync -> import/export allows to use async module too
experiments.topLevelAwait -> allows to use await on top-level
experiments.asyncWebAssembly -> WebAssembly Modules are async modules

So this landed in 5.0.0-alpha.15


Support Route timeout 、retry、websocket..etc

Description:
Support Route timeout 、retry、websocket..etc

@zhshw would appreciate it if you could raise an issue for each individual feature and add a little more information about why this feature is desired, closing this one for now due to the lack of info in it