Improve sparse pyrLK optical flow

Modified sparse pyrlk optical flow to allow input of an image pyramid, which thus allows caching of image pyramids on successive calls.

Can significantly reduce number of pyramid calculation calls.

Added uchar support for 1 and 4 channel images.

Very important for lowering memory usage in large images

Preliminary 3 channel support, unsigned short and int support.

The datatypes work however the accuracy tests fail.  So the cuda code exists for them but the c++ code wont let you get that far yet.

@jet47, since you are looking at it anyway, let me assign it to you

Thanks for reviewing this!  I believe I've addressed all of your comments except for conversion to [0,1] range.  It's not clear if you're referring to the 3 channel non texture fetch code path or the int32_t textures.
It would appear that cude doesn't support normalized float for int32_t, which is why I used a different filter method.  I suppose for this datatype I could use the non-texture code path, but it's a currently disabled datatype because pyrDown doesn't support int32_t.

I've got the following compilation errors on my machine:
pyrlk.cu(211): error: explicit type is missing ("int" assumed)

pyrlk.cu(211): error: no suitable conversion function from "int4" to "int" exists

pyrlk.cu(212): error: expression must have class type

pyrlk.cu(212): error: expression must have class type

pyrlk.cu(212): error: expression must have class type

pyrlk.cu(212): error: expression must have class type

pyrlk.cu(348): error: explicit type is missing ("int" assumed)

pyrlk.cu(348): error: no suitable conversion function from "int4" to "int" exists

pyrlk.cu(349): error: expression must have class type

pyrlk.cu(349): error: expression must have class type

pyrlk.cu(349): error: expression must have class type

pyrlk.cu(349): error: expression must have class type

pyrlk.cu(625): error: expected a ">"

pyrlk.cu(818): error: template parameter "PATCH_X" may not be redeclared in this scope

pyrlk.cu(818): error: template parameter "PATCH_Y" may not be redeclared in this scope

pyrlk.cu(818): error: template parameter "T" may not be redeclared in this scope

pyrlk.cu(627): error: expected a "," or ">"

pyrlk.cu(625): error: default argument not at end of parameter list

pyrlk.cu(838): error: type name is not allowed

pyrlk.cu(838): error: invalid combination of type specifiers

pyrlk.cu(862): warning: parsing restarts here after previous syntax error

pyrlk.cu(864): error: this declaration has no storage class or type specifier

pyrlk.cu(864): error: declaration is incompatible with "void cv::cuda::checkCudaError(cudaError_t, const char *, int, const char *)"
opencv/modules/core/include/opencv2/core/cuda/common.hpp(66): here

pyrlk.cu(864): error: expected a ")"

pyrlk.cu(866): error: expected a declaration

pyrlk.cu(868): error: expected a declaration

pyrlk.cu(871): error: a template argument list is not allowed in a declaration of a primary template

pyrlk.cu(903): error: a template argument list is not allowed in a declaration of a primary template

pyrlk.cu(936): error: a template argument list is not allowed in a declaration of a primary template

pyrlk.cu(1118): error: expression preceding parentheses of apparent call must have (pointer-to-) function type

pyrlk.cu(1118): error: identifier "c_winSize_x" is undefined

pyrlk.cu(1119): error: expression preceding parentheses of apparent call must have (pointer-to-) function type

pyrlk.cu(1119): error: identifier "c_winSize_y" is undefined

pyrlk.cu(1122): error: expression preceding parentheses of apparent call must have (pointer-to-) function type

pyrlk.cu(1122): error: identifier "c_halfWin_x" is undefined

pyrlk.cu(1123): error: expression preceding parentheses of apparent call must have (pointer-to-) function type

pyrlk.cu(1123): error: identifier "c_halfWin_y" is undefined

pyrlk.cu(1125): error: expression preceding parentheses of apparent call must have (pointer-to-) function type

pyrlk.cu(1125): error: identifier "c_iters" is undefined

pyrlk.cu(1199): error: expected a declaration

pyrlk.cu(1139): error: identifier "sparse_caller" is undefined

pyrlk.cu(1139): error: type name is not allowed

pyrlk.cu(1139): error: the global scope has no "call"

pyrlk.cu(1139): error: type name is not allowed

pyrlk.cu(1139): error: the global scope has no "call"

pyrlk.cu(1139): error: type name is not allowed

pyrlk.cu(1139): error: the global scope has no "call"

pyrlk.cu(1139): error: type name is not allowed

pyrlk.cu(1139): error: the global scope has no "call"

pyrlk.cu(1139): error: type name is not allowed

pyrlk.cu(1139): error: the global scope has no "call"

pyrlk.cu(1140): error: type name is not allowed

pyrlk.cu(1140): error: the global scope has no "call"

pyrlk.cu(1140): error: type name is not allowed

pyrlk.cu(1140): error: the global scope has no "call"

pyrlk.cu(1140): error: type name is not allowed

pyrlk.cu(1140): error: the global scope has no "call"

pyrlk.cu(1140): error: type name is not allowed

pyrlk.cu(1140): error: the global scope has no "call"

pyrlk.cu(1140): error: type name is not allowed

pyrlk.cu(1140): error: the global scope has no "call"

pyrlk.cu(1141): error: type name is not allowed

pyrlk.cu(1141): error: the global scope has no "call"

pyrlk.cu(1141): error: type name is not allowed

pyrlk.cu(1141): error: the global scope has no "call"

pyrlk.cu(1141): error: type name is not allowed

pyrlk.cu(1141): error: the global scope has no "call"

pyrlk.cu(1141): error: type name is not allowed

pyrlk.cu(1141): error: the global scope has no "call"

pyrlk.cu(1141): error: type name is not allowed

pyrlk.cu(1141): error: the global scope has no "call"

pyrlk.cu(1142): error: type name is not allowed

pyrlk.cu(1142): error: the global scope has no "call"

pyrlk.cu(1142): error: type name is not allowed

pyrlk.cu(1142): error: the global scope has no "call"

pyrlk.cu(1142): error: type name is not allowed

pyrlk.cu(1142): error: the global scope has no "call"

pyrlk.cu(1142): error: type name is not allowed

pyrlk.cu(1142): error: the global scope has no "call"

pyrlk.cu(1142): error: type name is not allowed

pyrlk.cu(1142): error: the global scope has no "call"

pyrlk.cu(1143): error: type name is not allowed

pyrlk.cu(1143): error: the global scope has no "call"

pyrlk.cu(1143): error: type name is not allowed

pyrlk.cu(1143): error: the global scope has no "call"

pyrlk.cu(1143): error: type name is not allowed

pyrlk.cu(1143): error: the global scope has no "call"

pyrlk.cu(1143): error: type name is not allowed

pyrlk.cu(1143): error: the global scope has no "call"

pyrlk.cu(1143): error: type name is not allowed

pyrlk.cu(1143): error: the global scope has no "call"

pyrlk.cu(1139): error: a value of type "int" cannot be used to initialize an entity of type "const func_t"

pyrlk.cu(1139): error: a value of type "int" cannot be used to initialize an entity of type "const func_t"

pyrlk.cu(1139): error: too many initializer values

pyrlk.cu(1140): error: a value of type "int" cannot be used to initialize an entity of type "const func_t"

pyrlk.cu(1140): error: a value of type "int" cannot be used to initialize an entity of type "const func_t"

pyrlk.cu(1140): error: too many initializer values

pyrlk.cu(1141): error: a value of type "int" cannot be used to initialize an entity of type "const func_t"

pyrlk.cu(1141): error: a value of type "int" cannot be used to initialize an entity of type "const func_t"

pyrlk.cu(1141): error: too many initializer values

pyrlk.cu(1142): error: a value of type "int" cannot be used to initialize an entity of type "const func_t"

Could you tell me a bit more about your machine?
I removed the dead code and C++11 features.  But a few things don't add up, like you have an error on line 864:
cudaSafeCall(cudaGetLastError());
Which I haven't changed.
Also you have errors inside of the dense kernel, which I haven't touched either.

I was able to build it on my machine (Ubuntu 12.04, GCC 4.6, CUDA 7.0) after some modifications. See
pyrlk.patch.txt.
But now I have sanity test failures. To reproduce them download opencv_extra repository from https://github.com/Itseez/opencv_extra and run the following command:
$ export OPENCV_TEST_DATA_PATH=<path to opencv_extra>/testdata
$ ./opencv_perf_cudaoptflow --perf_min_samples=1 --perf_force_samples=1 --perf_verify_sanity

Test log:
[----------] 8 tests from ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse
[ RUN      ] ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse.PyrLKOpticalFlowSparse/0
/home/vlad/opencv3.0/modules/ts/src/ts_perf.cpp:359: Failure
The difference between expect_min and actual_min is 0.12185859680175781, which exceeds eps, where
expect_min evaluates to 0.85204124450683594,
actual_min evaluates to 0.97389984130859375, and
eps evaluates to 2.2204460492503131e-16.
Argument "gpu_nextPts" has unexpected minimal value

params    = (("gpu/opticalflow/frame0.png", "gpu/opticalflow/frame1.png"), false, 8000, 21, 1, 1)
termination reason:  reached maximum number of iterations
bytesIn   =          0
bytesOut  =          0
samples   =          1
outliers  =          0
frequency =<PHONE_NUMBER>
min       =   47192916 = 47.19ms
median    =   47192916 = 47.19ms
gmean     =   47192916 = 47.19ms
gstddev   = 0.00000000 = 0.00ms for 97% dispersion interval
mean      =   47192916 = 47.19ms
stddev    =          0 = 0.00ms
[  FAILED  ] ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse.PyrLKOpticalFlowSparse/0, where GetParam() = (("gpu/opticalflow/frame0.png", "gpu/opticalflow/frame1.png"), false, 8000, 21, 1, 1) (93 ms)
[ RUN      ] ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse.PyrLKOpticalFlowSparse/1
/home/vlad/opencv3.0/modules/ts/src/ts_perf.cpp:359: Failure
The difference between expect_min and actual_min is 0.064218521118164062, which exceeds eps, where
expect_min evaluates to 0.92028141021728516,
actual_min evaluates to 0.98449993133544922, and
eps evaluates to 2.2204460492503131e-16.
Argument "gpu_nextPts" has unexpected minimal value

params    = (("gpu/opticalflow/frame0.png", "gpu/opticalflow/frame1.png"), false, 8000, 21, 1, 30)
termination reason:  reached maximum number of iterations
bytesIn   =          0
bytesOut  =          0
samples   =          1
outliers  =          0
frequency =<PHONE_NUMBER>
min       =   29338617 = 29.34ms
median    =   29338617 = 29.34ms
gmean     =   29338617 = 29.34ms
gstddev   = 0.00000000 = 0.00ms for 97% dispersion interval
mean      =   29338617 = 29.34ms
stddev    =          0 = 0.00ms
[  FAILED  ] ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse.PyrLKOpticalFlowSparse/1, where GetParam() = (("gpu/opticalflow/frame0.png", "gpu/opticalflow/frame1.png"), false, 8000, 21, 1, 30) (64 ms)
[ RUN      ] ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse.PyrLKOpticalFlowSparse/2
/home/vlad/opencv3.0/modules/ts/src/ts_perf.cpp:359: Failure
The difference between expect_min and actual_min is 0.096344947814941406, which exceeds eps, where
expect_min evaluates to 0.89337730407714844,
actual_min evaluates to 0.98972225189208984, and
eps evaluates to 2.2204460492503131e-16.
Argument "gpu_nextPts" has unexpected minimal value

params    = (("gpu/opticalflow/frame0.png", "gpu/opticalflow/frame1.png"), false, 8000, 21, 3, 1)
termination reason:  reached maximum number of iterations
bytesIn   =          0
bytesOut  =          0
samples   =          1
outliers  =          0
frequency =<PHONE_NUMBER>
min       =   56599094 = 56.60ms
median    =   56599094 = 56.60ms
gmean     =   56599094 = 56.60ms
gstddev   = 0.00000000 = 0.00ms for 97% dispersion interval
mean      =   56599094 = 56.60ms
stddev    =          0 = 0.00ms
[  FAILED  ] ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse.PyrLKOpticalFlowSparse/2, where GetParam() = (("gpu/opticalflow/frame0.png", "gpu/opticalflow/frame1.png"), false, 8000, 21, 3, 1) (85 ms)
[ RUN      ] ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse.PyrLKOpticalFlowSparse/3
/home/vlad/opencv3.0/modules/ts/src/ts_perf.cpp:359: Failure
The difference between expect_min and actual_min is 0.065741539001464844, which exceeds eps, where
expect_min evaluates to 0.91993236541748047,
actual_min evaluates to 0.98567390441894531, and
eps evaluates to 2.2204460492503131e-16.
Argument "gpu_nextPts" has unexpected minimal value

params    = (("gpu/opticalflow/frame0.png", "gpu/opticalflow/frame1.png"), false, 8000, 21, 3, 30)
termination reason:  reached maximum number of iterations
bytesIn   =          0
bytesOut  =          0
samples   =          1
outliers  =          0
frequency =<PHONE_NUMBER>
min       =   76048678 = 76.05ms
median    =   76048678 = 76.05ms
gmean     =   76048678 = 76.05ms
gstddev   = 0.00000000 = 0.00ms for 97% dispersion interval
mean      =   76048678 = 76.05ms
stddev    =          0 = 0.00ms
[  FAILED  ] ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse.PyrLKOpticalFlowSparse/3, where GetParam() = (("gpu/opticalflow/frame0.png", "gpu/opticalflow/frame1.png"), false, 8000, 21, 3, 30) (106 ms)
[ RUN      ] ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse.PyrLKOpticalFlowSparse/4
[ PERFSTAT ]    (samples = 1, mean = 1.89, median = 1.89, stddev = 0.00 (0.0%))
[ VALUE    ]    (("gpu/opticalflow/frame0.png", "gpu/opticalflow/frame1.png"), true, 8000, 21, 1, 1)
[       OK ] ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse.PyrLKOpticalFlowSparse/4 (30 ms)
[ RUN      ] ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse.PyrLKOpticalFlowSparse/5
/home/vlad/opencv3.0/modules/ts/src/ts_perf.cpp:359: Failure
The difference between expect_min and actual_min is 3.4332275390625e-05, which exceeds eps, where
expect_min evaluates to 0.91812229156494141,
actual_min evaluates to 0.91815662384033203, and
eps evaluates to 2.2204460492503131e-16.
Argument "gpu_nextPts" has unexpected minimal value

params    = (("gpu/opticalflow/frame0.png", "gpu/opticalflow/frame1.png"), true, 8000, 21, 1, 30)
termination reason:  reached maximum number of iterations
bytesIn   =          0
bytesOut  =          0
samples   =          1
outliers  =          0
frequency =<PHONE_NUMBER>
min       =    2928387 = 2.93ms
median    =    2928387 = 2.93ms
gmean     =    2928387 = 2.93ms
gstddev   = 0.00000000 = 0.00ms for 97% dispersion interval
mean      =    2928387 = 2.93ms
stddev    =          0 = 0.00ms
[  FAILED  ] ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse.PyrLKOpticalFlowSparse/5, where GetParam() = (("gpu/opticalflow/frame0.png", "gpu/opticalflow/frame1.png"), true, 8000, 21, 1, 30) (30 ms)
[ RUN      ] ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse.PyrLKOpticalFlowSparse/6
/home/vlad/opencv3.0/modules/ts/src/ts_perf.cpp:359: Failure
The difference between expect_min and actual_min is 0.89392757415771484, which exceeds eps, where
expect_min evaluates to -0.0052967071533203125,
actual_min evaluates to 0.88863086700439453, and
eps evaluates to 2.2204460492503131e-16.
Argument "gpu_nextPts" has unexpected minimal value

params    = (("gpu/opticalflow/frame0.png", "gpu/opticalflow/frame1.png"), true, 8000, 21, 3, 1)
termination reason:  reached maximum number of iterations
bytesIn   =          0
bytesOut  =          0
samples   =          1
outliers  =          0
frequency =<PHONE_NUMBER>
min       =    4825479 = 4.83ms
median    =    4825479 = 4.83ms
gmean     =    4825479 = 4.83ms
gstddev   = 0.00000000 = 0.00ms for 97% dispersion interval
mean      =    4825479 = 4.83ms
stddev    =          0 = 0.00ms
[  FAILED  ] ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse.PyrLKOpticalFlowSparse/6, where GetParam() = (("gpu/opticalflow/frame0.png", "gpu/opticalflow/frame1.png"), true, 8000, 21, 3, 1) (34 ms)
[ RUN      ] ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse.PyrLKOpticalFlowSparse/7
/home/vlad/opencv3.0/modules/ts/src/ts_perf.cpp:359: Failure
The difference between expect_min and actual_min is 0.79467868804931641, which exceeds eps, where
expect_min evaluates to 0.125,
actual_min evaluates to 0.91967868804931641, and
eps evaluates to 2.2204460492503131e-16.
Argument "gpu_nextPts" has unexpected minimal value

params    = (("gpu/opticalflow/frame0.png", "gpu/opticalflow/frame1.png"), true, 8000, 21, 3, 30)
termination reason:  reached maximum number of iterations
bytesIn   =          0
bytesOut  =          0
samples   =          1
outliers  =          0
frequency =<PHONE_NUMBER>
min       =    8783498 = 8.78ms
median    =    8783498 = 8.78ms
gmean     =    8783498 = 8.78ms
gstddev   = 0.00000000 = 0.00ms for 97% dispersion interval
mean      =    8783498 = 8.78ms
stddev    =          0 = 0.00ms
[  FAILED  ] ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse.PyrLKOpticalFlowSparse/7, where GetParam() = (("gpu/opticalflow/frame0.png", "gpu/opticalflow/frame1.png"), true, 8000, 21, 3, 30) (36 ms)
[----------] 8 tests from ImagePair_Gray_NPts_WinSz_Levels_Iters_PyrLKOpticalFlowSparse (478 ms total)

I've integrated the patch that you created. It seems to work fine so I'll push that soon.
If I understand the sanity check code correctly, it's basically doing some statistics on the tracked points as detected by optical flow.  If so, is it reasonable to assume there might be a bit of a difference in values due to the different texture fetch methods?  IE cudaReadModeElement vs cudaReadModeNormalizedFloat.  If I change the sanity test to pass in a float image and a float BGRA image then the test passes.  Would that be sufficient?
How were the expected values determined in the performance test?  Was it generated by the cpu or gpu optical flow code?  Was it verified to be correct?  I ask because one of the expected min values is -0.0052967071533203125 which is outside of the image.  Should I generate new truth data and more tests for the different datatypes?

Would that be sufficient?

Yes, your solution is OK.

How were the expected values determined in the performance test? Was it generated by the cpu or gpu optical flow code? Was it verified to be correct? I ask because one of the expected min values is -0.0052967071533203125 which is outside of the image. Should I generate new truth data and more tests for the different datatypes?

The expected values were generated by the same GPU code for previous run. It is a simple regression test that compares current results with previous launches.

Is there anything left that I need to do for this?

:+1:

Thank you for your contribution!

@dtmoodie Could you please squash commits into one to remove useless intermediate changes?
In case of problems try to follow this https://github.com/Itseez/opencv_contrib/pull/290#issuecomment-145509014. Also feel free to ask for help.

Sure thing, working on squashing now.  It looks like tortoiseGit doesn't want to squash due to the way I had the repo setup.  I'll have the squash finished and double checked in a bit.

Well done! :+1:

now we only need cuda::buildOpticalFlowPyramid to really take advantage of this..


Support id tokens from metadata service

This adds support for getting an ID token (as opposed to an oauth2 access token) from Google's metadata service.
Edit: it's a breaking change because it added a field to a public struct. Maybe it's worth marking the option structs as non_exhaustive to make such changes less disruptive in the future?

(as a drive-by commentator who would in theory find this useful)
Functionality to get an ID token would be welcome, but I don't think this is the right approach. Fundamentally, that TokenInfo represents an OAuth2 access token, and an ID token is quite explicitly not an access token (even though Cloud Run/Functions/etc. use them to validate access).
What ID tokens and access tokens have in common, is that they care about similar things (e.g. is there a metadata server, vs running on local workstation, or using impersonated credentials, or workforce federation). But both the exact method of getting the token, and the use of them, varies after that. One might also want to get ID tokens that include the service account email, or licensing details (requires extra parameters to get this information).
I would prefer that, if implemented, this should be a separate method, e.g. async fn id_token(...) -> Result<IdToken, Error>

Fundamentally, that TokenInfo represents an OAuth2 access token

I agree that it would be nice to have a better separation between ID tokens and OAuth2 access tokens, but as far as this crate stands today, TokenInfo already (optionally) contains both of them.

I would prefer that, if implemented, this should be a separate method, e.g. async fn id_token(...) -> Result<IdToken, Error>

Where would you put this method? One of the difficulties I had here was that the actual token-fetching logic is a few layers down from the public API, and I wasn't really sure about the best way of "threading" things through.

I had a closer look at the support for ID tokens in the library - and it's unfortunate. The id_token() call provided is a convenience method for accessing the ID token from an OAuth2 client; but it has nothing to do with the authentication required by services in Google Cloud - and the interface provided doesn't match the concepts needed to provide that authentication (for example, scopes are not defined for ID tokens, but it's currently a required parameter to retrieve one).
So I see why your code is the way it is, as the code is already very fuzzy here, and fixing that would take a much larger rework. Renaming "scope" as "audience" allows you to leverage the existing misunderstandings :)
But the larger view is that getting an ID token isn't an authentication flow; it (maybe) uses an already successful authentication flow. If we look at the available methods of getting an ID token (https://cloud.google.com/docs/authentication/get-id-token), we have:

Request a token from the local metadata service
Call the IAM credentials service using existing valid Google credentials

In the second case in particular, we would want a pre-existing Authenticator to make that call.
So this points to the idea that getting an ID token is layered on top of existing code to get a Google access token, rather than being at the same level. That code could exist in this, or another library (in general, it has been bundled together in the same library, if one looks at examples from other languages).


Add pyproject.toml for packaging aboutcode.pipeline module #1351

The dist archives can be manually built with:
$ cd scancode.io
$ source .venv/bin/activate
$ pip install flot
$ flot --pyproject pipeline-pyproject.toml --sdist --wheel --output-dir dist/

Also, regarding https://github.com/nexB/scancode.io/pull/1358#discussion_r1709767929, it seems that using the aboutcode.pipeline works fine:
pip install aboutcode.pipeline --find-links=scancode.io/dist --no-index --no-cache-dir
# Successfully installed aboutcode.pipeline-0.1

pip install aboutcode_pipeline --find-links=scancode.io/dist --no-index --no-cache-dir
# Successfully installed aboutcode_pipeline-0.1

pip install aboutcode-pipeline --find-links=scancode.io/dist --no-index --no-cache-dir
# Successfully installed aboutcode-pipeline-0.1

@keshav-space could you adapt and add the pypi-release-abc-pipeline.yml (let's use pypi-release-aboutcode-pipeline.yml) from https://github.com/nexB/scancode.io/pull/1332/files#diff-bbcc61089df1f0d385697aff0c2da0e9c972d8cf73acfe27f8cb437df2e92b46
Let's make sure we have a separate tag convention to trigger the releases of aboutcode.pipeline  independently of the ScanCode.io releases.

@tdruez, I've added the workflow to publish aboutcode.pipeline on PyPI, i've commented out the automatic trigger since we want to do the release manually.

@keshav-space I've enabled the automatic build/publish on tag. I've removed the release addition in GH though, not sure we want to put those releases there.


packery+imagesLoaded not working reliably

Hi there, I am evaluating packery + images loaded for my image gallery. Before I purchase packery, I wanted to make sure it works well for my solution.
The problem I am facing is that I am trying to show my image gallery (remote images) using packery with imagesLoaded. Sometimes when the view enters, it works and reflows.In that situation both my 'progress' and 'always' callbacks are called. However, sometimes when my view re-enters, it only calls 'always' and the images are all either vertically stacked in one column or underneath each other
However in that case when I drag, the images come out (but in one column). A video shows it best (note debug logs on the left - the code below explains the debug logs). The 2nd time i come to the view it messes up - but 3rd time is ok
Video: https://youtu.be/rctesH8_vSM
I use a custom image src directive - I don't think that will cause issues ? My directive is here https://github.com/pliablepixels/zmNinja/blob/master/www/js/app.js#L375
My code to handle rendering (and fix this issue, which so far is unsuccessful)
function initPackery()
{

        var progressCalled = false;

        var elem =  angular.element(document.getElementById("mygrid"));
        pckry = new Packery('.grid',
                             {
                                itemSelector: '.grid-item',
                                percentPosition: true,
                               columnWidth: '.grid-sizer',
                            });
         console.log ("**** mygrid is " + JSON.stringify(elem));

        imagesLoaded(elem).on('progress', function() {
                console.log ("******** SOME IMAGE LOADED");
                progressCalled = true;
                pckry.layout();
        });

        imagesLoaded(elem).on('always', function() {
                console.log ("******** ALL IMAGE LOADED");

                if (!progressCalled)
                {
                    // this is a hack - trying to figure out why this
                    // problem is occuring - my hack does not work
                    console.log ("*** BUG SOMECALLED WAS NOT CALLED");
                    pckry.destroy();
                    pckry = new Packery('.grid',
                             {
                                itemSelector: '.grid-item',
                                percentPosition: true,
                               columnWidth: '.grid-sizer',
                            });
                    pckry.layout();

                }

                pckry.getItemElements().forEach(function (itemElem) {
                      var draggie = new Draggabilly(itemElem);
                      pckry.bindDraggabillyEvents(draggie);
                       console.log ("**** MAPPED DRAG");
                     //  pckry.shiftLayout();
                });

        });

    }

Sorry to hear you're having trouble with Packery. Sounds like there are two issues that are happening

imagesLoaded doesn't always trigger in the same manner
Dragging causing problems

I would first make a simplified demo without images, so you can resolve the dragging issue.
If you're initializing Packery for the first time, use new Packery. If you're need to re-initialize packery after the DOM has changed, you may want to use reloadItems.
After you resolved the dragging issue, I'd then tackle the imageLoaded issue.

Thanks - the problem seems to  be related to angular and not packery, and how it handles loading DOM. I've fixed it (not related to packery)


Fix dependency generation issue encountered with blt_import_library + HIP + gtest

@davidbeckingsale discovered an issue with hip+gtest that appeared when HIP was added via the new blt_import_library.
The generator expression in FindHIP.cmake intended to build the include-related compiler flags via the INCLUDE_DIRECTORIES property for a target fails (is empty) when HIP is added as a "real" CMake target.  An identical generator expression for the INTERFACE_INCLUDE_DIRECTORIES property was appended to "solve" the problem, though it is not obvious to me why method used to import the HIP library affects unrelated include directories in an executable target that depends on it.
To mitigate similar issues in the future, smoketests were added for cuda+gtest and hip+gtest.

CC @kab163


question about license

Dear developers,
I have a question about the possibility of redistributing code from Termofisher (both executable and dll). I'm just wondering if this is ok, or not.
Many thanks,
Luca

Apache license added and renamed thermo license to THERMO_LICENSE

Hi @nielshulstaert, my question was not about adding a new kind of license but if you (and other people like me) are authorised to redistribute the dll code inside our own github repositories. Thanks!


Error to load the Xmodel on the board

I ran Vitis AI CPU docker container from host using ./docker_run.sh xilinx/vitis-ai-cpu:1.3.411. After successfully training my custom model using the following code
import os
import numpy as np
import tensorflow as tf
from sklearn.metrics import confusion_matrix
image_ordering_dim="tf"
def preprocess(X_train, X_test):
    """
    Convert from float64 to float32 and normalize normalize to decibels
    relative to full scale (dBFS) for the 4 sec clip.
    """
    X_train = X_train.astype('float32')
    X_test = X_test.astype('float32')

    X_train = np.array([(X - X.min()) / (X.max() - X.min()) for X in X_train])
    X_test = np.array([(X - X.min()) / (X.max() - X.min()) for X in X_test])
    return X_train, X_test
def prep_train_test(X_train, y_train, X_test, y_test, nb_classes):
    """
    Prep samples ands labels for Keras input by noramalzing and converting
    labels to a categorical representation.
    """
    print('Train on {} samples, validate on {}'.format(X_train.shape[0],
                                                       X_test.shape[0]))

    # normalize to dBfS
    X_train, X_test = preprocess(X_train, X_test)

    # Convert class vectors to binary class matrices
    Y_train = tf.keras.utils.to_categorical(y_train, nb_classes)
    Y_test = tf.keras.utils.to_categorical(y_test, nb_classes)

    return X_train, X_test, Y_train, Y_test
def keras_img_prep(X_train, X_test, img_dep, img_rows, img_cols):

    X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1)
    X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)

    return X_train, X_test, input_shape

def cnn(X_train, y_train, X_test, y_test, batch_size,
        nb_classes, epochs, input_shape):
    """

    """

    inputs = tf.keras.Input(shape=input_shape)
    x = tf.keras.layers.Conv2D(128, (3,3), activation='relu', input_shape=(input_shape),padding='same')(inputs)
    x = tf.keras.layers.MaxPooling2D((4,3),strides=(1, 3))(x)
    x = tf.keras.layers.Conv2D(32, (1,3), activation='relu',input_shape=(input_shape),padding='same')(x)
    x = tf.keras.layers.MaxPooling2D((1,3), strides=(1, 3))(x)
    x = tf.keras.layers.Flatten()(x)
    x = tf.keras.layers.Dense(512, activation='relu')(x)
    x = tf.keras.layers.Dense(512, activation='relu')(x)
    x = tf.keras.layers.Dropout(0.2)(x)
    outputs = tf.keras.layers.Dense(2, activation='softmax')(x)

    model = tf.keras.Model(inputs=inputs, outputs=outputs, name='classifier')

    model.compile(loss='categorical_crossentropy',
                  optimizer='adadelta',
                  metrics=['accuracy'])

    history = model.fit(X_train, y_train, batch_size=batch_size, epochs=epochs,
                        verbose=1, validation_data=(X_test, y_test))

    # Evaluate accuracy on test and train sets
    score_train = model.evaluate(X_train, y_train, verbose=0)
    print('Train accuracy:', score_train[1])
    score_test = model.evaluate(X_test, y_test, verbose=0)
    print('Test accuracy:', score_test[1])
    model.summary()

    return model, history

    X_train = np.load('train_samples.npz')
    y_train = np.load('train_labels.npz')
    X_test = np.load('test_samples.npz')
    y_test = np.load('test_labels.npz')

    X_train, y_train, X_test, y_test = \
        X_train['arr_0'], y_train['arr_0'], X_test['arr_0'], y_test['arr_0']

    batch_size = 32
    nb_classes = 2
    epochs = 7

    X_train, X_test, y_train, y_test = prep_train_test(X_train, y_train,
                                                       X_test, y_test,
                                                       nb_classes=nb_classes)

    # 513x125x1 for spectrogram with crop size of 125 pixels
    img_rows, img_cols, img_depth = X_train.shape[1], X_train.shape[2], 1

    X_train, X_test, input_shape = keras_img_prep(X_train, X_test, img_depth,
                                                  img_rows, img_cols)
    print('Fitting model...')
    model, history = cnn(X_train, y_train, X_test, y_test, batch_size,
                         nb_classes, epochs, input_shape)

After the quantization of the model I got this warning
!vai_q_tensorflow quantize \
    --input_frozen_graph frozen_graph_depression_test.pb \
    --input_fn input_func.calib_input \
    --output_dir quantized_depression \
    --input_nodes input_1 \
    --output_nodes dense_2/Softmax \
    --input_shapes ?,513,125,1\
    --calib_iter 32

INFO: Checking Float Graph...
2022-10-26 12:23:31.339312: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 434503680 exceeds 10% of system memory.
2022-10-26 12:23:31.479567: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 262656000 exceeds 10% of system memory.
2022-10-26 12:23:31.577909: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 85647360 exceeds 10% of system memory.
2022-10-26 12:23:31.608412: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 85647360 exceeds 10% of system memory.
INFO: Float Graph Check Done.
INFO: Calibrating for 32 iterations...
N/A% (0 of 32) |                         | Elapsed Time: 0:00:00 ETA:  --:--:--2022-10-26 12:23:42.735741: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 434503680 exceeds 10% of system memory.
100% (32 of 32) |########################| Elapsed Time: 0:02:33 Time:  0:02:33
INFO: Calibration Done.
2022-10-26 12:26:17.641835: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1687] [DECENT_WARNING] Shift bias of node conv2d/Conv2D is<PHONE_NUMBER>. It exceed range [-7, 16], modify quantize pos from 19 to<PHONE_NUMBER>
2022-10-26 12:26:17.641863: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1687] [DECENT_WARNING] Shift bias of node conv2d_1/Conv2D is<PHONE_NUMBER>. It exceed range [-7, 16], modify quantize pos from 20 to<PHONE_NUMBER>
2022-10-26 12:26:17.641867: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1687] [DECENT_WARNING] Shift bias of node dense/MatMul is<PHONE_NUMBER>. It exceed range [-2, 16], modify quantize pos from 21 to<PHONE_NUMBER>
2022-10-26 12:26:17.641871: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1687] [DECENT_WARNING] Shift bias of node dense_1/MatMul is<PHONE_NUMBER>. It exceed range [-6, 16], modify quantize pos from 20 to<PHONE_NUMBER>
2022-10-26 12:26:17.641873: W tensorflow/contrib/decent_q/utils/graph_quantizer.cc:1687] [DECENT_WARNING] Shift bias of node dense_2/MatMul is<PHONE_NUMBER>. It exceed range [-6, 16], modify quantize pos from 26 to<PHONE_NUMBER>
INFO: Generating Deploy Model...
INFO: Deploy Model Generated.
********************* Quantization Summary *********************
INFO: Output:
quantize_eval_model: quantized_depression/quantize_eval_model.pb
deploy_model: quantized_depression/deploy_model.pb
And the compilation went without any errors and I get the Xmodel.

VITIS_AI Compilation - Xilinx Inc.

[INFO] Namespace(inputs_shape=None, layout='NHWC', model_files=['quantized_depression/quantize_eval_model.pb'], model_type='tensorflow', out_filename='./depression_test_org.xmodel', proto=None)
[INFO] tensorflow model: quantized_depression/quantize_eval_model.pb
[INFO] parse raw model     :100%|█| 26/26 [00:00<00:00, 21880.40it/s]
[INFO] infer shape (NHWC)  :100%|█| 33/33 [00:00<00:00, 1010.51it/s]
[INFO] infer shape (NHWC)  :100%|█| 30/30 [00:00<00:00, 742.64it/s]
[INFO] generate xmodel     :100%|█| 30/30 [00:03<00:00,  7.62it/s]
[INFO] generate xmodel: /workspace/depression_test_org.xmodel
[UNILOG][INFO] The compiler log will be dumped at "/tmp/vitis-ai-user/log/xcompiler-20221026-122808-1062"
[UNILOG][INFO] Target architecture: DPUCZDX8G_ISA0_B4096_MAX_BG2
[UNILOG][INFO] Compile mode: dpu
[UNILOG][INFO] Debug mode: function
[UNILOG][INFO] Target architecture: DPUCZDX8G_ISA0_B4096_MAX_BG2
[UNILOG][INFO] Graph name: quantize_eval_model, with op num: 44
[UNILOG][INFO] Begin to compile...
[UNILOG][WARNING] xir::Op{name = max_pooling2d_1/MaxPool, type = pool-fix} has been allocated to CPU.
[UNILOG][INFO] Total device subgraph number 5, DPU subgraph number 2
[UNILOG][INFO] Compile done.
[UNILOG][INFO] The meta json is saved to "/workspace/./meta.json"
[UNILOG][INFO] The compiled xmodel is saved to "/workspace/./depression_test.xmodel"
[UNILOG][INFO] The compiled xmodel's md5sum is e5f0d20e2578e60dd9efc74b76946bfe, and been saved to "/workspace/./md5sum.txt"
However wen trying to load the Xmodel on the ZCU104 board  as follow

from time import time
# import sklearn
import numpy as np
import cv2
# from sklearn import svm
import matplotlib.pyplot as plt
%matplotlib inline
from six.moves import urllib
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
urllib.request.install_opener(opener)
from pynq_dpu import DpuOverlay
overlay = DpuOverlay("dpu.bit")
overlay.load_model("depression_test.xmodel")

AssertionError                            Traceback (most recent call last)
 in
1 from pynq_dpu import DpuOverlay
2 overlay = DpuOverlay("dpu.bit")
----> 3 overlay.load_model("depression_test.xmodel")
4 # overlay.load_model("depression_flatten_model.xmodel")
/usr/local/share/pynq-venv/lib/python3.8/site-packages/pynq_dpu/dpu.py in load_model(self, model)
169             self.graph = xir.Graph.deserialize(abs_model)
170             subgraphs = get_child_subgraph_dpu(self.graph)
--> 171             assert len(subgraphs) == 1
172             self.runner = vart.Runner.create_runner(subgraphs[0], "run")
AssertionError:
I don't get what's the problem

Hi @Afef00
Do you use PYNQ system image?
If not, you can refer to https://github.com/Xilinx/Vitis-AI/blob/1.3.1/demo/VART/inception_v1_mt_py/inception_v1.py python example.

@qianglin-xlnx Yes, I used PYNQ-DPU (PYNQ image 2.7). The problem is when I tried other models, such as those provided on Jupyter notebook or other Xmodels that I tested before, it works fine. I don't get what the problem is.

From the log you provide, the model you compiled has 2 DPU subgraphs.
[UNILOG][INFO] Total device subgraph number 5, DPU subgraph number 2
While your program assert at the number of subgraph, which the DPU subgraph must be 1 in your program.  Maybe you can modify this in your code.
--> 171 assert len(subgraphs) == 1

@qianglin-xlnx How can I modify this in the code?

I tried to modify the model architecture as follow:
inputs = tf.keras.Input(shape=input_shape)
  x = tf.keras.layers.Conv2D(128, (3,3), activation='relu', input_shape=(input_shape))(inputs)
  x = tf.keras.layers.MaxPooling2D((4,3))(x)
  x = tf.keras.layers.Conv2D(32, (1,3), activation='relu')(x)
  x = tf.keras.layers.MaxPooling2D((1,3))(x)
  x = tf.keras.layers.Flatten()(x)
  x = tf.keras.layers.Dense(512, activation='relu')(x)
  x = tf.keras.layers.Dense(512, activation='relu')(x)
  outputs = tf.keras.layers.Dense(2, activation='softmax')(x)

  model = tf.keras.Model(inputs=inputs, outputs=outputs, name='classifier')

However, when I tried to compile the model I got this error:

VITIS_AI Compilation - Xilinx Inc.

[INFO] Namespace(inputs_shape=None, layout='NHWC', model_files=['quantized_depression_V2/quantize_eval_model.pb'], model_type='tensorflow', out_filename='./depression_V2_org.xmodel', proto=None)
[INFO] tensorflow model: quantized_depression_V2/quantize_eval_model.pb
[INFO] parse raw model     :100%|█| 26/26 [00:00<00:00, 27134.09it/s]
[INFO] infer shape (NHWC)  :100%|█| 33/33 [00:00<00:00, 5077.85it/s]
[INFO] infer shape (NHWC)  :100%|█| 30/30 [00:00<00:00, 4038.81it/s]
[INFO] generate xmodel     :100%|█| 30/30 [00:01<00:00, 17.32it/s]
[INFO] generate xmodel: /workspace/depression_V2_org.xmodel
[UNILOG][INFO] The compiler log will be dumped at "/tmp/vitis-ai-user/log/xcompiler-20221031-153814-245"
[UNILOG][INFO] Target architecture: DPUCZDX8G_ISA0_B4096_MAX_BG2
[UNILOG][INFO] Compile mode: dpu
[UNILOG][INFO] Debug mode: function
[UNILOG][INFO] Target architecture: DPUCZDX8G_ISA0_B4096_MAX_BG2
[UNILOG][INFO] Graph name: quantize_eval_model, with op num: 44
[UNILOG][INFO] Begin to compile...
[UNILOG][WARNING] xir::Op{name = max_pooling2d_1/MaxPool, type = pool-fix} has been allocated to CPU.
[UNILOG][Check Failed: bank_addr_end >= min_runtime_size][/home/xbuild/conda-bld/xcompiler_1606385139459/work/src/pass/passes/GenInst/AssignBankGroup.imp:1858][XCOM_ALLOCATE_BANK_FAIL][There is not enough bank space for the tensor]
*** Check failure stack trace: ***
Aborted (core dumped)

Hi @Afef00
Sorry about the late reply. Is this issue solved?
My team does not have PYNQ-DPU. If you still have problem, you can ask PYNQ-DPU team for help.


feat/127 - initial implementation

more frequent pr's

Hi Kevin,
Seems the crud is still in progress.
Is it the tests that need to be reviewed or the schema or, is this pr about refactoring?
Just want to get an idea.

Hey Ian, this is my initial implementation of a get role assignments end point.  In the interests of smaller pr's I got it to the point where its working, and tests are passing so I have pushed and created this pr... does that help?

Sure, I was confused, I will look again.

I've worked on the front-end needing this request, and here's the sample data I'm hoping to get. If the format is different that's fine, but I need e.g. role names, user login id, instead of database primary keys. I haven't checked the PR to see what is being returned currently.
userRoleAssignments.value = [
{
assignment_id: 1,
user_id: 'foo-test',
user_domain: 'IDIR',
role: 'Reviewer',
},
{
assignment_id: 2,
user_id: 'bar-test',
user_domain: 'BCeID',
role: 'Submitter',
forest_client_number: '01234567'
},
{
assignment_id: 3,
user_id: 'do-test',
user_domain: 'BCeID',
role: 'Submitter',
forest_client_number: '00008765'
},
]

Tried hard to get something that might work for you @basilv.  My latest push will return the following struct on GET application/<app_id>/user_role_assignements  (below).  I'm confident I can get to a flatter structure that you identified above, just ran out of time today.  Still got a lot of cleanup to do, but its passing the tests, and its closer to what you were looking for.

[
    {
        "user_role_xref_id": 1,
        "user": {
            "user_type_code": "B",
            "cognito_user_id": "22ftw",
            "user_name": "Mike Bossy"
        },
        "role": {
            "role_name": "FAM_ADMIN",
            "create_user": "John Doe",
            "role_type_code": "C",
            "client_number": null
        }
    },
    {
        "user_role_xref_id": 2,
        "user": {
            "user_type_code": "B",
            "cognito_user_id": "22ftw",
            "user_name": "Mike Bossy"
        },
        "role": {
            "role_name": "FAM_TEST",
            "create_user": "Patrick Roy",
            "role_type_code": "C",
            "client_number": {
                "client_name": "acme forestry",
                "forest_client_number": 1101
            }
        }
    }
]

This structure is fine for me, I'd say just stick with this. I'll eventually want forest client name once we populate it properly. I'd like to use user_name as well, but I'm curious when this field is populated, I didn't think we'd have it available. I assume user_type_code B = BCEID, I = IDIR. (Not sure why we don't just spell those out rather than use a one-letter code).

This structure is fine for me, I'd say just stick with this. I'll eventually want forest client name once we populate it properly. I'd like to use user_name as well, but I'm curious when this field is populated, I didn't think we'd have it available. I assume user_type_code B = BCEID, I = IDIR. (Not sure why we don't just spell those out rather than use a one-letter code).

@basilv , we actually did have that (BCEID/IDIR, not B or I) as what you said in initial version. But DA consider it should be B or I as code with comment on what B/I is so that's why. I would prefer BCEID/IDIR though.... but maybe that's the standard from DA point of view.


Potential regression in Schema / nullability calculations after upgrade to 42.0.0

Describe the bug
@phillipleblanc  and @itsjunetime  have both hit upgrades related to nullability and other metadata in schemas after the DataFusion 42 upgrade.
In addition, @ion-elgreco has it something similar while updating in delta.rs (see https://github.com/delta-io/delta-rs/pull/2886)
I am filing this ticket to make this more visibility
To Reproduce
Not sure (maybe someone could create a self contained reproducer of the problem)
Expected behavior
No response
Additional context
This might have been introduced here: https://github.com/apache/datafusion/pull/11989
There is a discussion happening here https://github.com/apache/datafusion/pull/11989#issuecomment-2360601240

I'm running into this behavior after #11989, specifically seeing schema mismatches where the only thing that is different is that a field's metadata disappears at some point (so the schemas are the same except for a field's metadata). E.g.:
&physical_input_schema = Schema {
    fields: [
        Field {
            name: "alias1",
            data_type: Utf8,
            nullable: true,
            dict_id: 0,
            dict_is_ordered: false,
            metadata: {},
        },
    ],
    metadata: {},
}
&physical_input_schema_from_logical = Schema {
    fields: [
        Field {
            name: "alias1",
            data_type: Utf8,
            nullable: true,
            dict_id: 0,
            dict_is_ordered: false,
            metadata: {
                "some_key": "some_value"
            },
        },
    ],
    metadata: {},
}

I've yet to figure out exactly where the metadata is being dropped and I haven't figured out a reproducer either. I suggested comparing only the fields' non-metadata fields here, but @jayzhan211 pointed out that that's more of a workaround than an actual fix, as it's still a problem if the metadata is disappearing.
The issue that I'm running into, though, seems to be somewhat different than the issue that others (like @phillipleblanc) are running into, where some fields completely disappear from the schema (see here). I don't think these are the same issue, exactly (since they manifest differently), but they may have the same root cause/solution, so I think it's fair to keep them all under this issue unless needed otherwise.
I'll work on getting a fix or reproducer today

Just to add to the visibility: we are also observing the same behavior after updating to 42.0.0. This is the error message we are getting, to help with debugging:
Caused by:
    0: Arrow error: Invalid argument error: Column 'column1' is declared as non-nullable but contains null values
    1: Invalid argument error: Column 'column1' is declared as non-nullable but contains null values

We are getting this error when reading some parquet files using Datafusion. I have verified with other tools (parquet cli, DuckDB) column1 does not contain any null values and metadata of the files is correct.

Just to add to the visibility: we are also observing the same behavior after updating to 42.0.0. This is the error message we are getting, to help with debugging:
Caused by:
    0: Arrow error: Invalid argument error: Column 'column1' is declared as non-nullable but contains null values
    1: Invalid argument error: Column 'column1' is declared as non-nullable but contains null values

We are getting this error when reading some parquet files using Datafusion. I have verified with other tools (parquet cli, DuckDB) column1 does not contain any null values and metadata of the files is correct.

Is there any small parquet file that has the same error, if we can reproduce the error easily, it is easier to find the root cause.

Just to add to the visibility: we are also observing the same behavior after updating to 42.0.0. This is the error message we are getting, to help with debugging:
Caused by:
    0: Arrow error: Invalid argument error: Column 'column1' is declared as non-nullable but contains null values
    1: Invalid argument error: Column 'column1' is declared as non-nullable but contains null values

We are getting this error when reading some parquet files using Datafusion. I have verified with other tools (parquet cli, DuckDB) column1 does not contain any null values and metadata of the files is correct.

Is there any small parquet file that has the same error, if we can reproduce the error, it is easier to find the root cause.

We have been digging more regarding this error and it seems it is not related to the Datafusion upgrade, I apologize for the confusion. Therefore there's no file I can provide :(

In fixing one of the schema-metadata-drops, I ended by triggering the Physical input schema should be the same as the one converted from logical input schema. error in a test case.
I feel like this introduced check is inadvertently surfacing other schema-handling bugs that we have. Note that I'm unclear if the reproducer I found is a bug, or if we expect additional metadata changes on physical plan creation (not familiar with that part of the code).

We have also been digging in what we saw in InfluxDB 3.0 and I filed https://github.com/apache/datafusion/issues/12687 to track it separately. Let's close this omnibus issue and we file individual issues for specific problems as we find them


OpenAI - Support audio input modality

Extend OpenAiApi to support the latest version of the Chat Completion API, including input and output audio modality.
Support input audio modality in OpenAiChatModel via the existing multimodality support in Spring AI.

Fixes gh-1560

Thanks @ThomasVitale, took us some time but now this will be part of M5 ;)
Rebased, conflict resolved, minor improvements, squashed and merged at 33f431e4a733e684d3ea9723cd8b2eb1c87c4308

Thank you!


memory corruption while using appending to a slice across multiple goroutines

What version of Go are you using (go version)?

$ go version
1.13.8 and 1.14.0

Does this issue reproduce with the latest release?
Yes - 1.14.0
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/bion/Library/Caches/go-build"
GOENV="/Users/bion/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/bion/modo/coin"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/mb/50wl13bn0t386wx67ns686p80000gn/T/go-build304849267=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?
https://play.golang.org/p/Jwo8Ub1mU8L
I think I've simplified this as far as I can. It's tricky to get it to crash at this point (but it does occasionally happen!), but it reliably panics in ways I don't expect. This was originally a benchmark, which is reproduced here: https://gist.github.com/bionoren/c8e45a802dbe0c0be6f1924b56a375f7 and I ran it with

go test -run=xxx -bench=BenchmarkCollection_Transition -benchmem -count=10 .

What did you expect to see?
Benchmark passes, or at least panics in my code somewhere.
What did you see instead?
Sometimes it passes, sometimes it panics - usually due to null pointer exceptions, but I've seen several reasons and stack traces, often down in the json or ioutil packages. On rare occasion, it encounters a fatal error. I've seen a variety of these while trying to trim my example down:

runtime: nameOff 0x400 base 0x11cbd70 not in ranges:
	types 0x17a6520 etypes 0x1bcc82e
fatal error: runtime: name offset base pointer out of range

runtime: pointer 0xc00379ac60 to unused region of span span.base()=0xc001794000 span.limit=0xc001795e00 span.state=1
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime: nelems=8 nalloc=5 previous allocCount=4 nfreed=65535
fatal error: sweep increased allocation count

fatal error: unexpected signal during runtime execution
[signal SIGBUS: bus error code=0x2 addr=0x16dc358 pc=0x100c850]

panic: runtime error: invalid memory address or nil pointer dereference
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]

I realize this fails go vet, and I realize that's causing me to use append() on a slice across multiple go-routines, but I expected that to result in a slice of unusual contents, not what appears to be stack corruption resulting in a crash.

Have you run the program under the race detector? The Go memory model does not allow concurrent goroutines to append to the same slice.

(https://research.swtch.com/gorace is a good read if you haven't seen it already.)

I had not seen it. The go memory model article was particularly interesting.
Thank you! I knew concurrent appends to the same slice was a bad idea, but I hadn't realized it was officially not allowed.


Include Sanitized Capabilities with Suite Title

When running tests against multiple browsers it is impossible to determine which result in the mochawesome report goes to which browser execution.
Add option to include sanitized capabilities with the suite title

Can't wait for this, esp. with Allure giving me essentially the same if not worse issues.

@mherodev v1.1.1 has been released.  This will append the WDIO sanitizedCapabilities to the end of the suite name.


runtime-transactions: fix calculated charged_fee for non-evm txs

For non-evm transactions there are no fee refunds as there are for EVM transactions (where this is done to ensure EVM compatibility).

woa do we really do this in the client query area

Haha I was similarly taken aback. We have similarly misplaced (= not abstracted into a special layer) transformations in a few other places too, but I think it's tolerable (if not great) since these kinds of transformations are rare and lightweight in nexus for now.


Duplicate chapters in manga's

After looking at the HTML file, it seems that some mangas will have their single chapter splited into multiple parts. Currently the script assumes that chapter n.1 and chapter n.2 is a continuation of chapter n.
In reality, chapter n.1 and chapter n.2 makes up a single chapter, which is chapter n.
I will check other mangas to make sure that point chapters (chapter like n.1) is actually a part of a chapter.

So apparently the chapters pattern is not consistent. Some manga will 2 special chapters and it will be labeled as chapter N.5 and N.6. Im just gonna patch the chapter label parser to ignore all N.n chapters except chapter N.5 and N.6.


Initial implementation

Nice job indeed! I left only a few "response" comments and will need to go through the code more in details, but it's a good start @rawe0 :+1:

BTW I have created a https://github.com/cs3org/wopiserver/tree/cs3client branch and invited @rawe0 to the cs3org Organization, so that we can also start looking at the modifications of the wopiserver to adopt this library. We can do something similar for the cs3api4lab repo.

Go ahead @glpatcern (I cannot :) )


readme fix

it is only a fix in the readme

Coverage remained the same at 73.394% when pulling 54d89447d41cb409a4adacbc1a1f47da68941214 on frankenstein91:master into 26d52a6c9d8c377fa3e453cf80609372ae264c88 on nficano:master.


feat: Plugin configuration at ApisixRoute level instead of repeated for each rule

Hello Team,
ApisixRoute CRD allows defining plugins to use at rule level only, either with plugins or plugin_config_name element.
Currently, when 1 ApisixRoute has multiple rules, the plugins configuration must be repeated to each rule. Even though we can define the plugin configuration in an ApisixPluginConfig, we still need to specify the plugin_config_name element for each rule.
Please feel free to comment and propose better approach.
Let me know if any information is missing or I did not respect some rule when writing this issue, this is the first one I am opening on the project.
High level proposal

Allow plugin configurations at ApisixRoute level

Note: I am not sure whether correct place would be directly under the “spec” element or the “http” element

Use same elements types as existing ones for defining plugins configuration (plugins or plugin_config_name)
Precedence rules must be defined

Precedence rules should be consistent with other existing uses cases
The most specific location overrides global configuration
So if a plugin is defined at rule level, and same plugin is also defined at ApisixRoute level, the plugin configuration at rule level overrides the one from ApisixRoute level

Sample ApisixRoute BEFORE change (current situation)
Using plugins element
The proxy-rewrite plugin configuration must be repeated for each rule
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: route-1-httpbin
  namespace: httpbin
spec:
  http:
    - name: rule-1
      # Criteria for a request to match this rule
      match:
        paths:
          - /httpbin/*
        methods:
          - GET
      backends:
        # Name (metadata.name) of target ApisixUpstream object
        - serviceName: httpbin
          servicePort: 80
          resolveGranularity: service
      plugins:
        - name: proxy-rewrite
          enable: true
          config:
            regex_uri: [ "/httpbin/(.*)", "/$1" ]
    - name: rule-2
      # Criteria for a request to match this rule
      match:
        paths:
          - /httpbin/status/*
        methods:
          - POST
      backends:
        # Name (metadata.name) of target ApisixUpstream object
        - serviceName: httpbin
          servicePort: 80
          resolveGranularity: service
      plugins:
        - name: proxy-rewrite
          enable: true
          config:
            regex_uri: [ "/httpbin/(.*)", "/$1" ]

Using plugin_config_name element
The plugin_config_name element must be repeated for each rule
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: route-1-httpbin
  namespace: httpbin
spec:
  http:
    - name: rule-1
      # Criteria for a request to match this rule
      match:
        paths:
          - /httpbin/*
        methods:
          - GET
      backends:
        # Name (metadata.name) of target ApisixUpstream object
        - serviceName: httpbin
          servicePort: 80
          resolveGranularity: service
      # Plugin configuration is the name of ApisixPluginConfig object
      plugin_config_name: httpbin-plugin-config
    - name: rule-2
      # Criteria for a request to match this rule
      match:
        paths:
          - /httpbin/status/*
        methods:
          - POST
      backends:
        # Name (metadata.name) of target ApisixUpstream object
        - serviceName: httpbin
          servicePort: 80
          resolveGranularity: service
      # Plugin configuration is the name of ApisixPluginConfig object
      plugin_config_name: httpbin-plugin-config

Sample ApisixRoute AFTER change (current situation)
Using plugins element
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: route-1-httpbin
  namespace: httpbin
spec:
  # This plugin configuration should apply to all rules
  plugins:
    - name: proxy-rewrite
      enable: true
      config:
        regex_uri: [ "/httpbin/(.*)", "/$1" ]
  http:
    - name: rule-1
      # Criteria for a request to match this rule
      match:
        paths:
          - /httpbin/*
        methods:
          - GET
      backends:
        # Name (metadata.name) of target ApisixUpstream object
        - serviceName: httpbin
          servicePort: 80
          resolveGranularity: service
      # proxy-rewrite does not need anymore to be defined here, but can be overridden if required
      # Other plugins configurations specific to this rule could be added
    - name: rule-2
      # Criteria for a request to match this rule
      match:
        paths:
          - /httpbin/status/*
        methods:
          - POST
      backends:
        # Name (metadata.name) of target ApisixUpstream object
        - serviceName: httpbin
          servicePort: 80
          resolveGranularity: service
      # proxy-rewrite does not need anymore to be defined here, but can be overridden if required
      # Sample additional plugin configuration which should apply to this rule
      # in addition to the globally defined proxy-rewrite
      plugins:
        - name: request-id
          enable: true

Using plugin_config_name element
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: route-1-httpbin
  namespace: httpbin
spec:
  # This plugin configuration should apply to all rules
  plugin_config_name: httpbin-plugin-config
  http:
    - name: rule-1
      # Criteria for a request to match this rule
      match:
        paths:
          - /httpbin/*
        methods:
          - GET
      backends:
        # Name (metadata.name) of target ApisixUpstream object
        - serviceName: httpbin
          servicePort: 80
          resolveGranularity: service
      # proxy-rewrite does not need anymore to be defined here, but can be overridden if required
      # Other plugins configurations specific to this rule could be added
    - name: rule-2
      # Criteria for a request to match this rule
      match:
        paths:
          - /httpbin/status/*
        methods:
          - POST
      backends:
        # Name (metadata.name) of target ApisixUpstream object
        - serviceName: httpbin
          servicePort: 80
          resolveGranularity: service
      # proxy-rewrite does not need anymore to be defined here, but can be overridden if required
      # Sample additional plugin configuration which should apply to this rule
      # in addition to the globally defined proxy-rewrite
      plugin_config_name: request-id-config

Thank you Team for your good work !
Sébastien

Since we will differentiate layer4 and layer7 by spec.http and spec.stream, and they are very different.
I think this field can be changed to spec.http.plugins and spec.http.plugin_config_name, what do you think?

In addition, I think we also need to describe its priority and what the final result will be when these fields are configured at the same time.
Normally, configurations with a smaller scope should have higher priority.

Since we will differentiate layer4 and layer7 by spec.http and spec.stream, and they are very different. I think this field can be changed to spec.http.plugins and spec.http.plugin_config_name, what do you think?

Our current field is spce.http[].plugins and spec.http[].plugin_config_name.
Our spec.http is an array, rule level.
If this feature is supported, http should be an object, which would make an incompatible change, which would require a V3 API version.

Since we will differentiate layer4 and layer7 by spec.http and spec.stream, and they are very different. I think this field can be changed to spec.http.plugins and spec.http.plugin_config_name, what do you think?

Hello,
Indeed I see currently plugins can be configured only on http routes (according to ApisixRoute/v2 Reference: https://apisix.apache.org/zh/docs/ingress-controller/references/apisix_route_v2/)
So seems reasonnable

In addition, I think we also need to describe its priority and what the final result will be when these fields are configured at the same time.
Normally, configurations with a smaller scope should have higher priority.

Hello,
Yes I agree. Do you want me to edit proposal ?
Do you have an example (from other bug or documentation) I could use to define

If this feature is supported, http should be an object, which would make an incompatible change, which would require a V3 API version.

Hello,
That is a good remark. spec.http[].plugin_config_name indeed is a single value
I don't know what is best practice in such cases.
Versioning (v3) probably is cleaner.
Another - hacky - option would be keeping current spec.http[].plugin_config_name field and adding an optional array named spec.http[].plugin_config_names (with plural). In that case old configurations would be compatible. Priority would need to be defined when both plugin_config_name and plugin_config_names are present.
Thanks !


Flathub

Describe Your Suggestion
It would be great to request VPKEdit (GUI version) on Flathub, it makes it convenient to install and update a huge number of FOSS applications between numerous Linux distributions.
And in general, would you be okay with someone requesting VPKEdit on Flathub themselves? (of course with keeping of the authorship and the use of the original repository for the build, this is not even negotiable).
Since like it's not much Quake/GoldSrc-oriented FOSS + cross-platform + GUI tools published on Flathub yet, and that definitely needs to be fixed. At least TrenchBroom was requested on Flathub.
Although if you can do it yourself and maintain it, then here is the documentation on how to request: https://docs.flathub.org/docs/for-app-authors/submission

I'll look into this when I have some free time


Clarify MaxCDN URLs now that MaxCDN is retiring

In https://blog.stackpath.com/maxcdn-and-securecdn-are-retiring-heres-what-it-means-for-you/, it says

On August 26, 2022, technical support will no longer be available for:
...
On December 31, 2022, the services and portals above will no longer be in operation.

What does that mean for the emoji's hosted under https://twemoji.maxcdn.com?

I got a response from Stackpath sales that said in part:

Thank you for reaching out. Since the platform is being sunset, you will need to move the domain to an active CDN.
...
The next step is to create an account https://control.stackpath.com/register and add CDN. Let me know if you've done this so I can process the refund. Do you have any questions?

So it seems like at this stage the twemoji.maxcdn.com domain won't be available after December 31, 2022.

Thanks for bringing this to our attention @danielcompton. We don't have a solution yet but we're working on sorting one out!

Why not just use jsdelivr?
Example:
<EMAIL_ADDRESS>

Maybe someone who controls "twemoji.maxcdn.com" could point that domain to a custom CDN endpoint we can prepare?
The jsDelivr team has done a migration for rawgit.com in the past and later for bootstrapcdn.com without any downtime for the users


NNLC adjustments and updated models

Improve NNLC runner to mellow out response with new models

When no "deliberate" lateral jerk is desired, lateral jerk is not used at all for feedforward or error correction
Decreased tolerance on what counts as "deliberate" lateral jerk

Improve model lookup process
New batch of community-data-based NNLC models

EPS-specific models where possible
Time-aligned input training data for higher quality fits

@twilsonco - any chance I can get you some logs / routes for a Kia Seltos 2023 non-SCC before this is merged? Or is it too late?


Opening read-only document on Mono 4.0.0 throws exception

I'm currently trying to use the Open XML SDK on the latest Mono release, i.e., 4.0.0. When creating a read-only instance of WordprocessingDocument, using WordprocessingDocument.Open(fileName, false), the following exception is thrown:
[ERROR] FATAL UNHANDLED EXCEPTION: System.IO.IOException: Operation not valid when package is read-only
  at System.IO.Packaging.Package.CheckIsReadOnly () [0x00000] in <filename unknown>:0
  at System.IO.Packaging.PackagePart.CreateRelationship (System.Uri targetUri, TargetMode targetMode, System.String relationshipType, System.String id, Boolean loading) [0x00000] in <filename unknown>:0
  at System.IO.Packaging.PackagePart.LoadRelationships (System.Collections.Generic.Dictionary`2 relationships, System.IO.Stream stream) [0x00000] in <filename unknown>:0
  at System.IO.Packaging.PackagePart.get_Relationships () [0x00000] in <filename unknown>:0
  at System.IO.Packaging.PackagePart.GetRelationships () [0x00000] in <filename unknown>:0
  at DocumentFormat.OpenXml.Packaging.PackagePartRelationshipPropertyCollection..ctor (System.IO.Packaging.PackagePart packagePart) [0x00000] in <filename unknown>:0
  at DocumentFormat.OpenXml.Packaging.OpenXmlPart.Load (DocumentFormat.OpenXml.Packaging.OpenXmlPackage openXmlPackage, DocumentFormat.OpenXml.Packaging.OpenXmlPart parent, System.Uri uriTarget, System.String id, System.Collections.Generic.Dictionary`2 loadedParts) [0x00000] in <filename unknown>:0
  at DocumentFormat.OpenXml.Packaging.OpenXmlPartContainer.LoadReferencedPartsAndRelationships (DocumentFormat.OpenXml.Packaging.OpenXmlPackage openXmlPackage, DocumentFormat.OpenXml.Packaging.OpenXmlPart sourcePart, DocumentFormat.OpenXml.Packaging.RelationshipCollection relationshipCollection, System.Collections.Generic.Dictionary`2 loadedParts) [0x00000] in <filename unknown>:0
  at DocumentFormat.OpenXml.Packaging.OpenXmlPackage.Load () [0x00000] in <filename unknown>:0

This seems to be a bug in Mono (see http://lists.ximian.com/pipermail/mono-bugs/2011-February/109577.html). However, the question is whether there is a workaround on the level of the Open XML SDK.

I encountered the same issue, but through the SpreadsheetDocument interface. I gather they rely on the same code.
In my case I didn't explicitly require the file to be read-only so I left it as editable, but my understanding is that changes you make would happen to the document when it's flushed. I imagine this was not possible in your case?
I also wish there was more info on that Mono bug, it was created in Feb 2011. Only 4 years ago!

@jhare Right, it is a bit puzzling how long these bugs live.
@EricWhiteDev Would this be fixed by your new implementation of System.IO.Packaging (see also #40)?

@ThomasBarnekow @jhare I would certainly guess that this same issue will not show up when using the new System.IO.Packaging.  However, I have heard that there are issues with the System.IO.Compression namespace on Mono.  It fails in some circumstances where it should not.  I haven't researched this, though.

The read-only bug looks like it's one of the bugs I fixed on my branch a year ago. I never came to create a pull request since the last commit on my branch breaks quite some unit tests and I had to move forward to other stuff. I still keep an eye on this stuff though.
https://github.com/dmunch/mono/commit/bee81085af07a3ef1f64c024924be63f7b0f344c
I'm not sure if it's still worth to get those fixes in mono upstream, or of the new System.IO.Packaging is going to replace the mono implementation anytime soon? Any plans on this?

There's a pending PR on Mono that seems to fix it the same way as @dmunch did: https://github.com/mono/mono/pull/1255. I'll ping them to see if it can get merged.

I remember a long discussion on the mono-dev mailing list about pull requests: http://mono.1490590.n4.nabble.com/Pull-Requests-td4664224.html

The best approach is to stay engaged in the pull requests and bring the attention to the mailing list for us to discuss.

@dmunch, @akoeplinger I agree, I apologize for the slow response on this.  At the time, I was working on System.IO.Packaging, and it was taking longer than I planned for, we didn't have XUnit tests for the Open XML SDK, so difficult to validate fixes.  Also feeling pressure to get the SDK for Office 2016 put together.  But we are in a somewhat better shape now, although XUnit tests are not to where I want them.  I will examine that PR in detail later this week and get the fix in vNext branch.

fwiw, I just fell over this issue when trying to get some Windows document handling code running under Mono 4.2.2 on OSX, with an older version of the SDK.
It seems ok with a local build of Mono 4.3.3, but it would have been nice if the patch linked above could have found its way into a 4.2.x build, given how simple the fix seems to have been.

@Numpsy yeah, the release cycle for Mono 4.2 was quite long and for the current service release only high priority fixes went in. We'll be branching for Mono 4.4 in the near future and as you said it'll contain the fix.

So is there workaround?

It seems to be working in the 4.4.0 alpha, I don't know beyon that.

I use Mono 5.0.1 and I seem to be getting the same bug. Do you have a solution to this? Or as the above person has said, should I just use 4.4.0 alpha?

Still having the same problem using Xamarin + Mono <IP_ADDRESS> on MacOs

Hi,
Same issue here than the one mentioned above by @jeongsd and @acabutto (with Mono <IP_ADDRESS> on Ubuntu).
Any news on this issue? Or any workaround?

Hello, Mono <IP_ADDRESS> still have the same problem. Have you solve your problem and how? @jeongsd @acabutto @RocheVal


[FEATURE] New styles for profile section header

Description
we can change the profile section from
##this

to this

hence making it more promising in continuity with #2141
Screenshots
No response
Additional information
No response

This issue can be divided into 2 parts

making ui for the page (working on)
integrating and making it dynamic

If the issue is being split up in to smaller issues, please close this issue to avoid confusion and PRs can be created for each smaller issue

I have created a pull request for this, I am closing #2141

I have created a pull request for this, I am closing #2141 as the issue is been split up into #2171, #2182 and one more issue that is not created yet

Let's stick with the current design as this comes with Tailwind components we purchased and we can revisit this in the future.


Production demos import issues with moment path and chart.js version

Working with the production demo there are 2 issues (index.html 1-3 and others) :
  <!-- daterangepicker -->
  <script type="text/javascript" src="js/moment.min.js"></script>

Seemingly moment was moved to its own folder
FIX:
  <script type="text/javascript" src="js/moment/moment.min.js"></script>
Also included chart.js version 2.0.0-beta2 gives error with the Doughnut function
Reverting to 1.0.1-beta.4 is a temporary fix

Fixes included on https://github.com/puikinsh/gentelella/pull/54


Even more tweaks to the explorer and stuff

:white_check_mark: Build TimeHACK AppVeyor-199 completed (commit https://github.com/TimeHACKDevs/TimeHACK/commit/19a7a9f170 by @Alex-TIMEHACK)


Reporting an issue with the Architecture Components

Question about Architecture Components-Navigation?
1.Navigation Architecture Component - how to set/change custom back or hamburger icon with navigation controller?
2.Using NavigationView

I'd encourage you to [look through the official issue tracker[(https://issuetracker.google.com/issues?q=componentid:409828) to see if your feature request is already submitted or file a feature request / documentation request.


Set default thumbnails to pull from first page of multipage work

We rely on system-generated thumbnails for all newly-imported works, and we'd like those thumbnails to show the first page of multipage works. For example, if a work contains a 5 page PDF, page 1 would be the thumbnail. Right now the system appears to select a page of the PDF at random or from the end of the file, which leads to situations where the representative image for a work is confusing and lacks context. See example screenshots below of books in the catalog search that show the final page of the PDF as the thumbnail.

Links to the works shown above:

https://adl.b2.adventistdigitallibrary.org/concern/published_works/22266364_cardinal_1931?locale=en
https://adl.b2.adventistdigitallibrary.org/concern/published_works/22266362_cardinal_1930?locale=en
https://adl.b2.adventistdigitallibrary.org/concern/published_works/22266378_cardinal_1939?locale=en
https://adl.b2.adventistdigitallibrary.org/concern/published_works/22266368_cardinal_1933?locale=en

we completed similar work for Louisville

We can make a rake task to clean up after the fact. Marking this as lower priority.

This seems to be working after the cutover. See screenshot of items uploaded today (11/26/24).


No high coverage kmer found

the program was working well but today I reinstalled on a different machine and whenever I run it on fastq samples, I got this "NO high coverage Kmer found"
previously on the same samples, the program was giving positive results

Can you please give me your command.
And please also submit the HTML result if applicable.

./fastv -i ncbi/public/sra/SRR11092056.fastq -g SARS-CoV-2.genomes.fa -c SARS-CoV-2.kmer.fa
fastv report at 2020-05-22 22:25:00.pdf

You used wrong command, please use:
./fastv -i ncbi/public/sra/SRR11092056.fastq -g SARS-CoV-2.genomes.fa -k SARS-CoV-2.kmer.fa

The option -c is for kmer collection file.

I am so sorry, I replaced the kmer with kmer collection.
ok, regarding screening for all viruses, can you tell me which file is kmer and which is genome?
You can download k-mer files and genomes files of viruses from http://opengene.org/uniquekmer/viral/index.html. This is generated by extracting unique k-mers for all genomes in a big FASTA (http://opengene.org/viral.genomic.fasta), which contains all NCBI complete RefSeq release of viral sequences that can be found from https://ftp.ncbi.nlm.nih.gov/refseq/release/viral/.
regarding the last link is not loading

1, download the k-mer collection file for all viruses in RefSeq: http://opengene.org/viral.kc.fasta.gz
2, use following command:
./fastv -i your.fq.gz -k viral.kc.fasta.gz

If you want to pay particular attention to SARS-CoV-2 while scanning for all viruses, use:
./fastv -i your.fq.gz -c viral.kc.fasta.gz -g SARS-CoV-2.genomes.fa -k SARS-CoV-2.kmer.fa

Awesome!, what is the difference between both commands in term of output and speed?

From: Shifu Chen<EMAIL_ADDRESS>Sent: Saturday, May 23, 2020 1:39:54 AM
To: OpenGene/fastv<EMAIL_ADDRESS>Cc: Abouelkhair, Mohamed<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [OpenGene/fastv] No high coverage kmer found (#7)
[External Email]
If you want to pay particular attention to SARS-CoV-2 while scanning for all viruses, use:
./fastv -i your.fq.gz -c viral.kc.fasta.gz -g SARS-CoV-2.genomes.fa -k SARS-CoV-2.kmer.fa
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/OpenGene/fastv/issues/7#issuecomment-632989357, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK7NPHQGZR4TYCEYBNHZGODRS5OSVANCNFSM4NIJMO5A.

so I run the first command and I got the same message "no high coverage" and sequence and quality have different length!
and then in the HTML, I found a list of viruses, phage and bacteria even I screened it for viruses only.
So my question, should I neglect the message "no high coverage and sequence length " and what is the cutoff for considering these hits(viruses, phage and bacteria)?
fastv report at 2020-05-23 00_46_19.pdf

It depends.
Normally viruses with 90%+ coverage are with high confidence. So in your case, the sample was with SARS-CoV-2 and Influenza A

great, thank you so much

Awesome!, what is the difference between both commands in term of output and speed?
…

1, Output: the longer command provides two more sections: SARS-CoV-2 detection and SRS-CoV-2 genome coverage. The rest parts are same.
2, Speed: the longer command is a little slower, but still very fast.

I tried different ways to make it run of multiple input files at a time but I could not, Can you guide me how to do that?

Use shell to do this. I cannot write the code for you since I don't have enough bandwidth. But it's not difficult.

Got it!, I figured it out. I am using it now with a good code on multiple files.
Thank you so much

Hi Dr. Chen,
Please can you check your e-mail?


[Bug] empty white page

Description of the bug
Hello, the webpage loads a white background and nothing else.
I have tried with firefox and edge and obtained the same result, also in private/incognito mode.
Sometimes the real background of dash. seems to get loaded as well as some widgets, but then it all gets overwritten by a white page.
The source says that I need to enable javascript but I don't think that's the problem, I can load each and every page in the web without any issue, so I think it is already enabled.
I have tried every version from 3.9.7 to the latest one
How to reproduce
start an instance with
DASHDOT_WIDGET_LIST: os,cpu,storage,ram,network,gpu
Relevant log output
logs do not show anything about this

Info output of dashdot cli
yarn run v1.22.19
$ node dist/apps/cli/main.js info
INFO
=========
Yarn: 1.22.19
Node: v18.9.0
Dash: 4.5.2

Cwd: /app
Hash: afb66c4801e68076ad4562ba67636f05120f9e9a
In Docker: true
In Podman: false
In Docker (env): true
Done in 0.29s.

What browsers are you seeing the problem on?
Firefox, Microsoft Edge
Where is your instance running?
Linux Server
Additional context
without the gpu widget everything works correctly
I have an NVIDIA GTX970

Have you tried to use a chromium based browser? Thats what I tried using and it fixed it.

Hello there, sorry for the late answer. You are running in a docker container, which does not officially support GPU for now. You can try running from host, or start with the guide here to get started with running inside docker.


Optimize topicpull processing by skipping resource-only

Description
Updates topicpull to skip processing resource-only topics / topics that are treated as resource-only.
None of the default processing in DITA-OT needs to run topicpull on files that are explicitly resource-only. The topics will not be published; topicpull comes last in the preprocessing pipelines, so for normal cases, we are done with those topics at the end of preprocess. Skipping topicpull on those topics optimizes processing for builds with a large number of resource-only topics, and also avoids errors for the condition described below.
Motivation and Context
For all of our builds we set onlytopic.in.map=true by default. This parameter has a few problems around the edge cases, but many of the problems have been fixed in the last couple releases. One thing it does well is handle the condition where a.dita has a link to b.dita (where B is not in the map), and b.dita has unrelated cross references to other topics; the early stages of processing no longer try to follow those cross references, and the link targets are not copied to temp even when present.
However, the resource-only topics (with all other topics) are tossed into topicpull. When the resource-only file b.dita runs, it attempts to pull link text for cross references that will never be used. When those files were not pulled into temp to begin with, this results in Java errors and topicpull errors. For example:
[topicpull] I/O error reported by XML parser processing file:/C:/DITA-OT/TESTGITHUB/topicpull/temp/linkFromResourceOnly-NotNeeded.dita: C:\DITA-OT\TESTGITHUB\topicpull\temp31\linkFromResourceOnly-NotNeeded.dita (The system cannot find the file specified.)
[topicpull] file:/C:/DITA-OT/TESTGITHUB/topicpull/resourceonly.dita:6:77: [DOTX056W][WARN]: The file 'linkFromResourceOnly-NotNeeded.dita' is not available to resolve link information.

When onlytopic.in.map=true, files that are only link targets (not in the map) are treated as resource-only along with any topics that the map explicitly marks resource-only. In this case, there is no need to run topicpull on any of those files. Using the sample above, there is a potential edge case where b.dita has a cross reference within the title, so that link text must be resolved in order to establish link text for links to b.dita. This is already broken today, and should probably be handled by adding that link target to the map as resource-only.
Otherwise, in the scenario above, both a.dita and b.dita will generate output; neither is marked as resource-only, so all links are followed, and topicpull runs on each. So for that condition, there is no change to today's behavior.

How Has This Been Tested?
Created sample files to produce the error condition; this is fixed with the update.
Two sample sets in our integrationTest were affected by this update. In each case, we were doing extra (unnecessary) processing on topics that were link or conref targets; those topics were not rendered. So, it is safe to update those sample sets so that the resource-only topics are left with unresolved cross references.
Type of Changes
Blurs the line between bug fix (it removes an invalid error we have today) and breaking change (feature that optimizes processing while producing different results within some temporary artifacts).
Checklist

My code follows the code style of this project.

https://github.com/dita-ot/dita-ot/wiki/Java-Coding-Conventions
https://github.com/dita-ot/dita-ot/wiki/XSLT-Coding-Conventions

I have updated the unit tests to reflect the changes in my code.

I'm a bit worried this might break something we haven't thought of, but this is easy to revert.


Use environment variables for passwords instead of mounting as a file

Done here: https://github.com/stardog-union/helm-charts/pull/107


Display corruption in Blue Archive

[x] I have read the FAQ.
[x] I have searched in existing issues.

Environment

OS: Windows 10
scrcpy version: 1.20
installation method: Windows release
device model: ROG Phone 3
Android version: 11

Describe the bug
Display corruption happening while playing the game Blue Archive. This does not happen while not connected via scrcpy.
Examples: (seizure warning)
https://drive.google.com/file/d/1kP0qu7XYKhp0tchirLHxoQq588FR6TDC/view?usp=sharing
https://drive.google.com/file/d/1Y6qUO6wW9hdNoXkH2ALvYc_UQ2tb5JLO/view?usp=sharing
On errors, please provide the output of the console (and adb logcat if relevant)
https://gist.github.com/Renari/26495b59559606ddd89562e8dc935328

Looks like a device/encoder issue.
The game and the encoder probably share some hardware resource, and it is not managed correctly.
Does it also happen with screenrecord:
adb shell screenrecord /sdcard/file.mp4
adb pull /sdcard/file.mp4

?

Yeah it seems to happen with built in recording functionality as well, basically if anything is recording the screen it happens, I'm not sure scrcpy can do anything at this point.
The issue is replicated on two devices (S10, ROG Phone 3) and without scrcpy via using the Samsung Record feature and the Armory Crate record feature on both phones.

Going to go ahead and close this since it appears not to be a scrcpy issue.

@Renari Hi, I seem to be experiencing this problem, have you found a solution?
https://user-images.githubusercontent.com/8396456/152275859-59819873-8583-4945-97ad-2e40287d0e9a.mp4

ASUS released a patch for my device and that game specifically.


Problem on re-deployment

We are trying to re-deploy and received the following error in step 6: “ERROR: {“error”:{“code”:”DeploymentNotFound”,”message”:”Deployment ‘Deployment-10-04-2023-11-46-38’ could not be found.”}}
I noticed that the Cleanup.ps1 states: 'Please use a different ResourceGroup name on re-deployment!'. Am I correct to assume that we should modify the $ResourceGroupName param in the Deploy.ps1 to be able to successfully re-deploy?

Hi simply ensure you remove all the resources and resource group and ensure you purge the keyvault if rerunning the setup please also ensure your using the specific version of the Azure CLI on the install to ensure no errors.


How to remove both filters to use just custom filters

consumed using: Laravel 5.3 (webpack)

In the options i can toggle with filterByColumn: true|false,
but what i need is none of them because i will use just custom filters.
Regards,
boing6000

i've updated the code. Download version 1.5.3, and then set filterable to false. This will remove the filter.


NPE when saving CassandraRow with null values

14/11/24 00:30:26 WARN TaskSetManager: Lost task 0.0 in stage 129.0 (TID 180, <IP_ADDRESS>): java.lang.NullPointerException: Unexpected null value of column 1. Use get[Option[...]] to receive null values.
        com.datastax.spark.connector.CassandraRow.get(CassandraRow.scala:93)
        com.datastax.spark.connector.writer.GenericRowWriter.getColumnValue(GenericRowWriter.scala:14)
        com.datastax.spark.connector.writer.GenericRowWriter.getColumnValue(GenericRowWriter.scala:7)
        com.datastax.spark.connector.writer.AbstractRowWriter$$anonfun$fillBuffer$1.apply$mcVI$sp(AbstractRowWriter.scala:23)
        scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
        com.datastax.spark.connector.writer.AbstractRowWriter.fillBuffer(AbstractRowWriter.scala:22)
        com.datastax.spark.connector.writer.AbstractRowWriter.bind(AbstractRowWriter.scala:28)
        com.datastax.spark.connector.writer.TableWriter$$anonfun$measureMaxInsertSize$2.apply(TableWriter.scala:90)
        com.datastax.spark.connector.writer.TableWriter$$anonfun$measureMaxInsertSize$2.apply(TableWriter.scala:89)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        com.datastax.spark.connector.writer.TableWriter.measureMaxInsertSize(TableWriter.scala:89)
        com.datastax.spark.connector.writer.TableWriter.com$datastax$spark$connector$writer$TableWriter$$optimumBatchSize(TableWriter.scala:107)
        com.datastax.spark.connector.writer.TableWriter$$anonfun$write$1.apply(TableWriter.scala:132)
        com.datastax.spark.connector.writer.TableWriter$$anonfun$write$1.apply(TableWriter.scala:126)
        com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:98)
        com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:97)
        com.datastax.spark.connector.cql.CassandraConnector.closeResourceAfterUse(CassandraConnector.scala:149)
        com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:97)
        com.datastax.spark.connector.writer.TableWriter.write(TableWriter.scala:126)
        com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:24)
        com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:24)
        org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
        org.apache.spark.scheduler.Task.run(Task.scala:54)
        org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)

Hi -
I was wondering if there is a reason this issue has been marked as closed? There is an "In Progress" tag, so does that mean this issue is being worked on?
[I am using the connector version 1.1.1 and can attest to the issue still existing]

The issue was incorrectly marked as In progress.

It was incorrectly left marked as in progress, despite the fix having been
merged.
If you experience NPE, then this is a new bug. Please open a JIRA.
Thanks,
Piotr
[image: datastax_logo.png] http://www.datastax.com/
PIOTR KOŁACZKOWSKI
Lead Software Engineer, DSE Analytics |<EMAIL_ADDRESS>[image: twitter] https://twitter.com/pkolaczk [image: facebook.png]
https://www.facebook.com/datastax [image: linkedin]
https://www.linkedin.com/in/pkolaczk [image: g+.png]
https://plus.google.com/+Datastax/about
http://feeds.feedburner.com/datastax https://github.com/datastax/
2015-01-05 17:05 GMT+01:00 Sang Venkatraman<EMAIL_ADDRESS>
Hi -
I was wondering if there is a reason this issue has been marked as closed?
There is an "In Progress" tag, so does that mean this issue is being worked
on?
[I am using the connector version 1.1.1 and can attest to the issue still
existing]
—
Reply to this email directly or view it on GitHub
https://github.com/datastax/spark-cassandra-connector/issues/446#issuecomment-68729040
.

Never mind. This issue seems to be fixed -- I was getting NPE's while reading rows of data with null column values but that was because I was not using the get options API.


../fsevents.cc:63:6: warning: field 'async_resource' will be initialized after field 'lockStarted' [-Wreorder]

Feel free to close this issue but wanted to see if there is a reason for this warning.
This might be related to using node v11.6.0?
Here is the output from installing fsevents:
node-pre-gyp WARN Tried to download(404): https://fsevents-binaries.s3-us-west-2.amazonaws.com/v1.2.4/fse-v1.2.4-node-v67-darwin-x64.tar.gz
node-pre-gyp WARN Pre-built binaries not found for<EMAIL_ADDRESS>and<EMAIL_ADDRESS>(node-v67 ABI, unknown) (falling back to source compile with node-gyp)

  SOLINK_MODULE(target) Release/.node
  CXX(target) Release/obj.target/fse/fsevents.o

../fsevents.cc:63:6: warning: field 'async_resource' will be initialized after field 'lockStarted' [-Wreorder]
   : async_resource("fsevents:FSEvents"), lockStarted(false) {
     ^
1 warning generated.

  SOLINK_MODULE(target) Release/fse.node
  COPY ~/Code/GitlabRepos/my-serverless-project/node_modules/fsevents/lib/binding/Release/node-v67-darwin-x64/fse.node
  TOUCH Release/obj.target/action_after_build.stamp

and also:
node-pre-gyp WARN Tried to download(404): https://fsevents-binaries.s3-us-west-2.amazonaws.com/v1.2.4/fse-v1.2.4-node-v67-darwin-x64.tar.gz
node-pre-gyp WARN Pre-built binaries not found for<EMAIL_ADDRESS>and<EMAIL_ADDRESS>(node-v67 ABI, unknown) (falling back to source compile with node-gyp)

  SOLINK_MODULE(target) Release/.node
  CXX(target) Release/obj.target/fse/fsevents.o

In file included from ../fsevents.cc:6:
../../nan/nan.h:1064:44: warning: 'ToString' is deprecated: Use maybe version [-Wdeprecated-declarations]
      v8::Local<v8::String> string = from->ToString(v8::Isolate::GetCurrent());
                                           ^
~/.node-gyp/11.6.0/include/node/v8.h:2537:3: note: 'ToString' has been explicitly marked deprecated here
  V8_DEPRECATED("Use maybe version",
  ^

~/.node-gyp/11.6.0/include/node/v8config.h:326:29: note: expanded from macro 'V8_DEPRECATED'
  declarator __attribute__((deprecated(message)))
                            ^
../fsevents.cc:63:6: warning: field 'async_resource' will be initialized after field 'lockStarted' [-Wreorder]
   : async_resource("fsevents:FSEvents"), lockStarted(false) {
     ^

2 warnings generated.

  SOLINK_MODULE(target) Release/fse.node
  COPY ~/Code/GitlabRepos/my-serverless-project/node_modules/fsevents/lib/binding/Release/node-v67-darwin-x64/fse.node
  TOUCH Release/obj.target/action_after_build.stamp

Node 12 is out and the problem is back. Any guidance?


Guests can't join room guests are allowed in

I've setup my matrix room to allow guests:
#trains_and_things:matrix.org
by checking "Anyone who knows the room's link, including guests" in riot.
Now I setup a guest account and it won't let me join the room.
I'm using the matrix plugin from here: https://gitlab.com/vurpo/godot-matrix, it seems riot.im/app has no guest login so I do not know how to test a guest login outside of the code I've been using to determine if its an issue with Matrix or if there was some API changes that have recently broken the above code.

This doesn't seem to be related to matrix.to. If Riot doesn't let you in to the room as a guest then that is a Riot bug
https://github.com/vector-im/riot-web/issues/new

see https://github.com/vector-im/riot-web/issues/9264


More user friendly defaults

There is a module called tap-spec that prints nicely.
There is also faucet that has more terse defaults.
Currently when you run node test.js tape just spits out TAP to stdout.
What if tape checked whether stdout was a TTY and used faucet or tap-spec instead?
That way if you ever redirect stdout anywhere that's not a TTY like | some thing or > some file or | tee some file you will get the machine readable TAP just like you want.
However if you run node test.js plainly from the terminal you will get a more user friendly default output

It seems like the discussion here ended up being "tape should output TAP, and a piped TAP reporter should be used for anything else", so I think this can be closed. Happy to reopen if not.


Provide Kubeflow PaddleJob support for MultiKueue

What type of PR is this?
/kind feature

What this PR does / why we need it:
The PR introduces a new MultiKueue adapter to handle PaddleJob (Kubeflow).
We want to extend MultiKueue capabilities to satisfy the needs of early adopters.
Which issue(s) this PR fixes:

Relates #2552
Special notes for your reviewer:
Does this PR introduce a user-facing change?

MultiKueue: Support for the Kubeflow PaddleJob

/ok-for-test

/lgtm

@tenzen-y updated ptal


[WIP] AdSizes for AdMob banner ads

This PR adds a new AdSize class to the firebase_admob plugin for use in making banners (see Issue #8098). All the standard sizes are supported (see this AdMob guide for a list).
Right now, the public API and Android implementation are worked out, with iOS and finishing touches to come. I'm putting it up early so people can have a look at the API.

Thanks much @RedBrogdon for the contribution! Looks like there is a small merge conflict with master.  Can you take a quick look?

Please let me know I can be of any help on this?
Also kindly check on the issue https://github.com/flutter/flutter/issues/14222 if you can help on this.

The API is finished at this point, and it's implemented all the way down for iOS and Android. Tests are passing, and the code's linted.
PTAL!

Hope it's merged soon :)

@Purus, to answer your earlier question about what you might do to help: please give this a thorough test in whatever apps you're working on. It's playing along just fine for me, but I'm sure there are edge cases I haven't discovered.

Sure.. I am happy to test it in my development apps. I will test once this is merged.

When is this one likely to get merged?


Clusterloader2: Change the default IncludeFile path

This PR changes the defaultIncludeFilepath prefix from the clusterloader binary location to the clusterloader source path, which logically makes more sense.  It also checks if the IncludeFile path in the manifest file is an absolute path or not. If it's an absolute path, the default path prefix will not be joined.

/assign @mm4tt

/lgtm
/approve