Improve logging on build failure by displaying file contents of meson-log.txt

The current output from pip when encountering a build error is very poor. gh-231 will fix one part of that, by reducing the traceback of irrelevant pip/meson-python internals. There is a second part however which we need to improve on, which is showing the actual root cause of a build failure when possible. Now what happens is the user sees something like:
../../scipy/meson.build:40:0: ERROR: Command "/usr/bin/python3 -c import os; os.chdir(".."); import numpy; print(numpy.get_include())" failed with status 1.

A full log can be found at /tmp/pip-install-3okqoo15/scipy_a5afe2fae16c4250875f20d9db43706f/.mesonpy-u88p7q3f/build/meson-logs/meson-log.txt

and is then left wondering "why does import numpy fail, given numpy is a build dependency?". The answer is in meson-log.txt, which pip has unhelpfully already cleaned up. So it's impossible to diagnose the issue (latest example for scipy here).
Two options to improve here that I can think of:

Don't clean up the log file
Print to log file contents to stdout (possibly followed by repeating the root cause from the original stdout content)

(2) is probably better than (1).

Follow-up: the way to not clean up the build dir (and hence the log file) is:
pip install . --config-settings=builddir=build

I'm not sure that that is guaranteed to stay working, it seems like a side-effect of the way Project.with_temp_working_dir is implemented. And it's actually meson-python that cleans things up; pip's --no-clean works as advertised for the build env itself.

And it's actually meson-python that cleans things up; pip's --no-clean works as advertised for the build env itself.

We should consider this a bug, I think. There's no reason for us to clean things up on errors.

I think we should do both avoid cleaning up ourselves and improve terminal output. Right now I think pip will still clean up for a regular isolated build, so the user trying to diagnose a build failure still has a problem.

I had a look at this, it's a little complex with multiple nested context managers and a subprocess, so let me add my thoughts here first. This is the code in question that yields a new Project instance in a tmpdir:
https://github.com/mesonbuild/meson-python/blob/eacd23a31bf95036851ff3f3b6b7905ed1577394/mesonpy/__init__.py#L827-L829
That'll be something like .mesonpy-2tl1hhkq inside the current source dir. I'm thinking:

the build front-end is responsible for creating a tmpdir and cleaning it up (or not, if the user uses --no-clean)
in meson-python, we should remove any use of tempfile (which is only the lines above)
what we want to do instead I think is create a regular directory with the same scheme (.mesonpy-<random-string-to-avoid-collisions>.

That would fix half of this issue. The other half (figuring out if and when to access meson-log.txt) is more tricky, so I'd start with not cleaning up here to make the problem easier to diagnose in local builds.

I think there is some confusion on what is the role of the different directories. There are four relevant directories: the source directory, the build directory, the artifacts directory, the isolated build environment.
A good starting point to look at this is the PEP517 hook functions https://peps.python.org/pep-0517/#mandatory-hooks Let's take build_wheel() an example. The signature of this function is build_wheel(wheel_directory, config_settings=None, metadata_directory=None). The wheel_directory is the build artifacts directory, where the wheel should be placed. Clearly this should survive the build, thus is not a temporary build location. config_settings is a dictionary with the user settings. No directories there. metadata_directory is optional and for simplicity we can ignore it.
So the information that the backend explicitly gets from the frontend is only the artifacts directory. PEP 517 is not explicit on this, but it is assumed that the current working directory when the hooks are invoked is the source directory. The isolated build environment is the the Python environment executing the hook. There is no facility to pass a build directory.
I haven't found much information in the pip documentation, but looking at the pip source code, the --no-clean option refers to the build directories used by pip to download dependencies and eventually unpack sdists, not to build directories passed to the build backend. Indeed, pypa/build does not have an equivalent command line option.
The current meson-python behavior of creating a temporary build directory and cleaning it up on success is the current one.
I'm not sure that not cleaning up the temporary build directory on error is a good idea. I expect this to result in the users ending up with multiple .mesonpy-$random directories, without a clear way to distinguish them. I think that outputing the whole of meson-log.txt on error is not a great idea either. It can be very long and most often the cause for meson setup failing is easy to identify in the regular meson setup output. I would place that behind an option for extra verbose output. However, if it is behind an option, it would require the user to run the build again to get the verbose output. If this is is the case, the user can simply pass -Cbuilddir=foo and then do cat foo/meson-logs/meson-log.txt.
Why do not show meson-log.txt by default? I don't think it is useful in most circumstances and is an overwhelming amount of output. I think that the most common meson setup failures are errors in meson.build or missing dependencies. The content of meson-log.txt is not useful for diagnosing the first king of errors, and only seldomly useful for diagnosing the latter.

the --no-clean option refers to the build directories used by pip to download dependencies and eventually unpack sdists, not to build directories passed to the build backend.

Your description of the various directories sounds right to me. I believe --no-clean applies to downloaded sources from the package that is being built too - and in those cases, since the build starts from within that source dir, it therefore applies to the build dir as well.

I'm not sure that not cleaning up the temporary build directory on error is a good idea. I expect this to result in the users ending up with multiple .mesonpy-$random directories, without a clear way to distinguish them.

That is fair. I don't think we necessarily need the $random bit though. What setup.py based builds do is simply not use a tmpdir and create a dir named build inside the directory in which the build is invoked. This is simply left there. I don't think there's much wrong with that. We could do the same with our new default naming scheme (build/<abi-tag>). Then what happens is:

the frontend is responsible for cleaning up when sources are downloaded from PyPI or GitHub directly (and --no-clean will then preserve source and build dirs)
for a local build (e.g., pip wheel .) the default build dir remains available.

Why do not show meson-log.txt by default? I don't think it is useful in most circumstances and is an overwhelming amount of output.

Yes, I agree with this. I left it alone for now because this is clearly going to be confusing. Ideally I'd like to show only relevant content (e.g. last N lines of meson-log.txt if and only if the answer is hidden in there), but it's unclear if that's achievable. Let's pause on this part - the build-dir cleanup part is better to deal with first.

Your description of the various directories sounds right to me.

It is not completely right, though. When I wrote it I was thinking of the case of installing a local package, not an sdist downloaded from PyPI. In that case indeed the build directory created by meson-python is in the source directory that gets cleaned up by pip.
I'm not sure not cleaning up the build directory and leaving it to the build front-end to do that is of much help. How obvious is where the build directory is located when python -m pip install --no-clean is invoked? I just tried it and I've absolutely no clue of where to look for the build directory. The only way to discover it is to increase the verbosity of pip and look for the compilation log.
python -m pip install --no-clean --no-binary=siphash24 siphash24 --force-reinstall

results in a source directory located in /private/var/folders/6f/4l4d5n9x1y71cxyrprf_x35c0000gn/T//pip-install-qarix6w3/siphash24_e9768ce23fbc4806b9132cf1687685c8. macOS uses funny looking temporary directories, but even the part that is solely under the control of pip is hard to make sense of.
On the other hand, I can pass an explicit build directory
python -m pip install --config-settings=builddir=$(pwd)/build --no-binary=siphash24 siphash24 --force-reinstall

and the build directory and the build log are exactly where I want. However, this does not work when installing more than one package as the builddir=$(pwd)/build option cannot be assumed to be understood by the build system of every package, and similarly when the package to debug is a dependency of the package that is actually being installed. Both, however, can be simplified to the case of a single package to be installed, without any loss, I think.

How obvious is where the build directory is located when python -m pip install --no-clean is invoked? I just tried it and I've absolutely no clue of where to look for the build directory

You will when the build fails, because Meson gives you the full path to the meson-log.txt file that may contain the clue to your build failure. And that's the point here - you just copy the path from Meson's terminal output, and the file will actually exist.

Indeed, that's right. The output looks something like this with pip:
$ python -m pip wheel .
Processing /Users/daniele/src/meson-python/tests/packages/complex
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [17 lines of output]
      The Meson build system
      Version: 1.1.0
      Source dir: /Users/daniele/src/meson-python/tests/packages/complex
      Build dir: /Users/daniele/src/meson-python/tests/packages/complex/.mesonpy-4gnlztqx/build
      Build type: native build
      Project name: complex
      Project version: 1.0.0
      C compiler for the host machine: ccache cc (clang 14.0.0 "Apple clang version 14.0.0 (clang-14<IP_ADDRESS>)")
      C linker for the host machine: cc ld64 820.1
      Cython compiler for the host machine: cython (cython 0.29.34)
      Host machine cpu family: x86_64
      Host machine cpu: x86_64

      ../../meson.build:11:0: ERROR: Problem encountered: error

      A full log can be found at /Users/daniele/src/meson-python/tests/packages/complex/.mesonpy-4gnlztqx/build/meson-logs/meson-log.txt
      + meson setup /Users/daniele/src/meson-python/tests/packages/complex /Users/daniele/src/meson-python/tests/packages/complex/.mesonpy-4gnlztqx/build -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/Users/daniele/src/meson-python/tests/packages/complex/.mesonpy-4gnlztqx/build/meson-python-native-file.ini
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

I don't know why the line indicating the command that is being executed comes after the output of the command. I think having it before the output makes the log easier to understand. Adding flush=True to the print() call achieves just that. I'll prepare a PR.
I like the fact that meson-python does not leave behind a build directory. However, I appreciate that being able to get to the build log is important. However, to be able to get to the Meson log file when not installing a package from local sources, the --no-clean flag needs to be passed to pip. If a flag needs to be passed, is having to pass the build directory explicitly a worst solution?
I'm ok with using a build/$abi/ directory but this makes the issue of what to do if the directory already exist much more important: before this could only happen if the user specified a build directory, after this change it would be a much more common occurrence. Another issue is whether we need to detect when meson-python is trying to use a build directory that was not created by meson-python itself.
What do other build backend do? I wish the PEPs provided some guidelines for this.

However, to be able to get to the Meson log file when not installing a package from local sources, the --no-clean flag needs to be passed to pip. If a flag needs to be passed, is having to pass the build directory explicitly a worst solution?

Maybe not all that different indeed. It's better if both worked, because I suspect quite a few people will retry with --no-clean rather than read our docs. The "from local sources case has a bigger difference though: there pip does not clean up, so the link to the log file would be there if we'd also avoid cleaning up.

I'm ok with using a build/$abi/ directory but this makes the issue of what to do if the directory already exist much more important:

Yes, that is true. It may actually be a good thing? Say you do local development first, and then a pip install .. That'd work well with caching and avoid rebuilding from scratch.

What do other build backend do?

setuptools writes to build/. not sure what other build backends do.

If we use a stable build directory, I am not sure about what should happen when a package is build twice with different configurations (either meson arguments passed via the command line or via pyproject.toml). Currently, if we detect that the build directory is a valid build directory, we just issue a meson setup --reconfigure and apply the settings. This is done on the assumption that this happens only when the user explicitly specifies a build directory, thus they most likely know what they are doing. However, if a stable build directory is used, also unaware users may be surprised by the behavior. For example:
python -m build -w . -Csetup-args=-Ddebug=true

would generate a wheel with debug symbols. Then the user changes his mind and wants a regular wheel. Executing
python -m build -w .

would still generate a wheel with debug symbols, because the -Ddebug=true is already set in the build directory and meson setup --reconfigure changes only the options that are explicitly set. At least build shows the compilation log, thus there is a small chance that the user notices this. However, pip does not show the compilation log, unless the build fails, thus this is hard to notice.

For CI purposes (and particularly cibuildwheel, which doesn't really provide options to control what happens after a failed build) it would be useful to have an option (maybe passable via config-settings) to print out the meson log on failure. That'll be much easier to use than having to write a wrapper script that parses the error message to figure out where the build log got written (or whether it got written) and dumping it.

A config settings option to dump the meson log could be an idea. How should such an option be named

How about 'print-log', with options 'none' (default), 'on-error' and 'always'? One downside is that it might not be clear that this isn't logs from the actual compilations; so maybe 'print-configure-log'.
That could be a useful addition to mesonbuild itself rather than just for meson-py, in which case it would be passed as -Csetup-args=--print-log=on-error.

One issue is that the detailed setup log can be many thousands of line long and is most often not necessary to debug meson setup failures. On the contrary, showing the detailed log, would most likely make the reason for the failure much more difficult to spot, unless the user scrolls back to the regular meson output.  The problem arises in CI environment where having access to anything else than the commands output is more complex.
However, if you want to debug a meson setup failure in CI, it is already straightforward to get the log:
python -m build -w . -Cbuilddir=build || cat build/meson-logs/meson-log.txt

cibuildwheel AFAIK does not allow to customize the command used to build the package, or to execute a command on failure. Maybe a CIBW_ON_FAILURE_COMMAND configuration entry could be added? @henryiii

Another aspect to consider is that having a print-log option requires running the setup stage one more time to get the log after the user encounters a failure. While having the log available on the filesystem for inspection does not require running the build again, or knowing about the setting.

However, if you want to debug a meson setup failure in CI, it is already straightforward to get the log:

This is useful I think, we should probably add that to the docs. In GitHub Actions jobs, having this log as a separate job step so it doesn't interfere with reading the regular output that came before is useful:
    - name: Show meson-log.txt
      if: always()
      run: cat build/meson-logs/meson-log.txt

In Github Actions, there is also a way to tag output in a way that it is hidden by default, and you need to click on the small arrow to display it https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#grouping-log-lines

Good point, I like how cibuildwheel uses that feature. It may be tricky to echo the log file after the build step though - it may not be run if the step has "halt on failure" behavior, or if it does get run it may change the exit status back to 0.
Once we figure out a nice way of showing the log here, it's something we may want to add to the docs.

In Github Actions, there is also a way to tag output in a way that it is hidden by default, and you need to click on the small arrow to display it

Update: we've got this implemented in NumPy now (example log). While it's potentially helpful, the meson-log.txt output is ~10,000 lines long and even when it's hidden it causes the pytest output in the next foldout to become really slow to scroll. So I'm not 100% happy with that solution yet. We'll probably move it back to its own job step.

Suggestion by @oscarbenjamin in https://github.com/mesonbuild/meson-python/discussions/602#discussioncomment-8870523:
[what] would be very helpful in this situation is an easy way to get it to show the full logs like a MESON_PYTHON_SHOW_LOGS environment variable

This is even more annoying if you're using something else that directly invokes pip internally and you don't even have the option to specify a build-dir

@oddbookworm note that every pip option can be specified as an environment variable for exactly this reason: https://pip.pypa.io/en/stable/topics/configuration/#environment-variables. So for a --config-settings flag, you can set PIP_CONFIG_SETTINGS instead (documented at https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-C).


Kernels 2022-01-29

Things done

[x] Fits CONTRIBUTING.md.

Probably worth cherry-picking a security fix: https://www.openwall.com/lists/oss-security/2022/01/29/1


build libtensorflow-core.a error

buf -lpthread -lm -lz
Undefined symbols for architecture x86_64:
"nsync::nsync_mu_init(nsync::nsync_mu_s_)", referenced from:
tensorflow::mutex::mutex() in env.o
tensorflow::mutex::mutex() in random.o
"nsync::nsync_mu_lock(nsync::nsync_mu_s_)", referenced from:
tensorflow::mutex::lock() in env.o
tensorflow::mutex::lock() in random.o
tensorflow::mutex::lock() in histogram.o
"nsync::nsync_mu_unlock(nsync::nsync_mu_s_*)", referenced from:
tensorflow::mutex::unlock() in env.o
tensorflow::mutex::unlock() in random.o
tensorflow::mutex::unlock() in histogram.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [/Users/9kacha/Desktop/TFFF/tensorflow/tensorflow/contrib/makefile/gen/host_bin/proto_text] Error 1
'[' 2 -ne 0 ']'
echo 'armv7 compilation failed.'
armv7 compilation failed.
exit 1

how to fix this error ?

rename ./tensorflow/contrib/makefile/downloads/nsync/builds/default.linux.c++11/nsync.a to libnsync.a  and then modify makefile  -L$(DOWNLOADSDIR)/nsync/builds/default.linux.c++11/   -lnsync

@chinaluffy did that fix your problem?

@liyungithub I'm having the same issue. I can't figure out what you mean by "makefile -L$(DOWNLOADSDIR)/nsync/builds/default.linux.c++11/ -lnsync"

I saw this and did the following:
./tensorflow/contrib/makefile/compile_nsync.sh
Then added: HOST_NSYNC_LIB=pwd/tensorflow/contrib/makefile/downloads/nsync/builds/default.linux.c++11/nsync.a at the end of my make command.

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue? Please update the label and/or status accordingly.

Nagging Awaiting Response: It has been 14 days with no activityand the awaiting response label was assigned. Is this still an issue?

Nagging Awaiting Response: It has been 14 days with no activityand the awaiting response label was assigned. Is this still an issue?

Also having this problem I cannot build the nsync lib with compile_nsync.sh via build_all_android.sh. compile_android_protobuf.sh fails also.
My current error is just failing to find basic headers like string.h and stdlib.h. I believe it has something to do with using the r16 NDK, have the header paths changed? I started trying to add paths to the build scripts but am unsure if this will lead to a solution.

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

It has been 29 days with no activity and the awaiting response label was assigned. Is this still an issue?

It has been 66 days with no activity and the awaiting response label was assigned. Is this still an issue?

We are closing this issue for now due to lack of activity. Please comment if this is still an issue for you. Thanks!


Avoid multiple copies of binary data during the conversion from Result to Row

Hi, @dongjoon-hyun Could you help review this? Thanks.

LGTM!

Just a quick question - why not just update Utils.hbaseFieldToScalaType directly?

@nickdevp Utils.hbaseFieldToScalaType is also invoked in [parseRowKey](https://github.com/hortonworks-spark/shc/blob/master/src/main/scala/org/apache/spark/sql/execution/datasources/hbase/HBaseTableScan.scala#L90
https://github.com/hortonworks-spark/shc/blob/master/src/main/scala/org/apache/spark/sql/execution/datasources/hbase/HBaseTableScan.scala#L90), where row is used multiple times and can not be used as the return value of 'Utils.hbaseFieldToScalaType' directly.

Hi, @nickdevp Could you please give your feedback about this PR? If this PR looks good for you, I will merge it.

The code changes do fix the issue as I've described it. However, from a design point of view, you now have at least 2 places which need to be changed if the logic for converting a spark binary field to a scala type needs to be altered.
As an alternate solution, I would probably add an argument to Utils.hbaseFieldToScalaField that specifies whether the data needs to be copied.

@nickdevp I think for the current PR we chose to follow conventions for similar stuff in Spark. Opening a follow up issue to address the maintainability issue you raised. Its a valid concern.


node-red-contrib-persistent-global-context moduel not found

I have installed the node via the palette and it appears to be successful, no errors reported.
I then trid to import the example but received an error 'unknown node'.
I looked at the node in the palette and it says 'module_not_found'

I think it's the backslashes in the package.json and one in the GlobalContext.json string in the .js file. They can be swapped out for single forward slashes.
To test you can download the zip and edit / install it by hand:
The two lines in package.json become:
"main": "src/persistent-global-context.js"
and
"persistent-global-context": "src/persistent-global-context.js"
and in src/persistent-global-context.js:
const path = RED.settings.userDir + '/GlobalContext.json';
then in your .node-red folder, run npm install <path_to_pkg>, restart node-red, and it should show up just fine. And it DOES work! (thx!)

Fixed and published the node. Thank you for pointing out the cause of the error @noeldodd. 👍


Cow: Watch petbaggagepolicy.com

Cow requests the watch of the watch_keyword petbaggagepolicy\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

petbaggagepolicy\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by Nick in Charcoal HQ


ophyd master pyepics control layer get_pv PV reuse needs work

The path forward will likely be one of the following, in order of preference:

Fixing pyepics epics.PV (another PR) to more easily handle running callbacks on already-created instances
Making pyepics_shim.PV rely only on pyepics epics.ca (i.e., not the epics.PV class)
Vendoring epics.PV and making it ophyd-specific

Problem area:
https://github.com/NSLS-II/ophyd/blob/1ca2a260e990fe406d718f68f3738cf9d1085ce3/ophyd/_pyepics_shim.py#L173-L191

Closed by #720


Update PyTorch Benchmarks

Re-enables PyTorch BertLarge FP32 models.
Removed all FP16 models since we're focusing more on integer/sub-byte inference.
Removed PyTorch ResNet models since there is an error generating the mlir and we are now focusing on LLMs.
Removed PyTorch EfficientNet models because there is no script to regenerate them and LLM focus.

This PR doesn't make changes to MiniLM so the 20% regression is likely noise.


Does not work in iOS Chrome

Not working for iOS Chrome 74.0.3729.155.
This is because window.innerHeight has not been updated when an orientationchange event occurs. There is a workaround below, but I'd like to see it implemented in the library.
Workaround:
var test = vhCheck({
  onUpdate: (obj: any) => {
    orientationChanged().then(() => {
      const cssVarName = "vh-offset";
      const sizes = my.vhCheck.recompute();
      document.documentElement.style.setProperty(`--${cssVarName}`, `${sizes.value}px`);
    });
  }
});

See this for the orientationChanged function: https://stackoverflow.com/a/44579732

@apptaro thanks for the code.
I'll be glad to see that in the code but I don't have the time right now to do it.
Can you come up with a PR?


[DOCS] reencode_video clarifications

URL(s) with the issue
https://docs.voxel51.com/api/fiftyone.utils.video.html#fiftyone.utils.video.reencode_video
Description of proposal (what needs changing)
According to the documentation, **kwargs passed to the fiftyone.utils.video.reencode_video represent "keyword arguments for eta.core.video.FFmpeg(**kwargs). However, under the hood, reencode_video calls _transform_video, with parameters reencode=True, force_reencode=True,, which in turn disregard the parameters passed through **kwargs. I'm not sure whether this behavior is ill-documented, or if this is a bug. It seems however that without the default parameters used for reencoding the videos are not displayed within the app. Some feedback on this matter would be greatly appreciated.
Willingness to contribute
The FiftyOne Community encourages documentation contributions. Would you or another member of your organization be willing to contribute a fix for this documentation issue to the FiftyOne codebase?

[ ] Yes. I can contribute a documentation fix independently
[ ] Yes. I would be willing to contribute a documentation fix with guidance from the FiftyOne community
[x] No. I cannot contribute a documentation fix at this time

Hi @apourchot 👋
I can see how this is slightly confusing. Let me explain ourselves and we'll see what you think!
The reencode_video() and reencode_videos() methods were really intended to only be used in "easy mode": they hard code H.264 MP4 as the codec, as that is a common codec that all browsers support and thus the FiftyOne App will reliably be able to render videos in this format. **kwargs should probably just be removed from that method.
The transform_video() and transform_videos() methods are more general versions of reencode_video() and reencode_videos() that do actually pass any **kwargs that you provide (namely in_opts and out_opts) directly through to eta.core.video.FFmpeg, allowing you to have fine-grained control over the ffmpeg commands that are ultimately executed.

Thanks @brimoor, this clarifies things a bit :smile:
I'm still confused as to which ffmpeg options are supported by fiftyone. I don't think that this is documented anywhere. I'm working with medical data and trying to display videos with the highest quality possible, but using simple presets like VIDEO_ENCODING_OPTS = [     "-c:v",     "libx264",     "-preset",     "slow",     "-crf",     "18",     "-profile:v",     "high", ] will generate videos which are fine on their own, but are not displayed within the app.
I realize this might be beyond the scope of this issue but any knowledge that you could share on this matter would be greatly appreciated.

FiftyOne just relies on the standard HTML5 <video> tag to render videos in the App, which implicitly relies on your web browser's ability to render the video you've provided.
It is a bit surprising that the VIDEO_ENCODING_OPTS you mention aren't renderable in the browser, because in general H.264 seems pretty stable in browsers, but certainly many non-H.264 codecs aren't supported, so I'm not that surprised.
Just to clarify, you have some video, say /path/to/video.mp4, that you can view using something like quicktime, but when you do this, the video doesn't render in the FiftyOne App?
import fiftyone as fo

dataset = fo.Dataset()
dataset.add_sample(fo.Sample(filepath="/path/to/video.mp4"))

session = fo.launch_app(dataset)

Yes, I can visualize them fine using any standard video player, but the app won't display them, saying "This video failed to load. The file may not exist, or its type (video/mp4) may be unsupported. You can use fiftyone.utils.video.reencode_videos() to re-encode videos in a supported format.". Here is an example of such a video.

Yeah sorry, there's just something unsupported about that codec by your (and my) browser
import fiftyone as fo
import fiftyone.utils.video as fouv

sample = fo.Sample(filepath="/Users/Brian/Desktop/unsupported.mp4")

dataset = fo.Dataset()
dataset.add_sample(sample)

# video doesn't load
session = fo.launch_app(dataset)

fouv.reencode_videos(dataset)

# now it does
session.refresh()


[Bug report] index.md and README.md conflict

Bug report
Description

When index.md and README.md both exist, the index page rendered by vuepress is different in Windows and macOS (Linux).
Windows render the README.md as the index page, but macOS use index.md.
I think index.md should have a higher priority than README.md?
Steps to reproduce
Follow this guide.
Then
echo '# README' > docs/README.md
echo '# Index' > docs/index.md

Reproduction link / repo: https://github.com/yjl9903/vuepress-index

Environment info
Windows

Browser: Edge 100.0.1185.44
Output of vuepress info:

# Paste output of `vuepress info` here

  System:
    OS: Windows 10 10.0.19043
    CPU: (16) x64 11th Gen Intel(R) Core(TM) i7-11700 @ 2.50GHz
    Memory: 26.52 GB / 31.90 GB
  Binaries:
    Node: 16.14.2 - ~\AppData\Local\Volta\tools\image\node\16.14.2\node.EXE
    Yarn: Not Found
    npm: 8.5.0 - ~\AppData\Local\Volta\tools\image\node\16.14.2\npm.CMD
  Utilities:
    Git: 2.35.3.
  Browsers:
    Chrome: Not Found
    Edge: Spartan (44.19041.1266.0), Chromium (100.0.1185.44)
  npmPackages:
    @vuepress/bundler-vite:  2.0.0-beta.39
    @vuepress/bundler-webpack: Not Found
    @vuepress/cli:  2.0.0-beta.38
    @vuepress/client:  2.0.0-beta.38
    @vuepress/core:  2.0.0-beta.38
    @vuepress/markdown:  2.0.0-beta.38
    @vuepress/plugin-active-header-links:  2.0.0-beta.38
    @vuepress/plugin-back-to-top:  2.0.0-beta.38
    @vuepress/plugin-container:  2.0.0-beta.38
    @vuepress/plugin-docsearch: Not Found
    @vuepress/plugin-external-link-icon:  2.0.0-beta.38
    @vuepress/plugin-git:  2.0.0-beta.38
    @vuepress/plugin-google-analytics: Not Found
    @vuepress/plugin-medium-zoom:  2.0.0-beta.38
    @vuepress/plugin-nprogress:  2.0.0-beta.38
    @vuepress/plugin-palette:  2.0.0-beta.38
    @vuepress/plugin-prismjs:  2.0.0-beta.38
    @vuepress/plugin-pwa: Not Found
    @vuepress/plugin-pwa-popup: Not Found
    @vuepress/plugin-register-components: Not Found
    @vuepress/plugin-search: Not Found
    @vuepress/plugin-shiki: Not Found
    @vuepress/plugin-theme-data:  2.0.0-beta.38
    @vuepress/plugin-toc: Not Found
    @vuepress/shared:  2.0.0-beta.38
    @vuepress/theme-default:  2.0.0-beta.38
    @vuepress/utils:  2.0.0-beta.38
    vue:  3.2.33
    vue-loader: Not Found
    vue-router:  4.0.14
    vuepress: ^2.0.0-beta.39 => 2.0.0-beta.39
    vuepress-vite:  2.0.0-beta.39
    vuepress-webpack: Not Found

macOS

Browser: Edge 100.0.1185.39
Output of vuepress info:

# Paste output of `vuepress info` here

  System:
    OS: macOS 12.3.1
    CPU: (8) arm64 Apple M1
    Memory: 1.92 GB / 16.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 16.14.2 - ~/.volta/tools/image/node/16.14.2/bin/node
    Yarn: Not Found
    npm: 8.5.0 - ~/.volta/tools/image/node/16.14.2/bin/npm
  Utilities:
    Git: 2.32.0 - /usr/bin/git
  Browsers:
    Chrome: Not Found
    Edge: 100.0.1185.39
    Firefox: Not Found
    Safari: 15.4
  npmPackages:
    @vuepress/bundler-vite:  2.0.0-beta.39
    @vuepress/bundler-webpack: Not Found
    @vuepress/cli:  2.0.0-beta.38
    @vuepress/client:  2.0.0-beta.38
    @vuepress/core:  2.0.0-beta.38
    @vuepress/markdown:  2.0.0-beta.38
    @vuepress/plugin-active-header-links:  2.0.0-beta.38
    @vuepress/plugin-back-to-top:  2.0.0-beta.38
    @vuepress/plugin-container:  2.0.0-beta.38
    @vuepress/plugin-docsearch: Not Found
    @vuepress/plugin-external-link-icon:  2.0.0-beta.38
    @vuepress/plugin-git:  2.0.0-beta.38
    @vuepress/plugin-google-analytics: Not Found
    @vuepress/plugin-medium-zoom:  2.0.0-beta.38
    @vuepress/plugin-nprogress:  2.0.0-beta.38
    @vuepress/plugin-palette:  2.0.0-beta.38
    @vuepress/plugin-prismjs:  2.0.0-beta.38
    @vuepress/plugin-pwa: Not Found
    @vuepress/plugin-pwa-popup: Not Found
    @vuepress/plugin-register-components: Not Found
    @vuepress/plugin-search: Not Found
    @vuepress/plugin-shiki: Not Found
    @vuepress/plugin-theme-data:  2.0.0-beta.38
    @vuepress/plugin-toc: Not Found
    @vuepress/shared:  2.0.0-beta.38
    @vuepress/theme-default:  2.0.0-beta.38
    @vuepress/utils:  2.0.0-beta.38
    vue:  3.2.33
    vue-loader: Not Found
    vue-router:  4.0.14
    vuepress: ^2.0.0-beta.39 => 2.0.0-beta.39
    vuepress-vite:  2.0.0-beta.39
    vuepress-webpack: Not Found

just because when listing file from a dir, different operation system may give different sequence, Mac and Linux are case sensitive for filename while windows do not allow files with names of different casing in same dir. so Readme is before index in macOS and Linux, while behind in windows.
We do not recommend to use them together, also do not grantee the behavior.

just because when listing file from a dir, different operation system may give different sequence, Mac and Linux are case sensitive for filename while windows do not allow files with names of different casing in same dir. so Readme is before index in macOS and Linux, while behind in windows.
We do not recommend to use them together, also do not grantee the behavior.

Yes, I know this is principle behind.
But what I really want to say is that there exist some use case. For example, GitHub render README.md in the repo, but I also want to use a custom index page, so I create another index.md to design my home page for my docs.
If we mix two files, then my README.md may have a frontmatter rendered by GitHub, and some html used for the docs. Otherwise, we use index.md and then we lose the README.md in the github repo.
And the slidev docs is just an example slidevjs/docs-cn.

I try to fix this issue in #842 .
What we need to do is sorting, and I think it will not bring much overhead.

That could be an issue, but I don't think sorting the pages is a good solution.

As a workaround, you could try to add '!**/README.md' to exclude README.md files
E.g.:
export default defineUserConfig({
  pagePatterns: ['**/*.md', '!**/README.md', '!.vuepress', '!node_modules'],
})


#29 Add attributes from inputCol to metadata of discretized column

Hi,
I added the latter functionality, i.e. adding metadata to the produced DataFrame.
Feel free to merge it into Master, if you're satisfied with the commit ;)
Cheers
Marc

Good job. I forgot to include name and index in the attributes :)


Skip broken windows test due to #417

Kick the can down the road on #417 but at least we get the build back to green.

Had to skip all the tests that touch the /mgbellemare/Arcade-Learning-Environment repo on windows-latest (confirmed they are still running on mac and linux):

test_sb3_agent_evaluate()
test_sb3_agent_learn()
test_papag_agent_evaluate()
test_papag_agent_learn()
test_sb3_agent_evaluate()
test_sb3_agent_learn()
test_cli_freeze()

@andyk this is ready for you to take a look at!


Missing take method in pyarrow's Table class

I noticed that the method take is implemented for the class RecordBatch but not for Table in pyarrow. On the other hand, Take does exists for RecordBatch and Table on arrow's C++ code.
I'm working on a project that would benefit from take on pyarrow's Table so I wrote the code + some unit tests.
Unless there's a design/strategic reason not to have take on Table, I'd happy to create a patch.
Thanks

PR would be welcome. I opened https://issues.apache.org/jira/browse/ARROW-8513, so closing this
Keep en eye out that I added ChunkedArray.take in https://github.com/apache/arrow/pull/6970

Waiting for #6970 to be approved/merged


[fr] Deleted reference to removed file

en PR https://github.com/kubernetes/website/pull/28087/files
/assign @remyleone

cc @remyleone

/lgtm
/approve


Refactor codes of parsing flags and reading configuration

Coverage decreased (-9.6%) to 20.601% when pulling 4f2931ebb6a5081f9441b158c18f335bfb814e22 on config into b7e9ded3f510cc2968219994e9f15c0c33416f47 on master.


Show timestamp every 3rd message

I find [MessageGroup]() a little bit confusing. How can I show a timestamp every 3rd row?

Write some logic that creates a new message group for every 3 messages.

@atainter There is no way to use indexpath or something in collectionview?

NMessenger does not use the typical data for indexpath model.


add weather app and update README.md

This looks awesome @adityeah2k
Thank you for contributing! 🚀


Navigate to staging pose cannot be terminated

Terminate during navigation is not handled.
I think instead of the spin_until_future_complete here, there should be a loop that checks if the action was terminated (which has to be propagated from above).

This seems correct to me.
  auto future_goal_handle = nav_to_pose_client_->async_send_goal(goal);  // Note: Sends goal
  if (executor_.spin_until_future_complete(
      future_goal_handle, 2s) == rclcpp::FutureReturnCode::SUCCESS)
  {
    auto future_result = nav_to_pose_client_->async_get_result(future_goal_handle.get());
    if (executor_.spin_until_future_complete(
        future_result, timeout) == rclcpp::FutureReturnCode::SUCCESS)  // Note: Spins until completed, by any means
    {
      auto result = future_result.get();
      if (result.code == rclcpp_action::ResultCode::SUCCEEDED && result.result->error_code == 0) {  // Note: if we got to the pose, we're successful and can exit
        return;  // Success!
      }
    }
  }

  // Attempt to retry once using single iteration recursion
  if (!recursed) {
    goToPose(pose, max_staging_duration, true);  // Note: Else, if we're not successful, goal didn't send, or terminated action, attempt again
    return;
  }

  throw opennav_docking_core::FailedToStage("Navigation request to staging pose failed.");
When the navigation tree fails, the error propagates up correctly to where DockRobot was called from. But in the other direction it doesn't, so if you cancel the dock action by hand during navigation to staging area, the robot will finish navigating there, and then stops.
Later at the precise docking stage this is handled correctly in a loop:
    while (rclcpp::ok()) {
      try {
        // Approach the dock using control law
        if (approachDock(dock, dock_pose)) {
          ...
        }

        // Cancelled, preempted, or shutting down (recoverable errors throw DockingException)
        stashDockData(goal->use_dock_id, dock, false);
        publishZeroVelocity();
        docking_action_server_->terminate_all(result);

And in approachDock we check for cancellation.
    // Stop if cancelled/preempted
    if (checkAndWarnIfCancelled(docking_action_server_, "dock_robot") ||
      checkAndWarnIfPreempted(docking_action_server_, "dock_robot"))
    {
      return false;
    }

But above for the navigation part we don't check so it doesn't terminate:
      navigator_->goToPose(
        initial_staging_pose, rclcpp::Duration::from_seconds(goal->max_staging_time));
      RCLCPP_INFO(get_logger(), "Successful navigation to staging pose");

Ohhh, ok. That is not what I thought you were going with this. That seems like a great idea then
We can have a while loop for spinning the executor for a 1s timeout, check if cancellation is requested, and keep looping up to the max_staging_duration duration. If we find that a cancel was asked for, then we send the cancel request to the navigation action.
The only API change that should be required is passing goToPose in a lambda or std::function that wraps action_server->is_preemption_requested() so that the loop can know the state of the docking action
Sound good?

All right. I wrote the code, I'm gonna open the PR as soon as I've had the time to test.


⚠ ClamAV has degraded performance

In d63d554, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 65 ms

Resolved: ClamAV performance has improved in 925d927.


Offset property no longer working

Describe the bug
I think the offset property is no longer working. For example if I look at this sample in the documentation I expect the second dot to go first, but it goes second.
Also, on my work in progress portfolio website matthijn.github.io I have basically copied this text example and the animation order for me is incorrect, in the animation sample they use anime version 2. There it does work as intended.
The code I have in my animation with the offset can be found here. (See it in action here)
To Reproduce
Use the offset property
Expected behavior
I expect the animation to run on the requested offset time
Screenshots
See docs sample
Desktop (please complete the following information):

Mac OS (probably any OS)
Chrome / Safari tested
3.x

Smartphone (please complete the following information):
Untested

Also, this library is absolutely great! <3

I thought it was something I was doing. I just started with this library and yes it doesn't work

looks like this works already in 3.2.1, this can be closed now.


Page link block in body of article should display as a button

When adding a page link as it's own block in the body of an article it should display as a button with the specified text (relates to #164).
Note that links to pages within an article block should be unchanged and still display as text links.

fixed by #180


Fixes the high CPU usage

https://github.com/microsoft/vscode/issues/81535
https://github.com/microsoft/vscode/commit/080a1fa86fa42efee6a9e3cd0a5b9ca320f0090f

We'll get this change tonight as part of the vscode merge

@anthonydresser @aaomidi I think it's fine to wait for the merge if we expect it to get in EOD.  Otherwise, I'd recommend taking the fix by itself to make sure tomorrow's insider build is in good shape.  Thanks!

@anthonydresser Thanks! I'll close this PR once the merge is finalized :)

If the change is the exact same as vscodes there shouldn't be any issues taking this

@aaomidi do we need this any more?

Nope! We're good now :)


Scrape Wikipedia for starter list of parks

We could begin our spreadsheet with the list of parks from Wikipedia.

This should be a super-simple task. We can use the IMPORTHTML function in Google Spreadsheets.

We can handle geocoding with Geocode by Awesome Table.

The city provides a GIS layer for the polygons for the park boundaries. We'll use this for highlighting the parks in the app, and do some ground truthing to note specific features/locations in each park.


Security: Retry previously successful peers before peers that have always failed

Motivation
Zebra tries the oldest failed peers before recently failed peers, regardless of their last_success_time.
This design is vulnerable to attacks that take advantage of this specific retry order to starve the local node.
This design is also vulnerable to issues after a network connection failure, where every peer is in a failed state. It can also get into a failure mode where it cycles through disconnected successful peers, but never tries failed previously successful peers.
This issue is mitigated by:

the existing Zebra implementation, which ignores gossiped peers unless it specifically requested them
#1869 Zebra should limit the number of addresses it uses from a single node
#1873 Zebra's address book can use all available memory
#1848 Limit reconnection rate to each individual peer address

Solution
This fix depends on #1849 and #1871.
Zebra should retry previously successful peers that have recently failed, before it retries peers that have always failed.

[x] simplify the next peer address order to:

if last_success_time and last_failure_time are None, newest untrusted_last_seen_time (NeverAttempted state)
oldest last_success_time (either Responded or Failed states)
if last_success_time is None, oldest last_failure_time (Failed state)

[x] provide an accessor method on MetaAddr to simplify the interface to these times

Alternatives
This is a serious usability issue, so we must do something.
Context
zcashd does not have this issue.

This is a medium-risk change that isn't particularly important.

This was partly done during recent security fixes, and the rest of the ticket is obsolete.


Customize ShortSha length in templates [master]

It would be nice if we could somehow customize the length of the ShortSha in the templates over app.ini or somehow else:
Such that all GUI related SHA hashes have some predefined length. In our setting, we need a length of 12.
The relevant code is here.
https://github.com/go-gitea/gitea/blob/0e24af6951fa65d5094ab6da0f182697aab2a6f0/modules/base/tool.go#L57
Templates use
{{ShortSha .LatestCommit.ID.String}}

@gabyx why did you need 12-length shortsha? 100k+ commits?

I think the easyest way to go is with app.ini setting

it'd be good if we could somehow just have this as a per repo setting - or even have git do this I guess the question is how expensive is it to generate these shortsha lengths?

Method without git config:

Just store the option somewhere global app.ini or per repo .gitea/....

Method git config:
First good solution would be to read git config core.abbrev and use this to shorten the commit sha. However you dont want to read this config value for every conversion. Also its unsafe, since we dont know if gitea relies on this setting for parsing stuff from the log or so...
So the user can define on the server a git config --global core.abbrev.
Later solution could be to add this to the app.ini for a global invocation of git config --set --global core.abbrev XXX at startup.


Remove US restriction for Place Auto Completion.

GNI and Landing Site Demo:

https://screenshot.googleplex.com/asB3SXcEF76
https://screenshot.googleplex.com/r5rhA9184jc
https://screenshot.googleplex.com/NQknLy8ezjQ

Can you update the search box hint text as well?

Nice! So the new id mapping json works?

Sorry.
Yes, the json mappings seem ok so far.
I've updated the search text hint.


Add enquote as smart quote substitution option

fixes #897, MWE is over there

[x] add it to the wiki

Applied changes. Now you can either type a closing quote and it will skip the closing } or you can of course just type the closing } yourself and IJ will skip over it.


The zIndex of legend need to be decreased

L7 Version:
Platform:
Mini Showcase(like screenshots):
CodePen Link:

Current zIndex of legends is 1000, which is equal to<EMAIL_ADDRESS>modal, and this will cause some problems.

https://github.com/antvis/L7/pull/1849
可以等 @antv/l7 2.18.0 发布之后再试一下


partialy implement ConvertCommand

Codecov Report

Merging #24 (27ad0ee) into master (dc808ad) will increase coverage by 0.75%.
The diff coverage is 87.50%.

@@             Coverage Diff              @@
##             master      #24      +/-   ##
============================================
+ Coverage     84.37%   85.13%   +0.75%
- Complexity      364      366       +2
============================================
  Files            48       48
  Lines          1101     1110       +9
  Branches         63       64       +1
============================================
+ Hits            929      945      +16
+ Misses          147      138       -9
- Partials         25       27       +2

Impacted Files
Coverage Δ

...java/io/github/md2conf/toolset/ConvertCommand.java
83.33% <84.61%> (+74.24%)
:arrow_up:

...c/main/java/io/github/md2conf/toolset/MainApp.java
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dc808ad...27ad0ee. Read the comment docs.


README.md references to IS8601 should be ISO8601

There are a few placed in README.md which refer to "IS8601". They should refer to "ISO8601", which matches the #define.

Fixed... See https://github.com/ropg/ezTime/commit/8e8a459a09fcced5486b16853d447182b53d4e07


ci: jenkins: Use Jenkins workspace for GOPATH

We had our GOPATH being created/set to our home dir. That worked
fine for short lived Jenkins VMs currently utilised for the QA CI,
but might not be suitable for longer lived Jenkins slaves or ones
that perform parallel runs.
Jenkins provides a defined ${WORKSPACE} for each job - use that
location for GOPATH - which will also give us the benefit of
dropping and cleaning up properly if we ever use the 'remove
workspace after completion' feature of Jenkins.
Fixes: #667
Signed-off-by: Graham Whaley<EMAIL_ADDRESS>

@sboeuf @chavafg this is a bit of an RFC right now - what do you think?
For my metrics CI I will have more permanent (not single shot VM) jenkins slaves, so feel this is a move in the right direction - I may (not quite decided yet) set 'remove workspace' on my Jenkins so I don't just gather up lots of go/src/repos for all my old PRs for instance.

@chavafg , just out of interest, as I suspect this is unrelated to the PR - how is the kubernetes qa for stability - is the above 'failure' semi-expected at present for instance?

Hi @grahamwhaley
k8s tests are more stable now, but I checked the logs of the k8s execution and found that this was an unusual problem with the network, so we don't need to be worried about this:
ERROR: Get https://api.github.com/repos/clearcontainers/tests/pulls/668: net/http: TLS handshake timeout

Hi @sboeuf @chavafg @jodh-intel . I've updated the PR after staring some more an taking on the feedback:

Added the check for WORKSPACE being set
Annotated the script
Dropped the CI var as I don't think we actually use it

@chavafg - afaict, to do the 'install all latest components', all you actually have to do is run .ci/setup.sh is it not? It looks like that is maybe the only bit of this script you are really utilising when running it by hand?, so I've taken the presumption that I can leave the WORKSPACE code and check in here, and you can invoke .ci/setup.sh to get your local repos and install up to date.
... now I see if the CI passes with the changes....

Oh, and after some more contemplation I think I won't need to split the script - post this merging I will add a check around .ci/run.sh to check for METRICS_CI, and do an if/else to pick run_metrics_ci.sh for the METRICS Jenkins...

lgtm

oops, forgot the Fixes....


update README.md: ActiveHash::Matcher::Like

The tests in like_test.rb said the like matcher is ActiveHash::Matcher::Like but README.md said it was ActiveHash::Match::Like.
Just updated the text in README.md.

Thanks!


[SPI_STM32.c] INFO_WO_NSS_DEFINE macro inconsistent accross device series

STM32F7 series:
INFO_WO_NSS_DEFINE macro expect RCC_PERIPHCLK_SPI# define from HAL drivers,
stm32f7xx_hal_rcc_ex.h use RCC_PERIPHCLK_SPDIFRX define instead of what expected from INFO_WO_NSS_DEFINE macro.

This is addressed in commit e82c911372b9e0617aae0cac5893b4f984632ce7 .


EsetProtect Package update

Required items, please complete
Change(s):

See guidance below

Reason for Change(s):

See guidance below

Version Updated:

Required only for Detections/Analytic Rule templates
See guidance below

Testing Completed:

See guidance below

Checked that the validations are passing and have addressed any issues that are present:

See guidance below

Guidance <- remove section before submitting

Before submitting this PR please ensure that you have read the following sections and filled out the changes, reason for change and testing complete sections:
Thank you for your contribution to the Microsoft Sentinel Github repo.

Details of the code changes in your submitted PR.  Providing descriptions for pull requests ensures there is context to changes being made and greatly enhances the code review process.  Providing associated Issues that this resolves also easily connects the reason.

Change(s):

Updated syntax for XYZ.yaml

Reason for Change(s):

New schema used for XYZ.yaml
Resolves ISSUE #1234

Version updated:

Yes
Detections/Analytic Rule templates are required to have the version updated

The code should have been tested in a Microsoft Sentinel environment that does not have any custom parsers, functions or tables, so that you validate no incorrect syntax and execution functions properly.  If your submission requires a custom parser or function, it must be submitted with the PR.

Testing Completed:

Yes/No/Need Help

Note: If updating a detection, you must update the version field.

Before the submission has been made, please look at running the KQL and Yaml Validation Checks locally.
https://github.com/Azure/Azure-Sentinel#run-kql-validation-locally

Checked that the validations are passing and have addressed any issues that are present:

Yes/No/Need Help

Note: Let us know if you have tried fixing the validation error and need help.

References:

Guidance for Detection checks
General contribution guidance
PR validation troubleshooting

Hi @v-sabiraj,
I will review and share the feedback
Thanks

Hi @v-sabiraj,
Can you please add test evidence document.
Thanks

Hi @v-sabiraj,
Can you please add test evidence document.
Thanks

Hi @v-sabiraj,
Can you please add test evidence document. If there are further changes need to be done for it, please move it to draft.
Thanks

Hi @v-sabiraj,
Can you please provide update on this
Thanks

Hi @v-sabiraj,
Can you please provide update on this
Thanks

Closing this for now.


list of new directories -- could be in new module from where it is now

https://github.com/raman-noodles/raman-spectra-decomp-analysis/blob/ca322ff1c1207a45d667eb6071d3a5735d4346f8/ramandecompy/interpolatespectra.py#L199

used in a lot of places


Minibatch training for parametric umap

I'm trying to train a parametric umap model on ~100M clip embeddings sized 512D. It's a bit too large to run the whole dataset in memory; does the parametric umap model support batch training?
Thanks!

Yes, you can either load up part of the dataset and train, then load up another part and train on that, etc or you can write your own mini batch loader in tensorflow and train on that.
https://colab.research.google.com/drive/1WkXVZ5pnMrm17m0YgmtoNjM_XHdnE5Vp?usp=sharing
for the most part you’re going to get diminishing returns in terms of visual embedding quality when you add more data, and for most part embedding the biggest random sample of your data that will fit in memory will be just fine.


Refactoring validation function

What this PR does / why we need it:
Refactoring validation function

Special notes for your reviewer:
Release note:

The changes for checking for an empty repo and only printing the hash are good. The commits are getting more selfcontained and are easier to reason with. Great. If you are committing a fix you can also use git rebase -i $commit_to_base_on to edit/fixup the commits. That should only be done for non public code as it rewrites history so.
This reduces the fix, typo, test, try commits in between.
As the general notions is commit often, push infrequently rewriting history is usually fine for these cases.

lgtm


Add error handling for overflows

Fixes #189
We consider this as a bug, as it may be possible for the user to hit a very large amount for the balance, even if individual transaction amounts are small and not extreme.

LGTM

If we have this check then don't need to handle overflow in summary right? Because summary is the sum of a subset of transactions.

i think so

Fixes #220

LGTM


Use metrics-server instead of heapster for minikube

See changes in minikube.sh which should get adopted.
https://github.com/kyma-project/kyma/pull/2893

I switched the addon and validated that kubectl top command is still working


Add invite property

This replaces the rsvp-invite property, references in #139

please create a new version for the readme, I just released 4.7.3

Done - rewritten to 3.7.4. Sorry. I couldn't sleep and have been writing PRs.

Thanks a lot fo the small and clear pull requests

I've been practicing.

it makes it so much easier to review ;)
thanks for the tests btw.


allow to show stage background in no-animate mode

Hey, that is on purpose.

this will suspend stageBackground option 👎

Hey, the animated version is different from the non-animated version. There might be stacking context issues that are not handled by the animated version, so non-animated also acts as a fallback and solves those issues. The reason why we don't have the background for the stage is:
We don't have the dark overlay for the page in the non-animated version, it is achieved by the stage with the transparent background and having a high outline value in the non-animated version. The stage is placed above/below (depending upon the computed z-index value) and if it has the background, the element below it will get invisible, so to make the "highlighted" element below it visible it has the transparent background.
Look at the image below from the issue https://github.com/kamranahmedse/driver.js/issues/60 for the reason

I know this issue and I think it's good to add a stage highlight background to the active element, which help user to know which elements he should focus.
See my case below, the green box is the stage background.

All rights, may background:transparent!important be required for that fix. It doesn't work in my case. so I fix it in other way. ignore me


Superset fails to use Trino's nested fields in charts and filters

Bug description
Heya Superset team!
I ran into a bug that prevents me from using nested columns in charts and especially dashboard filters.
This happens with datasets that use a Trino database with the Enable row expansion in schemas enabled (introduced in Superset 3.1.0), which, as the name states, expands nested columns and effectively adds them to the dataset columns list.
This happens due to a possible problem with Superset's query builder, where a given nested column, let's say metadata.field_a is parsed into the query as "metadata.field_a", while Trino only accepts either "metadata"."field_a" or metadata.field_a.
This causes the column selection in chart exploration to create invalid queries (although it can be mitigated by always using Custom SQL). Additionally, this completely breaks the usage of dashboard filters, where the custom SQL workaround is not possible.
I provide evidence below and am available to provide more details if needed.
Looking forward towards your response.
How to reproduce the bug
In chart exploration:

Create a new chart
Select a Trino based dataset with nested columns
Add a nested column to dimensions
Press create chart
The chart now should display Cannot load filter Error: line 1:8: Column '<nested_column_name>' cannot be resolved

In dashboard filters:

Create a dashboard or open an existing one
Add a filter
Select a Trino based dataset with nested columns
Select a nested column as the column
Save
The filter now should display Cannot load filter Error: line 1:8: Column '<nested_column_name>' cannot be resolved

Screenshots/recordings
Stacktrace
superset Query SELECT "metadata.uuid" AS "metadata.uuid",
superset        COUNT(*) AS count
superset FROM raw_runtime_broker.intenthit
superset GROUP BY "metadata.uuid"
superset ORDER BY count DESC
superset LIMIT 1000 on schema raw_runtime_broker failed
superset Traceback (most recent call last):
superset   File "/app/superset/connectors/sqla/models.py", line 1795, in query
superset     df = self.database.get_df(sql, self.schema, mutator=assign_column_label)
superset   File "/app/superset/models/core.py", line 612, in get_df
superset     self.db_engine_spec.execute(cursor, sqls[-1])
superset   File "/app/superset/db_engine_specs/base.py", line 1596, in execute
superset     raise cls.get_dbapi_mapped_exception(ex) from ex
superset   File "/app/superset/db_engine_specs/base.py", line 1594, in execute
superset     cursor.execute(query)
superset   File "/usr/local/lib/python3.9/site-packages/trino/dbapi.py", line 592, in execute
superset     self._iterator = iter(self._query.execute())
superset   File "/usr/local/lib/python3.9/site-packages/trino/client.py", line 810, in execute
superset     self._result.rows += self.fetch()
superset   File "/usr/local/lib/python3.9/site-packages/trino/client.py", line 830, in fetch
superset     status = self._request.process(response)
superset   File "/usr/local/lib/python3.9/site-packages/trino/client.py", line 609, in process
superset     raise self._process_error(response["error"], response.get("id"))
superset trino.exceptions.TrinoUserError: TrinoUserError(type=USER_ERROR, name=COLUMN_NOT_FOUND, message="line 1:8: Column 'metadata.uuid' cannot be resolved", query_id=20240207_121856_11010_858tg)
superset 2024-02-07 12:18:56,581:WARNING:superset.connectors.sqla.models:Query SELECT "metadata.uuid" AS "metadata.uuid",
superset        COUNT(*) AS count
superset FROM raw_runtime_broker.intenthit
superset GROUP BY "metadata.uuid"
superset ORDER BY count DESC
superset LIMIT 1000 on schema raw_runtime_broker failed
superset Traceback (most recent call last):
superset   File "/app/superset/connectors/sqla/models.py", line 1795, in query
superset     df = self.database.get_df(sql, self.schema, mutator=assign_column_label)
superset   File "/app/superset/models/core.py", line 612, in get_df
superset     self.db_engine_spec.execute(cursor, sqls[-1])
superset   File "/app/superset/db_engine_specs/base.py", line 1596, in execute
superset     raise cls.get_dbapi_mapped_exception(ex) from ex
superset   File "/app/superset/db_engine_specs/base.py", line 1594, in execute
superset     cursor.execute(query)
superset   File "/usr/local/lib/python3.9/site-packages/trino/dbapi.py", line 592, in execute
superset     self._iterator = iter(self._query.execute())
superset   File "/usr/local/lib/python3.9/site-packages/trino/client.py", line 810, in execute
superset     self._result.rows += self.fetch()
superset   File "/usr/local/lib/python3.9/site-packages/trino/client.py", line 830, in fetch
superset     status = self._request.process(response)
superset   File "/usr/local/lib/python3.9/site-packages/trino/client.py", line 609, in process
superset     raise self._process_error(response["error"], response.get("id"))
superset trino.exceptions.TrinoUserError: TrinoUserError(type=USER_ERROR, name=COLUMN_NOT_FOUND, message="line 1:8: Column 'metadata.uuid' cannot be resolved", query_id=20240207_121856_11010_858tg)
superset <IP_ADDRESS> - - [07/Feb/2024:12:18:56 +0000] "POST /api/v1/chart/data HTTP/1.1" 400 73 "https://superset.factory-staging.scottytechnologies.com/explore/

Screenshots
The chart evidence and error

The dashboard filter evidence and error

SQL Lab evidence on how SQL looks and should look like
This is what Superset currently generates in charts and filters

Evidence on what works and what doesn't work with Trino

Superset version
3.1.0
Python version
3.9
Node version
Not applicable
Browser
Firefox
Additional context
I have the following feature flags enabled:
FEATURE_FLAGS = {
      "ALERT_REPORTS": True,
      "DASHBOARD_RBAC": True,
      "ENABLE_TEMPLATE_PROCESSING": True,
    }

There are no customization made on Superset, but the container image is packaged by me in order to include necessary dependencies like the Trino connector (python lib: trino==0.327.0).
The Superset is deployed in Kubernetes using the official Superset helm chart.
About the data source I am querying, I've already provided details on it on the post.
Checklist

[X] I have searched Superset docs and Slack and didn't find a solution to my problem.
[X] I have searched the GitHub issue tracker and didn't find a similar bug report.
[X] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

This is an interesting bug, because I was aware of the quoting issue you described happening in the original PRESTO_EXPAND_DATA feature in version <= 2.0 and made a point to make sure these fields were correctly quoted when writing the new expansion feature; you can see the logic of that here and I called it out here.
It's been working for us with deeply nested fields and many different charts, but we've backported the feature to 2.1 so it's possible it's interacting with something else that went into 3.1 and this logic is being skipped in some contexts.
I'm happy to give you a hand understanding the flow of some of the logic here if you want to try digging deeper into how this query was generated. Feel free to reach out on slack.

For a primer on the flow here, you can see the base spec's select_star calls database.get_columns, which then calls back to self.db_engine_spec.get_columns". It then uses cls._get_fields` to get the SQLA field definitions and builds the select star query from them.
get_columns on the trino spec contains a call to self._expand_columns linked above, which should correctly quote the fields and list them on the column defs as query_as. _get_fields on the base spec will take query_as as a literal column if it exists.
So at first glance I don't see an obvious problem where something's changed since the original design, but perhaps some part of this logic is being bypassed in your example.

Closing this as stale, assuming you've just moved on with corrected quotes, and most others aren't running into this problem. If it's still a bug in current Superset and/or people want to dig deeper, just say the word and we can re-open this.

This is still an issue, specifically in charts and dashboards. In SQL Lab we can easily mitigate the problem but not in other parts of Superset.

it is because apache supeset queries that recived by Trino calls nested fields in a string which is not supported by Trino
for example: "info.operator" is not valid while info.operator is valid, is there anything to work around?


Creating a non-text editor Pane. Input forms don't respond to input, or have just limited keyboard input

In my editor,  elements don't change value.
<input type="number" value="500" />
Up and down arrows are active but don't change the value in the input element
<input type='text' value="Hello" />
Typing anything in the field moves the cursor to the end of "Hello". Delete and backspace don't work either. Arrow keys don't do anything.
I've also embedded Ace editor. This seems to work, except for the "enter" key.
Is Atom intercepting keyboard input somehow? I'm not sure what's going on here.
Thanks

Research native-key-bindings.  Things are improved but not perfect.
On Thu, Apr 16, 2015 at 5:22 PM, jonathanjibo<EMAIL_ADDRESS>wrote:

In my editor, elements don't change value.
Up and down arrows are active but don't change the value in the input
element
Typing anything in the field moves the cursor to the end of "Hello".
Delete and backspace don't work either. Arrow keys don't do anything.
I've also embedded Ace editor. This seems to work, except for the "enter"
key.
Is Atom intercepting keyboard input somehow? I'm not sure what's going on
here.
Thanks
—
Reply to this email directly or view it on GitHub
https://github.com/atom/atom/issues/6408.

Whoops! This should have been closed a while back :blush:


Use GET action to generate content type options in AWX role wizard instead of POST

Normal users won't have POST in the OPTIONS response but they will have GET

As a team we decided to merge this one, since the test failures are not related.


Fix coverage upload

https://hynek.me/articles/ditch-codecov-python/ I suppose?

It can be. 👍

@Kludex looks like you need to register here: https://app.codecov.io/gh/Kludex/no-optional — so that it's possible to send coverage there too.

I've added the CODECOV_TOKEN.
This needs to be in place:
- name: Upload coverage reports to Codecov
  uses: codecov/codecov-action@v3
  env:
    CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

That token is not mandatory for public GHA runs but there's indeed stability problems without it..


Client Proxy Settings

Steps to reproduce
Function Request
Additional info
Circles instances have been blocked in China.
Please provide client proxy settings
or the built-in Tor Expert Bundle" for the purpose of bundling together inside other apps.

Tor Expert Bundle
The tor expert bundle contains the tor and pluggable transports binaries, bridge strings, and geoip data used in Tor Browser. The two versions shown indicate the version of Tor Browser these particular binaries were built for and the version of the contained tor daemon. These packages are intended for developers who need to bundle tor with their applications.

Application version
1.0.33
App Store name
F-Droid
Phone model
pixel7
Operating system version
GrapheneOS
Homeserver
No response
Contact info
No response

Please provide client proxy settings

Sorry, I'm not quite sure what you want here.  You want the ability to configure an application-specific proxy in the Circles app?

Please provide client proxy settings

Sorry, I'm not quite sure what you want here. You want the ability to configure an application-specific proxy in the Circles app?

Just like the Firefox desktop proxy setting.

or the built-in Tor Expert Bundle

Just like the Firefox desktop proxy setting.

@TarasSmakula How hard would it be to allow manual proxy configuration like this in our Settings?  Is this something we could do without too much trouble?


setState very slow

    public addShowInfo(showInfo: ShowInfo) {
        console.log("start set showinfo state")
        this.setState({ showInfo: showInfo }, ShowInfoStoreActions.AddShowInfo);
        console.log("finish set showinfo state")
    }

I have this simple setState which takes about 3-4 seconds to run.  Any idea why?

After doing some experimenting I found that the reason for the slowness is that I have a very large store (9000 records of contacts).  So I guess I will have to do different stores for each large data set unless you have a better option.

Since store state is immutable, data is cloned during a call to setState(). If you have a ton of data then I suspect that is causing the slowdown since each record would be cloned and it sounds like you’re adding a lot of data all at once. I don’t have a way to avoid that since the data needs to be immutable (aside from not putting everything into the store all at once).
It would be possible to add a feature to turn off cloning during setState() but I’d have to think through the other problems that would introduce. My first thought is that if setState() is being slow due to the immutable nature of the store that another approach for the data might need to be considered.

I agree, I am using CouchDb as a DB and as a result some of the documents are fairly large.  I guess there are going to be many things I have to rethink to use the store pattern. ;-)
I think your library is really great and I think I would have the same issue with ngRx because it will have to do a deep clone.  Also, think you for getting back so quick.

I can see how that would present a challenge with large and deeply nested data. As a heads-up, when you set ObservableStore.globalSettings.isProduction to true then only a shallow copy is made (in Dev a deep copy is always made which is slower of course) so it should be faster in that mode.
I’m going to give more thought to adding a parameter to setState() that would disable cloning in this type of scenario. That introduces other challenges since the data would be mutable, but by setting that a dev would be explicitly taking over control and be responsible for not mutating the data. I’ll think it over more.

Oh, what a different setting isProduction to true makes.  Thank you.

Awesome! Good to hear. Heard about that trick (deep cloning in dev to prevent mutations and shallow cloning in prod) from one of the NgRx guys.

As a heads-up, due to some edge cases that came up I ended up having to do deep cloning when setting and getting state even in production. So, if you're still using it to store large amounts of data you may see an impact with v2.2.8. However, setState() and getState() now have a deepClone boolean property that can be passed (https://github.com/danwahlin/observable-store#store-api). So, when adding all of the records you mentioned above you could do something like the following potentially:
this.setState(yourState, 'Your Action', true, **false**);
this.getState(false)
The last false value would tell it to NOT do a deep clone on the state being added. That's good for performance, but any changes to the data outside of the store could affect the store data since it'd be stored by reference without any cloning in that case.


:arrow_up:x.x.o(_): bump

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (9b07823) 100.00% compared to head (55be6d8) 100.00%.
Report is 202 commits behind head on main.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff            @@
##              main     #2617   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            8         8
  Lines          334       334
  Branches        20        20
=========================================
  Hits           334       334

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


feat(dashboard): Support for grpc streaming messages metrics

From 1.8, we added grpc streaming metrics

gRPC Request Message Count (istio_request_messages_total): This is a COUNTER incremented for every gRPC message sent from a client.

gRPC Response Message Count (istio_response_messages_total): This is a COUNTER incremented for every gRPC message sent from a server.

Should reflect this in the grafana dashboard.

cc @douglas-reid @mandarjog @bianpengyuan

todo: Add Collecting Metrics for GRPC Services.

@carolynhu Should we retarget this for 1.10?


Add option parameter for imap_open into .env file

add MAIL_FETCH_OPTIONS into .env.example
changed controller wich is calling login
changed app/Services/Pop3.php login to use $options

Looks like theere are conflicts

I see - i relativ new to PR and ask myself how should i fix it.
If you look at https://github.com/BadChoice/handesk/pull/10/files i see only a few lines of code that have minimal changes.
Can you not check an accept them ?

Ok - i found my mistake
Did not update my fork and worked on old local version ....
I updated my fork and i'll creadte a new PR


Pi!

@dorhinj defined exp and cos and sin as power series on the complex numbers, and proved that these series converged. However I don't think anyone has proved that these functions are continuous! The reason I am interested in this is that one definition of pi is that it's two times the smallest positive real r such that cos(r)=0. However, to make this definition work we need that cos is continuous and we need the intermediate value theorem and we need the fact that cos takes negative values. We just convinced ourselves that cos(2)<0 because it's 1-2+2/3-2^6/6!+2^8/8!-... and the sum is alternating and the absolute values of the terms are decreasing in value from 2^6/6! onwards, so cos(2)=-1/3-2^6/6!+...<-1/3<0. Now we need that cos is continuous (and everyone seems to be a bit hazy about how this works, although 2nd years and above all know that a power series is actually differentiable within its radius of convergence, so certainly continuous), and once we have that we should be able to define pi.

Current state: still don't continuity of cos -- @dorhinj how much work is this to PR? Can I help?

Update: exp is hopefully on the way!


Expected authority name changed to include team identifier

Expected authority name changed to include team identifier:
Authority=Developer ID Application: Zoom Video Communications, Inc. (BJ4HAAB9B3)
Authority=Developer ID Certification Authority
Authority=Apple Root CA

@sgmills Sorry, I forgot you submitted this PR before I started working on some issues in my autopkg run this evening.  My PR caf1cb9 resolves this for both the Zoom-ForIT and Zoom-OutlookPlugin recipes.
Thanks!


Week 3: Innovation - Iteration

Score:
Self Grade: 2/3
Points will be awarded on these weighted elements:

To-Do
Link

Iteration Mini-Lab
LINK

Deployment on AWS
-

Focus / Habits
LINK

2/3, didn't get deployment but rest is all good


remove requests dependency

This updates the analyzer and mutator code to remove an unused
dependency on requests from the requirements.txt files.

starting with something light, this just addresses the security warnings from github (cc @willb )

lgtm, thanks Mike!


Unhide when errored with has

https://app.productive.io/2650-4site-interactive-studios-inc/tasks/task/6488920

You responded before I had time to post my comment on the Productive task. Everything you asked for is captured there.

@fernanDOTdo I've merged main into this and altered it so it doesn't cause anything to break. It's ready to be merged.
Please note that because of the :has selector this now requires client themes to update postcss-preset-env. On WWF version  "^9.3.0" worked. This should automatically pull in autoprefixer too.


Add config to change the pipeline controller cron schedule timezone

We couldn't find a config to change the cron schedule timezone.
We would like an option to change the timezone to our local timezone.

E.g to "Africa/Nairobi" or GMT+3

Just to clarify, the cron schedule is currently based on the local time of the machine on which the controller runs; we can certainly add timezone too.


argument should be not single but * (array)

should!

ummm...
is it possible? (Vision API's specification is ...???)

and is it necessary???
[a, b, c, d].each do |file|
   # exec method
end

is avobe OK?


Improve GH CI/CD workflow

Ensure we publish working policies

@kkaempf, about pull_request

Due to the dangers inherent to automatic processing of PRs, GitHub’s standard pull_request workflow trigger by default prevents write permissions and secrets access to the target repository. However, in some scenarios such access is needed to properly process the PR. To this end the pull_request_target workflow trigger was introduced.

What kind of security problems are you referring to in this case?
Based on the blog post I linked, following their advice:

Avoid using pull_request_target if the workflow doesn’t need write repository permissions and doesn’t use any repository secrets. They can simply use the pull_request trigger instead.
Assign repository privileges only where needed explicitly through pull_request and workflow_run as in our previous example.
Add a condition to the pull_request_target to run only if a certain label is assigned the PR, like safe to test that indicates the PR has been vetted by someone with write privileges to the target repository:

I think it's safe to use pull_request, since we don't need write permissions and we don't use secrets, only the exposed GITHUB_TOKEN envvar.


pretrained shape network

Thank you for releasing the code! I see you mentioned in the supplementary materials that you did some pretraining on the shape network. Can you share this pretrained model? Also, would it still be able to learn the mask well if trained from scratch?

Hi, the pre-trained model can be found in assets/smpl_init.pth.
Yes, it's still possible to learn the mask from scratch but you first need to disable in_shape_loss, since in that case, the shape network will be initialized as a sphere and this loss doesn't make sense anymore. But sometimes, training from scratch could also fail (according to my experience). A coarse human shape initialization is encouraged in any case though.


GSM MAP: Unable to decode SendRoutingInfo and InsertSubscriberData

I use the ASN.1 files for GSM MAP from Wireshark sources (I also tested some others) and I always have errors when decoding SendRoutingInfo and InsertSubscriberData. (BER decoder)
I can send the examples if needed.
The version I use is 0.9.27.

I have found the problems:

For InsertSubscriberData, my asn uses COMPONENTS OF keyword. If I do not use it, no problem,  I workarounded the problem.
for SendRoutingInfo it is a version problem, but as explained previously, the compiler does not manage versions.


Can't find pngwutil.c building tensorflow

I am getting this error when compiling tensorflow on RHEL 6.6.
[<user>@blah tensorflow]$ ../bazel/output/bazel --output_base=/lvol/<user> build --genrule_strategy=standalone --spawn_strategy=standalone --verbose_failures -j 1 -c opt //tensorflow/cc:tutorials_example_trainer 2>&1|tee compile0.log
......
WARNING: Sandboxed execution is not supported on your system and thus hermeticity of actions cannot be guaranteed. See http://bazel.io/docs/bazel-user-manual.html#sandboxing for more information. You can turn off this warning via --ignore_unsupported_sandboxing.
____Loading package: tensorflow/cc
____Loading...
____Loading package: @bazel_tools//tools/cpp
____Loading package: @local-jdk//
____Loading package: tensorflow/core/platform/default/build_config
____Loading package: @png_archive//
____Loading package: @jpeg_archive//
____Loading complete.  Analyzing...
____Found 1 target...
____Building...
____[0 / 1] BazelWorkspaceStatusAction stable-status.txt
____[3 / 28] Compiling google/protobuf/src/google/protobuf/arena.cc
____[27 / 82] Compiling google/protobuf/src/google/protobuf/any.cc
____[108 / 169] Compiling google/protobuf/src/google/protobuf/any.cc [for host]
____[161 / 260] Compiling google/protobuf/src/google/protobuf/compiler/code_generator.cc [for host]
____[252 / 330] Compiling external/re2/re2/bitstate.cc
ERROR: missing input file '@png_archive//:libpng-1.2.53/pngwutil.c'.
ERROR: /lvol/<user>/external/png_archive/BUILD:33:1: @png_archive//:png: missing input file '@png_archive//:libpng-1.2.53/pngwutil.c'.
____Building complete.
Target //tensorflow/cc:tutorials_example_trainer failed to build
ERROR: /lvol/<user>/external/png_archive/BUILD:33:1 1 input file(s) do not exist.
____Elapsed time: 277.881s, Critical Path: 6.96s

Here is what the tree looks like. All the soft-links point to valid files.
[<user>@atl4-05 tensorflow]$ ls /lvol/<user>/external/png_archive/
aclocal.m4      config.guess  example.c          LICENSE        pngbar.jpg  png.h       pngrutil.c   pngwrite.c   test-pngtest.sh
ANNOUNCE        config.h.in   INSTALL            ltmain.sh      pngbar.png  pngmem.c    pngset.c     pngwtran.c   TODO
autogen.sh      config.sub    install-sh         Makefile.am    png.c       pngnow.png  pngtest.c    pngwutil.c   WORKSPACE
BUILD           configure     KNOWNBUG           Makefile.in    pngconf.h   pngpread.c  pngtest.png  projects     Y2KINFO
CHANGES         configure.ac  libpng-1.2.53.txt  missing        pngerror.c  pngread.c   pngtrans.c   README
CMakeLists.txt  contrib       libpng.3           mkinstalldirs  pnggccrd.c  pngrio.c    pngvcrd.c    scripts
compile         depcomp       libpngpf.3         png.5          pngget.c    pngrtran.c  pngwio.c     test-driver

Here is what the /lvol//external/png_archive/BUILD file looks like.
package(default_visibility = ["//visibility:public"])

prefix_dir = "libpng-1.2.53"

PNG_SOURCES = [
    "png.c",
    "pngerror.c",
    "pngget.c",
    "pngmem.c",
    "pngpread.c",
    "pngread.c",
    "pngrio.c",
    "pngrtran.c",
    "pngrutil.c",
    "pngset.c",
    "pngtrans.c",
    "pngwio.c",
    "pngwrite.c",
    "pngwtran.c",
    "pngwutil.c",
]

genrule(
    name = "configure",
    srcs = glob(
        ["**/*"],
        exclude = [prefix_dir + "/config.h", "configure"],
    ),
    outs = [prefix_dir + "/config.h"],
    cmd = "pushd external/png_archive/%s; workdir=$$(mktemp -d -t tmp.XXXXXXXXXX); cp -a * $$workdir; pushd $$workdir; ./configure --enable-shared=no --with-pic=no; popd; popd; cp $$workdir/config.h $(@D); rm -rf $$workdir;" % prefix_dir,
)

cc_library(
    name = "png",
    srcs = [prefix_dir + "/" + source for source in PNG_SOURCES],
    hdrs = glob(["**/*.h"]) + [":configure"],
    includes = [prefix_dir],
    linkopts = ["-lz"],
    visibility = ["//visibility:public"],
)

What version of bazel do you happen to be running?
@davidzchen, in case he knows :)

I checked out bazel from github and built it myself 2 days ago. So HEAD ?

This does seem like a bazel bug, btw -- this doesn't even hit the tensorflow side of things before it fails.  Can you report a bug there and then come back to us if it turns out that it's a problem on our end?

Sorry for the delay. I don't see anything obviously wrong with the BUILD file either. Can you open a bug at bazelbuild/bazel?

Closing for now until there's more evidence there's something the TF team can help with.

I am using bazel 0.3.0 and ran into the same problem as well.

forgot to mention some important details in the previous post : I downloaded the dependency packages and transfer to a server without internet access. I have to uncompress them and load from the uncompressed directories when I put tar.gz and zip files on the local. Not sure but It seems the reported missing files are random and all of them from the repositories corresponding to tar.gz and zip files.


forge verify-bytecode panicks with end index out of range

Component
Forge
Have you ensured that all of these are up to date?

[X] Foundry
[X] Foundryup

What version of Foundry are you on?
forge 0.2.0 (08b2d3e 2024-06-07T00:17:47.048257430Z)
What command(s) is the bug in?
forge verify-bytecode
Operating System
Linux
Describe the bug
When verifying bytecode on the morpho-blue-bundlers, I get the following error messages:
The application panicked (crashed).
Message:  range end index<PHONE_NUMBER>3709517144 out of range for slice of length 15743
Location: crates/verify/src/bytecode.rs:626

This is a bug. Consider reporting it at https://github.com/foundry-rs/foundry

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 7 frames hidden ⋮
   8: core::slice::index::slice_end_index_len_fail_rt::hf817bd92ee257898
      at <unknown source file>:<unknown line>
   9: core::slice::index::slice_end_index_len_fail::h9965f4285b8d5d41
      at <unknown source file>:<unknown line>
  10: forge_verify::bytecode::try_match::hd38840daf41bf2ac
      at <unknown source file>:<unknown line>
  11: forge_verify::bytecode::VerifyBytecodeArgs::run::{{closure}}::h66995c85df5ed43f
      at <unknown source file>:<unknown line>
  12: forge::main::h46f5b6d0ffb38582
      at <unknown source file>:<unknown line>
  13: std::sys_common::backtrace::__rust_begin_short_backtrace::h9fc915f4253178a8
      at <unknown source file>:<unknown line>
  14: main<unknown>
      at <unknown source file>:<unknown line>
  15: __libc_start_main<unknown>
      at <unknown source file>:<unknown line>
  16: _start<unknown>
      at <unknown source file>:<unknown line>

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.

Reproduce

checkout morpho-blue-bundlers
do yarn and forge install
add the following to foundry.toml

[etherscan]
base = { key = "${BASESCAN_KEY}" }

have a proper setup for the environment variables BASE_RPC and BASESCAN_KEY
run forge verify-bytecode 0x123f3167a416cA19365dE03a65e0AF3532af7223 CompoundV2MigrationBundlerV2 --rpc-url $BASE_RPC

Able to reproduce the bug with the given setup (cc @yash-atreya)
BASESCAN_KEY=<BASESCAN_API_KEY> forge verify-bytecode 0x123f3167a416cA19365dE03a65e0AF3532af7223 CompoundV2MigrationBundlerV2 --rpc-url <ALCHEMY_API_KEY>

Yields
Verifying bytecode for contract CompoundV2MigrationBundlerV2 at address 0x123f3167a416cA19365dE03a65e0AF3532af7223
The application panicked (crashed).
Message:  range end index<PHONE_NUMBER>3709517144 out of range for slice of length 15743
Location: crates/verify/src/bytecode.rs:621

This is a bug. Consider reporting it at https://github.com/foundry-rs/foundry

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   1: __libc_start_call_main<unknown>
      at ./csu/../sysdeps/nptl/libc_start_call_main.h:58
   2: __libc_start_main_impl<unknown>
      at ./csu/../csu/libc-start.c:360

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
Aborted (core dumped)


filter_bags_by_df

Added a simple function that filters bags using a dataframe
Added a simple function to get a represantative tile for each slide

Thanks @matte-esse - looks good to me. I made some minor modifications to the interface to more closely adhere to pandas-style usage, and added more options for filtering.
Example of new syntax:
dataset.filter_bags_by_df(source, dest, data=df, by="column", threshold=0.3, direction="greater")

Can you run these changes through some test data? If it makes it past testing you can feel free to merge.

Tested ✔


Reconstruct nested values from function

Related issue(s)
Resolves #492
Description
Summary of changes
Flatten data returned from a function is reconstructed into a value using the function return type
Test additions / changes
Regression test case added in 'struct' test case
Checklist

[X] I have tested the changes locally.
[X] I have formatted the changes with Prettier and/or cargo fmt with default settings.
[X] I have linked this PR to the issue(s) that it resolves.
[X] I have reviewed the changes on GitHub, line by line.
[X] I have ensured all changes are covered in the description.

N.B. I could not validate that this circuit is passing: https://github.com/vezenovm/aztec-circuits-noir/pull/5/files
because it uses dependencies (notes..) that are not provided.
@vezenovm, could you check that this PR works for this circuit?


Django example fix

(Describe the goal of this PR. Mention any related Issue numbers)
The given example of django is not compatible with slack-bolt v1.1.1, after the org-level installation support.
I updated the django models of SlackBot and SlackInstallation, fixed find_bot method of InstallationStore, and implemented find_installation method to it.
Category (place an x in each of the [ ])

[ ] slack_bolt.App and/or its core components
[ ] slack_bolt.async_app.AsyncApp and/or its core components
[ ] Adapters in slack_bolt.adapter
[ ] Document pages under /docs
[x] Others

Requirements (place an x in each [ ])
Please read the Contributing guidelines and Code of Conduct before creating this issue or pull request. By submitting, you are agreeing to those rules.

[x] I've read and understood the Contributing Guidelines and have done my best effort to follow them.
[x] I've read and agree to the Code of Conduct.
[x] I've run ./scripts/install_all_and_run_tests.sh after making the changes.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@vokey Thanks a lot! LGTM


Retrain BEVDet Using TIER4 Dataset

Checklist

[x] I've read the contribution guidelines.
[x] I've searched other issues and no duplicate issues were found.
[x] I've agreed with the maintainers that I can plan this task.

Description
Due to the poor detection performance of the current BEVDet model on the TIER4 dataset, we plan to retrain the BEVDet model on the TIER4 dataset.
Reference this BEVDet train code baseline: Link
TIER4 Dataset  Link
Note: Now, the data provided by the above link only has 39 scenes, totaling 2874 keyframes, which does not meet the training requirements. We will first use the TIER4 dataset to complete the training process.
Purpose
Retrain the BEVDet model on the TIER4 dataset.
Possible approaches

Use the same network configuration as the deployed onnx model to train on the Nuscenes dataset to ensure that the model can converge normally.
train on the TIER4 dataset.

Definition of done
to write a script for converting TIER4 dataset to BEVDet train pkl file format

We retraining BEVDet with TIER4 dataset.
The statistical results  of TIER4 dataset are as follows:
39 scenes(approximately 2600 keyframes), divide into 31 train sets and 7 val sets, and 1 scene discarded(due to missing files) .
After 20 epochs of training, the loss converges. The model(tensorrt model) is able to detect some objects in a single frame of TIER4 data.
a frame from train set:

a frame from val set:

but  the evaluation results of the model on val sets are very poor.
mAP: 0.0066
mATE: 0.9937
mASE: 0.7847
mAOE: 0.8268
mAVE: 0.8569
mAAE: 0.7370
NDS: 0.0834
Eval time: 27.3s

Per-class results:
Object Class	AP	ATE	ASE	AOE	AVE	AAE
car	0.022	0.599	0.193	0.240	1.111	0.658
truck	0.014	0.939	0.332	0.100	0.665	0.237
bus	0.000	1.000	1.000	1.000	1.000	1.000
trailer	0.000	1.000	1.000	1.000	1.000	1.000
construction_vehicle	0.000	1.000	1.000	1.000	1.000	1.000
pedestrian	0.000	1.000	1.000	1.000	1.000	1.000
motorcycle	0.030	1.399	0.322	1.101	0.079	0.000
bicycle	0.000	1.000	1.000	1.000	1.000	1.000
traffic_cone	0.000	1.000	1.000	nan	nan	nan
barrier	0.000	1.000	1.000	1.000	nan	nan

The possible reason is that the amount of train dataset is too small, and the model has poor generalization on the validation set.

Organizing the model retraining process in this repos

The detection results of TIER4 data on a model trained on the NuScenes dataset:


Package as gem

Packaging the application as a gem will make install as simple as possible.

At the moment, this is blocked on this: https://github.com/Shopify/ssllabs.rb/issues/23 - until that's resolved, can't move forward with release.


Update to Laravel 11

Is there any chance or hints on how to update this project for Laravel 11?

I’d like to do it as soon as I have some time!
But if you want to start a PR, feel free to open it! 🙏

I'd love to, but I don't know where to start. I'm a total Laravel beginner and hoping to learn from this and other projects, but it's hard to find something practical that works with Laravel 11. Good to know you are responsive to the idea, so if I do make any progress I'll certainly send a PR.

Done here https://github.com/guillaumebriday/laravel-blog/commit/6621a9f783b3e2122e3485b40d0921d33c2c120d
Thanks

Thank you guillaumebriday! Even though I've been an avid GH user for over a decade I'm still not used to submitting PRs to other projects so until I RTFM, again, here's my initial suggestion in README.md (drop the $ so this section can be posted directly into a shell)...
git clone https://github.com/guillaumebriday/laravel-blog.git
cd laravel-blog
composer install
npm install
cp .env.example .env
touch database/database.sqlite
php artisan key:generate
php artisan storage:link
php artisan horizon:install
php artisan telescope:install
composer require laravel/pail
php artisan migrate:fresh --seed
composer run dev

and, in composer.json...
    "scripts": {
    ...
        ],
        "dev": [
            "Composer\\Config::disableProcessTimeout",
            "npx concurrently -k -c \"#93c5fd,#c4b5fd,#d4d4d8,#fdba74\" \"php artisan serve\" \"php artisan queue:listen --tries=1 --timeout=120\" \"php artisan pail\" \"npm run dev\" --names=server,queue,logs,vite"
        ]
    },


Suggestion: configurable default user data

Right now I only need to change the default user id, but I guess it could be useful (not sure is easy or even possible), to emulate different channel ids, and similar stuff.

Hi @cesarvarela, This is a planned feature. We have an improved bot connection UI in the works that allows for editing of the user list (both name and id). In the meantime, it is possible to edit this value directly in the server.json of the app. The location of this file is depends on your platform. On Windows, it is located below %APPDATA%\botframework-emulator


Update-GSUser support for unarchiving

Update-GSUser allows archiving a user via the switch parameter Archived. However I did not find a way to unarchive a user in the same manner. Ideally the parameter should be changed from type [switch] to [bool] where $true represents archiving and $false unarchiving.
Current behavior for archiving
Update-GSUser -User<EMAIL_ADDRESS>-Archived

Desired behavior
# Archiving
Update-GSUser -User<EMAIL_ADDRESS>-Archived $true

# Unarchiving
Update-GSUser -User<EMAIL_ADDRESS>-Archived $false

This would match the Google Admin SDK API method users.update.
PUT<EMAIL_ADDRESS>{
  "archived": false
}

Hi @jeffreymcclain, sorry for my late reply as well.
Update-GSUser -User<EMAIL_ADDRESS>-Archived:$false does work as expected. Thanks for pointing it out. I was not aware you can explicitly set a swich parameter.
I have hence closed the ticket.