一直停在文本加载处

一直停在文本加载处
(chatglm) D:\langchain-ChatGLM-master>python cli_demo.py
Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a revision is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████| 8/8 [00:05<00:00,  1.49it/s]
No sentence-transformers model found with name C:\Users\MSN/.cache\torch\sentence_transformers\GanymedeNil_text2vec-large-chinese. Creating a new one with MEAN pooling.
No sentence-transformers model found with name C:\Users\MSN/.cache\torch\sentence_transformers\GanymedeNil_text2vec-large-chinese. Creating a new one with MEAN pooling.
Input your local knowledge file path 请输入本地知识文件路径：D:\langchain-ChatGLM-master\content\state_of_the_search.txt
卡在最后一步一个多小时了，请问这正常吗？
python3.8，cuda11.7，环境与项目相同，怎么破

感觉不太正常，请问先存占用情况怎么样？
bookwoods @.***>于2023年4月15日 周六22:29写道：

一直停在文本加载处
(chatglm) D:\langchain-ChatGLM-master>python cli_demo.py
Explicitly passing a revision is encouraged when loading a model with
custom code to ensure no malicious code has been contributed in a newer
revision.
Explicitly passing a revision is encouraged when loading a configuration
with custom code to ensure no malicious code has been contributed in a
newer revision.
Explicitly passing a revision is encouraged when loading a model with
custom code to ensure no malicious code has been contributed in a newer
revision.
Loading checkpoint shards:
100%|█████████████████████████████████████████████████████████| 8/8
[00:05<00:00, 1.49it/s]
No sentence-transformers model found with name
C:\Users\MSN/.cache\torch\sentence_transformers\GanymedeNil_text2vec-large-chinese.
Creating a new one with MEAN pooling.
No sentence-transformers model found with name
C:\Users\MSN/.cache\torch\sentence_transformers\GanymedeNil_text2vec-large-chinese.
Creating a new one with MEAN pooling.
Input your local knowledge file path
请输入本地知识文件路径：D:\langchain-ChatGLM-master\content\state_of_the_search.txt
卡在最后一步一个多小时了，请问这正常吗？
python3.8，cuda11.7，环境与项目相同，怎么破
—
Reply to this email directly, view it on GitHub
https://github.com/imClumsyPanda/langchain-ChatGLM/issues/108, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABLH5EV7RDGZFC743SMP3OTXBKWFVANCNFSM6AAAAAAW7OMQXY
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

感觉不太正常，请问先存占用情况怎么样？ bookwoods @.>于2023年4月15日 周六22:29写道：
…
一直停在文本加载处 (chatglm) D:\langchain-ChatGLM-master>python cli_demo.py Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a revision is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████| 8/8 [00:05<00:00, 1.49it/s] No sentence-transformers model found with name C:\Users\MSN/.cache\torch\sentence_transformers\GanymedeNil_text2vec-large-chinese. Creating a new one with MEAN pooling. No sentence-transformers model found with name C:\Users\MSN/.cache\torch\sentence_transformers\GanymedeNil_text2vec-large-chinese. Creating a new one with MEAN pooling. Input your local knowledge file path 请输入本地知识文件路径：D:\langchain-ChatGLM-master\content\state_of_the_search.txt 卡在最后一步一个多小时了，请问这正常吗？ python3.8，cuda11.7，环境与项目相同，怎么破 — Reply to this email directly, view it on GitHub <#108>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLH5EV7RDGZFC743SMP3OTXBKWFVANCNFSM6AAAAAAW7OMQXY . You are receiving this because you are subscribed to this thread.Message ID: @.>

显卡24G的4090显存占用12G

请问现在解决了吗

还没有，放弃了，再试试其他办法

如果使用默认的6b+text2vec模型，建议使用显存至少15GB的显卡，详细请见 README.md 中硬件配置相关内容
12G可以考虑使用chatglm-6b-int4模型+text2vec
bookwoods @.***>于2023年4月17日 周一19:21写道：

还没有，放弃了，再试试其他办法
—
Reply to this email directly, view it on GitHub
https://github.com/imClumsyPanda/langchain-ChatGLM/issues/108#issuecomment-1511159993,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABLH5ET5ETW6LSRBCKM7ILTXBURT7ANCNFSM6AAAAAAW7OMQXY
.
You are receiving this because you commented.Message ID:
@.***>


When submit application is failed, the exit code is swallowed

Step to reproduce:

Submit a valid jar via bin/gear app -jar foo.jar (The application is running)
Submit it again. Expects an error, because the application exists already. But the exit code is always 0.

This will cause restapi handling in dashboard unexpectedly.
[ERROR] [07/11/2015 01:09:49.360] [ClientContext] Failed to submit
java.lang.Exception: Application name dag already existed
	at org.apache.gearpump.cluster.master.AppManager$$anonfun$clientMsgHandler$1.applyOrElse(AppManager.scala:96)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
	at org.apache.gearpump.cluster.master.AppManager.aroundReceive(AppManager.scala:43)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
	at akka.actor.ActorCell.invoke(ActorCell.scala:487)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
	at akka.dispatch.Mailbox.run(Mailbox.scala:220)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

This issue impacts #1167


Allow '&' and disallow ':'  in Windows filenames.

Fixes #161

Thanks Klaus (forgot to merge this earlier!)


Pick correct compilerOptions when checking if we can share emitSignatures

OldCompilerOptions is set only if referencemaps are created vs not created. Module.None does not create reference map but other values do create it. Which is what determines if we can say copy semantic diagnostics etc..
But emit signatures are purely based on declaration path and their contents so they should only use the compilerOptions directly from old state.
Fixes #50902

The fix works on my project (ported manually in JS code).
Thanks for the quick reaction.

Can/should this fix be cherrypicked to the 4.8 branch?
Looks quite safe to me and v4.9 will be released in about 8 weeks.

@DanielRosenwasser and @RyanCavanaugh who can decide if it would be fit for 4.8 patch


ember-cli-qunit 0.3.11 autostart handling

Hi.
Two days ago ember-cli-qunit was released, and because of this change https://github.com/ember-cli/ember-cli-qunit/commit/98c8d370d574adc4deb2ab438edd0d248b22155b no tests are run anymore.
Giving the autostart=true config as a URL parameter did not work either.
Any idea how to resolve this in ember-cli-blanket or is that an issue for ember-cli-qunit or even blanket.js?
Thanks

If you look at the top lines of lib/start.js you can see that the
ember-cli-qunit behavior has flip flopped a few times on this.  Since we
set auto start to false to allow instrumentation to finish, qunit.start
must be called explicitly.  As a temporary measure you can uncomment the
qunit.start in start.js and we'll look into a better way to detect this
behavior so that any further flip flops don't have this impact.
On Friday, April 10, 2015, Claudio Bley<EMAIL_ADDRESS>wrote:

Hi.
Two days ago ember-cli-qunit was released, and because of this change
ember-cli/ember-cli-qunit@98c8d37
https://github.com/ember-cli/ember-cli-qunit/commit/98c8d370d574adc4deb2ab438edd0d248b22155b
no tests are run anymore.
Giving the autostart=true config as a URL parameter did not work either?
Any idea how to resolve this in ember-cli-blanket or is that an issue for
ember-cli-qunit or even blanket.js?
Thanks
—
Reply to this email directly or view it on GitHub
https://github.com/sglanzer/ember-cli-blanket/issues/41.

@sglanzer any reason why blanket can't do the same thing as ember-cli in https://github.com/ember-cli/ember-cli-qunit/commit/98c8d370d574adc4deb2ab438edd0d248b22155b and so let the user have the final say on whether tests should automatically start?

Looks reasonable to me, the only requirement is that autostart must be false while the instrumentation is occurring.
If you want to submit a pull request I'll merge it, otherwise we'll try to get to this sometime today.

@sglanzer cool.  It sounds like you don't need any changes from ember-cli-qunit, but let me know if you do

@hjdivad just a note since it's going to work in this case, but ember-cli-qunit is actually overriding the value of autostart downstream.  Ember-cli-blanket is upstream, so it won't hurt in this case, but it may be something you want to fix.
var autostart = QUnit.config.autostart !== false;
QUnit.config.autostart = false; // Overrides the value

   setTimeout(function() {
     TestLoader.load();
     // Doesn't set the value back, everything downstream will get 'false'
     if (autostart) {
      QUnit.start();
    }
   }, 250);

@sglanzer that's a great point.  Unfortunately ember-cli-qunit has the same problem ember-cli-blanket has: we need QUnit to not actually autostart (in ember-cli-qunit's case to work around a tedious chrome deficiency when using sourcemaps).
I don't see a way for us to not interfere downstream except by requiring users to use a different flag from the standard QUnit.config.autostart.  Do you?

@hjdivad Can we just set the value back into the property after consuming it?  I.e. Instead of calling Qunit.start() in the addon, just set qunit.config.autostart back to the value of 'autostart' and let it fall through?

@sglanzer at that point it's too late I think.  QUnit will check autostart during dom ready, but if we start that early we'll run into the issue with chrome not having loaded sourcemaped files yet.
The problem will, presumably, go away on its own once chrome fixes their bug and we don't need this 250ms delay.

@sglanzer we could add something to QUnit's api to have more control over when autostart actually starts but I doubt they'd want to complicate their API for something expected to be temporary.

@sglanzer sorry I read your comment as if you were talking about ember-cli-qunit.  What are the actual runtime requirements of ember-cli-blanket?  Do you just need the tests to start after a short delay?  If so you probably don't need to do anything at all for newer versions of ember-cli-qunit.

@hjdivad No, I was referring the ember-cli-qunit, but obviously if it's a workaround then we can't/shouldn't do anything about it.  Blanket needs to instrument all of the files in the application, not just the test files, which is why we make use of the autostart false as well.

Unfortunately I'm bogged down with a product demo and won't be able to address this until the weekend.  If anyone else wants to have a go before then, feel free

I've added a fix for this available in 0.5.1 - check it out and if it addresses your issue we can close this.

cc @avdv
I'm guessing you guys don't need anything more from ember-cli-qunit but if you do, please ping me.  Thanks ^_^

Thanks for your help @hjdivad, always nice to chat with other people working on ember-cli projects =)

Thanks guys, it works as desired. Closing.


too many devices error

i unplugged every single thing in my pc and it still said to many devices or android ones i dont get it pls help

The issues section is not for support.


mp4 to animated avif

Can this be used to convert an mp4 to an animated avif?

Here you go https://www.bmi.com/award-shows/r-and-b-hip-hop-2021/
[https://www.bmi.com/images/news/2021/RB-HH_2021_16-9news.jpg]https://www.bmi.com/award-shows/r-and-b-hip-hop-2021/
2021 BMI R&B/Hip-Hop Awardshttps://www.bmi.com/award-shows/r-and-b-hip-hop-2021/
BMI is excited to celebrate the exceptional songwriters, producers and music publishers whose music has changed the game and raised the bar for R&B and Hip-Hop. It’s our thrill to reveal the winners of the Song of the Year, Songwriter of the Year, Producer of the Year and Publisher of the Year, and to salute our top producers and recognize the 35 most-performed R&B/Hip-Hop songs of the past year.
www.bmi.com
Icarian Records & Publishing
Tel +1<PHONE_NUMBER>
Website https://dylannirvana.comhttps://dylannirvana.com/
SoundCloud https://soundcloud.com/dylannirvana
Disco Library https://library.dylannirvana.com/
Dylan Nirvana. “Like Elvis Costello. Morrissey meets Motörhead” - The Village Voice.

From: Lovell Fuller @.>
Sent: Friday, October 8, 2021 10:12 AM
To: lovell/avif-cli @.>
Cc: Dylan Nirvana @.>; Author @.>
Subject: Re: [lovell/avif-cli] mp4 to animated avif (#5)
No, sadly not, sorry.
I would be interested to know a bit more about the context/scenario where using an AVIF image sequence might be considered more appropriate or a better fit than using an mp4 video.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/lovell/avif-cli/issues/5#issuecomment-938676185, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB7VTGPO7MJAMHVQIAEYCB3UF334LANCNFSM5FTTYOXQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

https://codelabs.developers.google.com/codelabs/avif#5
It's possible.

To chime in about a use case: browsers do not have native support for alpha transparency in H264/MP4 videos, so more esoteric formats are required (HEVC/H265 encoded Quicktime for Safari, and VP9 encoded WebM for everyone else). Since H265 requires licensing and x265 (the HEVC version of the massively popular freeware library called x264) does not support alpha transparency, options for producing well-optimized versions of alpha-channel .mov files for Safari are limited.
In these situations, a small animated AVIF image can be a better tradeoff, even though it's not going to benefit from hardware-accelerated playback or have built-in playback controls.

@coreyward seethru used to be a solution to the alpha transparency problem, but I'm unsure if it's still suitable.

Any chance that we could also see APNG to animated AVIF?

Animated AVIF (sequence) output is dependent on https://github.com/strukturag/libheif/issues/377
Animated APNG input is dependent on https://github.com/randy408/libspng/issues/4
Both are then dependent on the subsequent exposure of these features in libvips and sharp.

It is a little offtiopc but I want to make if helpful for people that also working with animated avif and found this issue.
This library don't support animated avif. I tried to convert gif to avif.
Current state
npx avif --input cat-jump.gif --output avif-cat-jump.avif

dont work. Gives static image.
https://ezgif.com

works
magick convert

works for some formats but not avif

gif to webm -> gives green screen
gif to webp -> works
gif to apng -> gives green screen
gif to avif -> gives static image

ffmpeg -i cat-jump.gif -c:v av1_nvenc -preset fast output.avif

it end with invalid file
 feh output.avif
feh WARNING: output.avif - Does not look like an image (magic bytes missing)

or
fileinfo output.avif
Error in findFileFormatStream: truncated file
Leptonica Error in pixReadHeader: unknown format in file: output.avif
Error in writeImageFileInfo: failure to read header of output.avif

but if we split it on 2 steps and convert gif -> mp4 -> avif, then it works
ffmpeg -hwaccel nvdec -i cat-jump.gif -c:v h264_nvenc -preset fast -an output.mp4
ffmpeg -i output.mp4 -c:v libaom-av1 -cpu-used 8 output.avif

mediainfo output.avif
General
Complete name                            : output.avif
Format                                   : avis
Codec ID                                 : avis (avis/avif/msf1/iso8/mif1/miaf/MA1B)
File size                                : 103 KiB

Video
ID                                       : 1
Format                                   : AV1
Format/Info                              : AOMedia Video 1
Codec ID                                 : av01
Width                                    : 640 pixels
Height                                   : 358 pixels
Display aspect ratio                     : 16:9
Color space                              : YUV
Bit depth                                : 8 bits
Color range                              : Limited
Matrix coefficients                      : BT.470 System B/G

Image
ID                                       : 1
Format                                   : AV1
Format/Info                              : AOMedia Video 1
Format profile<EMAIL_ADDRESS>Codec ID                                 : av01
Width                                    : 640 pixels
Height                                   : 358 pixels
Display aspect ratio                     : 16:9
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Stream size                              : 131 KiB
Color range                              : Limited
Matrix coefficients                      : BT.470 System B/G
Codec configuration box                  : av1C+av1C

I attaching gif on which I tested it.
Thanks to this approach I was able to compress from 700 to 100 K
du -h cat-jump.gif output.mp4 output.avif
704K    cat-jump.gif
716K    output.mp4
104K    output.avif

Also a bit off-topic, but helpful for people who just want a ready-to-go option for Windows to convert GIF animations and WEBP animations to AVIF. To convert to AVIF, the Windows batch script below uses a lossless intermediate format consisting of a series of temporary PNG files. (This does not require encoding to create an intermediate MP4 file.)
This Windows batch script can do the following:

Convert animated GIF to animated AVIF
Convert animated WEBP to animated AVIF
Convert static GIF/WEBP image to static AVIF image.
Optional: Convert static PNG/JPG/JPEG image to static AVIF image.
Optional: For an animation: Add a suffix to output filename which indicates it is an animation.
Optional: Automatically skip overwriting of existing files in the output directory.
Optional: Preserve the file date and time of the input file in the output file.
Optional: Make the output files read-only.
Optional: Delete all existing files from the output directory when this script starts.
Optional: Reduce the size (width x height) of a static image during conversion, if it is bigger than a specified maximum.
See the beginning of the script (after "BEGINNING OF: USER SETTINGS") for these options.

Limitations of this script:

Animations with unequal frame durations are not fully supported. After conversion, each frame will have the same duration and the total duration will be the same as in the original animation.
For animations, all metadata will be lost during the conversion process.

To use this script:
Click the small clipboard icon in the upper right corner of the code block to copy the entire script code to your clipboard. Open Notepad, paste the copied script code, and save the file with a .bat extension. Double-click the .bat file to run the script.
To specify the input directory containing the image/animation files to be converted, or to change the options:
Open the .bat file in Notepad, locate the text after "BEGINNING OF: USER SETTINGS", make the desired changes, and save the .bat file.
The output directory of this script is a subdirectory of its input directory. The output directory will be created if it does not exist.
This script requires certain files to be present on the computer. See at the beginning of the script.
@echo off
setlocal enabledelayedexpansion

rem Script to convert animations and images to AVIF format

rem See "User Settings" below to set the input directory and to set the options.

rem This Windows batch script can do the following:
rem [1] Convert animated GIF to animated AVIF.
rem [2] Convert animated WEBP to animated AVIF.
rem [3] Convert static GIF/WEBP image to static AVIF image.
rem [4] Optional: Convert static PNG/JPG/JPEG image to static AVIF image.
rem [5] Optional: For an animation: Add a suffix to output filename which indicates it is an animation.
rem [6] Optional: Automatically skip overwriting of existing files in the output directory.
rem [7] Optional: Preserve the file date and time of the input file in the output file.
rem [8] Optional: Make the output files read-only.
rem [9] Optional: Delete all existing files from the output directory when this script starts.
rem [10] Optional: Reduce the size (width x height) of a static image during conversion, if it is bigger than a specified maximum.

rem This script requires certain files to be present on the computer. The following programs should be available and their location should be added to PATH in the system environment variables of Windows:
rem [1] FFmpeg (only this file is required: ffmpeg.exe) -- [https://phoenixnap.com/kb/ffmpeg-windows].
rem [2] ImageMagick (only this file is required: magick.exe) -- [https://imagemagick.org/script/download.php].
rem [3] ExifTool (only this file is required: exiftool.exe) -- [https://oliverbetz.de/pages/Artikel/ExifTool-for-Windows].
rem [4] NirCmd (only this file is required: nircmdc.exe) - Only needed for the option to preserve the file date and time. For this, download "NirCmd" from [https://www.nirsoft.net/utils/nircmd.html] and put "nircmdc.exe" in "C:\Windows".

rem Note: The output directory is a subdirectory of the input directory.
rem Note: The output directory will be created if it does not exist.
rem Note: Limitation 1: Animations with unequal frame durations are not fully supported. After conversion, each frame will have the same duration and the total duration will be the same as in the original animation.
rem Note: Limitation 2: For animations, all metadata will be lost during the conversion process.

rem BEGINNING OF: USER SETTINGS

rem [1] Path to the input directory containing your image/animation files that should be converted
set "input_dir=D:\Images to convert"
Rem Note: The output directory is by default a subdirectory of the input directory, which will be created if it doesn't exist
Rem Note: See below for the name of the output directory

rem [2] Decide whether or not to also convert JPG, JPEG, and PNG files to AVIF files
set "convert_jpg_jpeg_png=true"  Rem If true, also convert jpg, jpeg and png files
rem Note: GIF and WEBP files are converted by default to AVIF files

rem [3] Only for files that contain an animation: Decide whether or not to add a suffix to the output filename (indicating that it is an animation)
set "add_suffix=true"  Rem If true, add a suffix to the output file name of an animation
Rem Note: See below for the text of this suffix

rem [4] Decide whether or not to automatically skip overwriting of existing files in the output directory
set "always_skip_overwriting=false"  Rem If true, always automatically skip overwriting of existing files in the output directory
rem Note: If not true, users will be asked for each existing file whether to overwrite it or not

rem [5] Decide whether or not to always preserve the file date and time of the input file in the output file
set "keep_file_date_time=false"  Rem If true, keep file date and time of input file in output file

rem [6] Decide whether or not to always make each output file read-only
set "make_output_files_read_only=false"  Rem If true, make each output file read-only

rem [7] Decide whether or not to always delete all existing files from the output directory when this script starts
set "del_output_dir=false"  Rem If true, delete existing files from output directory when this script starts

rem [8] Decide whether or not to reduce the size of static images, if their area (width x height) is bigger than a specified maximum (see below for this maximum)
set "reduce_size_of_large_static_images=false"  Rem If true, reduce the size of large static images
rem Note: The size of an image will be reduced to just under the specified maximum

rem [9] If reducing the size of static images, decide what their maximum allowed area (width x height) is
set "max_area=2073600"  rem Example value: 2073600 (= 1920 x 1080)
rem Note: If an image is just over the maximum, its size will not be reduced

rem END OF: USER SETTINGS

rem Path to the output directory for your converted image/animation files
set "output_dir=!input_dir!\Output to AVIF"  Rem Output directory is subdirectory of input directory

rem Notify conversion starting
echo Conversion starting...

rem Check for required programs
set "missing_programs="
set "missing_count=0"

for %%p in (ffmpeg magick exiftool) do (
where %%p >nul 2>&1
if errorlevel 1 (
if "!missing_programs!"=="" (
set "missing_programs=%%p.exe"
) else (
set "missing_programs=!missing_programs!, %%p.exe"
)
set /a missing_count+=1
)
)

rem Display error message if required program is not available
if not "!missing_programs!"=="" (
echo.
if !missing_count! gtr 1 (
echo Error: The following required programs are not available: !missing_programs!
echo Please ensure these programs are installed and available in the PATH.
) else (
echo Error: The following required program is not available: !missing_programs!
echo Please ensure this program is installed and available in the PATH.
)
echo.
goto :end
)

rem Check for the program nircmdc.exe
if /i "!keep_file_date_time!"=="true" (
where nircmdc >nul 2>&1
if errorlevel 1 (
set "continue=N"   rem Initialize continue
echo Warning: The program nircmdc.exe is not available. The file date and time of the input file will not be preserved in the output file.
set /p "continue=Continue? [y/N]: "
if /i "!continue!" neq "y" (
goto :end
)
)
)

rem Check if the output directory exists
if exist "!output_dir!" (
if /i "!del_output_dir!"=="true" (
rem Remove the output directory and all its contents
rmdir /s /q "!output_dir!"
)
)

rem Create the output directory if it does not exist
if not exist "!output_dir!" (
mkdir "!output_dir!"
)

rem Set text to append to output files that contain an animation
if /i "!add_suffix!"=="true" (
set "ani_suffix= [ANI]"
rem Determine the length of the suffix
for /l %%i in (12,1,100) do if "!ani_suffix:~%%i,1!"=="" set "ani_suffix_length=%%i" & goto :suffix_length_determined
)
:suffix_length_determined

rem Define temporary subdirectory in the temp folder
set "temp_dir=%TEMP%\gif_webp_temp"
if not exist !temp_dir! mkdir !temp_dir!

rem Count total number of relevant files
set "count=0"
for %%f in ("!input_dir!\*.gif") do (set /a count+=1)
for %%f in ("!input_dir!\*.webp") do (set /a count+=1)
if /i "!convert_jpg_jpeg_png!"=="true" (
for %%f in ("!input_dir!\*.jpg" "!input_dir!\*.jpeg" "!input_dir!\*.png") do (set /a count+=1)
)

rem Initialize progress counter
set "current=0"

rem Initialize input_files_found
set "input_files_found=false"

rem START PROCESSING OF GIF FILES

rem Loop through all gif files in the input directory
for %%f in ("!input_dir!\*.gif") do (
set "filename=%%~nf"
set "extension=%%~xf"
set /a current+=1
set "input_files_found=true"

rem Store the input file name and path in a new variable
set "input_file=%%f"

rem Convert file to a series of png files in temp directory using ImageMagick (suppress messages)
magick "!input_file!" -coalesce "!temp_dir!\!filename!_frame_%%03d.png" 2>NUL

rem Check if there are any png files in the temp directory
dir /b "!temp_dir!\!filename!_frame_*.png" >nul 2>&1
if errorlevel 1 (
set "frames=0"
) else (
rem Count the number of png files in the temp directory to determine the number of frames
for /f %%b in ('dir /b "!temp_dir!\!filename!_frame_*.png" ^| find /c /v ""') do set "frames=%%b"
)

if "!frames!" == "0" (
rem Unable to extract frames from this file
echo !current! of !count! - GIF - !filename! - FILE [PARTIALLY] CORRUPT - CONVERSION FAILED

rem Create output filename with error indication
set "output_filename=!filename! [[[[[[ERROR]]]]]]"

if not exist "!output_dir!\!output_filename!.avif" (
rem Create empty .avif file in the output directory as a warning that the conversion has failed
type nul > "!output_dir!\!output_filename!.avif"
if /i "!keep_file_date_time!"=="true" (nircmdc clonefiletime "!input_file!" "!output_dir!\!output_filename!.avif" >nul 2>&1)
if /i "!make_output_files_read_only!"=="true" (attrib +r "!output_dir!\!output_filename!.avif")
)

) else (

rem If the number of frames is greater than one, then the file is an animation
if !frames! gtr 1 (

rem BEGINNING OF: PROCESS GIF ANIMATION

rem BEGINNING OF: DETERMINE DURATION AND FRAMERATE

rem Read the duration of the file using ExifTool (suppress warnings)
for /f "tokens=*" %%i in ('exiftool -Duration "!input_file!" 2^>nul') do set "duration=%%i"

rem Beginning of cleanup to get duration in milliseconds
rem If the duration is an empty string, set its value to zero
if "!duration!"=="" (
set "duration=0"
) else (
rem Remove "Duration                        : " from the variable
set "duration=!duration:Duration                        : =!"
rem Remove " s" from the variable
set "duration=!duration: s=!"
rem Remove the period from the variable
set "duration=!duration:.=!"
rem Check if the variable starts with a zero. If it does, remove this zero
if "!duration:~0,1!" == "0" set "duration=!duration:~1!"
rem Add "0" to the end of the variable. After this, the duration is expressed in milliseconds.
set "duration=!duration!0"
rem Check if the value of the variable is valid. If not, set its value to zero
rem Check if the variable now contains only digits. If not, set its value to zero
for /f "delims=0123456789" %%a in ("!duration!") do (set "duration=0")
)
rem Store the duration in milliseconds in a new variable
set "duration_ms=!duration!"
rem End of cleanup to get duration in milliseconds

rem Calculate the framerate from the duration and number of frames
rem Note: duration_ms is in milliseconds, which is 1000 times smaller than seconds
if !duration_ms! neq 0 (
set /a "framerate=(frames * 1000) / duration_ms"
rem Use remainder to improve rounding of framerate (if necessary)
set /a "remainder=(frames * 1000) %% duration_ms"
set /a "half_duration=duration_ms / 2"
if !remainder! geq !half_duration! (set /a framerate+=1)
)

Rem Set initial values
set "add_warning_suffix=false"
set "duration_is_unknown=false"

rem In a partially corrupted file, the read duration in milliseconds may be too low or negative
rem In a partially corrupted file, the calculated frame rate may be too low or negative
if !duration_ms! lss 8 (
set "result=problem"
) else if !framerate! lss 3 (
set "result=problem"
) else (
set "result=no_problem"
)
if "!result!"=="problem" (
set "framerate=14"  rem Default to 14 fps, if duration is less than 8 ms or framerate is less than 3 fps
set "add_warning_suffix=true"
set "duration_is_unknown=true"
set "warning_suffix= [[[WARNING - Duration may differ from original]]]"
)

rem END OF: DETERMINE DURATION AND FRAMERATE

rem Notify file being processed
if /i "!duration_is_unknown!"=="true" (
echo !current! of !count! - GIF - !filename! - !frames! ... !framerate! [frames, duration in ms, framerate] - Duration could not be determined, framerate set to default
) else (
echo !current! of !count! - GIF - !filename! - !frames! !duration_ms! !framerate! [frames, duration in ms, framerate]
)

rem If add_suffix is true and filename does not already end with ani_suffix, add it
if /i "!add_suffix!"=="true" (
if "!filename:~-!ani_suffix_length!!" neq "!ani_suffix!" (
set "output_filename=!filename!!ani_suffix!"
) else (
set "output_filename=!filename!"
)
) else (
set "output_filename=!filename!"
)

rem Add warning_suffix for exceptional cases (see above)
if /i "!add_warning_suffix!"=="true" (
set "output_filename=!output_filename!!warning_suffix!"
)

rem Check whether to always automatically skip overwriting of existing files in the output directory
set "skip_automatically=false"
if exist "!output_dir!\!output_filename!.avif" (
if /i "!always_skip_overwriting!"=="true" (
echo       Automatically skipping existing file "!output_dir!\!output_filename!.avif"
set "skip_automatically=true"
)
)

rem Decide whether to convert a series of png files to an animated avif
rem If conditions are met, do the conversion
call :decide_whether_to_convert_png_files_to_animated_avif

rem Delete the temporary png files
del "!temp_dir!\!filename!_frame_*.png"

) else (

rem END OF: PROCESS GIF ANIMATION

rem BEGINNING OF: PROCESS STATIC GIF IMAGE

rem Here, the number of frames is equal to one
rem If the number of frames is equal to one, then the file is a static image

rem Get width and height of the current static gif using ImageMagick
for /f "tokens=1,2 delims= " %%d in ('magick identify -format "%%w %%h" "!input_file!"') do (
set "width=%%d"
set "height=%%e"
)

rem Set initial values
set "reduce_image_area_size=false"
set "new_width=!width!"
set "new_height=!height!"

rem Call the subroutine to decide whether to reduce the area (width x height) of the current image, and if so, calculate new dimensions (new_width, new_height)
call :calculate_dimensions

rem Notify file being processed
if /i "!reduce_image_area_size!"=="true" (
echo !current! of !count! - GIF - !filename! - !width!x!height! - Reduced to !new_width!x!new_height!
) else (
echo !current! of !count! - GIF - !filename! - !width!x!height!
)

rem Set filename for output
set "output_filename=!filename!"

rem Add reduction_suffix if size of image area will be reduced
if /i "!reduce_image_area_size!"=="true" (
set "reduction_suffix= [REDUCED]"
set "output_filename=!filename!!reduction_suffix!"
)

rem Check whether to always automatically skip overwriting of existing files in the output directory
set "skip_automatically=false"
if exist "!output_dir!\!output_filename!.avif" (
if /i "!always_skip_overwriting!"=="true" (
echo       Automatically skipping existing file "!output_dir!\!output_filename!.avif"
set "skip_automatically=true"
)
)

rem Convert static image to avif if conditions are met
call :decide_whether_to_convert_static_image_to_avif
)
)
)

rem END OF: PROCESS STATIC GIF IMAGE

rem START PROCESSING OF WEBP FILES

rem Loop through all webp files in the input directory
for %%f in ("!input_dir!\*.webp") do (
set "filename=%%~nf"
set "extension=%%~xf"
set /a current+=1
set "input_files_found=true"

rem Store the input file name and path in a new variable
set "input_file=%%f"

rem Convert file to a series of png files in temp directory using ImageMagick (suppress messages)
magick "!input_file!" -coalesce "!temp_dir!\!filename!_frame_%%03d.png" 2>NUL

rem Check if there are any png files in the temp directory
dir /b "!temp_dir!\!filename!_frame_*.png" >nul 2>&1
if errorlevel 1 (
set "frames=0"
) else (
rem Count the number of png files in the temp directory to determine the number of frames
for /f %%b in ('dir /b "!temp_dir!\!filename!_frame_*.png" ^| find /c /v ""') do set "frames=%%b"
)

if "!frames!" == "0" (
rem Unable to extract frames from this file
echo !current! of !count! - GIF - !filename! - FILE [PARTIALLY] CORRUPT - CONVERSION FAILED

rem Create output filename with error indication
set "output_filename=!filename! [[[[[[ERROR]]]]]]"

if not exist "!output_dir!\!output_filename!.avif" (
rem Create empty .avif file in the output directory as a warning that the conversion has failed
type nul > "!output_dir!\!output_filename!.avif"
if /i "!keep_file_date_time!"=="true" (nircmdc clonefiletime "!input_file!" "!output_dir!\!output_filename!.avif" >nul 2>&1)
if /i "!make_output_files_read_only!"=="true" (attrib +r "!output_dir!\!output_filename!.avif")
)

) else (

rem If the number of frames is greater than one, then the file is an animation
if !frames! gtr 1 (

rem BEGINNING OF: PROCESS WEBP ANIMATION

rem BEGINNING OF: DETERMINE DURATION AND FRAMERATE

rem Read the duration of the file using ExifTool (suppress warnings)
for /f "tokens=*" %%i in ('exiftool -Duration "!input_file!" 2^>nul') do set "duration=%%i"

rem Beginning of cleanup to get duration in milliseconds
rem If the duration is an empty string, set its value to zero
if "!duration!"=="" (
set "duration=0"
) else (
rem Remove "Duration                        : " from the variable
set "duration=!duration:Duration                        : =!"
rem Remove " s" from the variable
set "duration=!duration: s=!"
rem Remove the period from the variable
set "duration=!duration:.=!"
rem Check if the variable starts with a zero. If it does, remove this zero
if "!duration:~0,1!" == "0" set "duration=!duration:~1!"
rem Add "0" to the end of the variable. After this, the duration is expressed in milliseconds.
set "duration=!duration!0"
rem Check if the value of the variable is valid. If not, set its value to zero
rem Check if the variable now contains only digits. If not, set its value to zero
for /f "delims=0123456789" %%a in ("!duration!") do (set "duration=0")
)
rem Store the duration in milliseconds in a new variable
set "duration_ms=!duration!"
rem End of cleanup to get duration in milliseconds

rem Calculate the framerate from the duration and number of frames
rem Note: duration_ms is in milliseconds, which is 1000 times smaller than seconds
if !duration_ms! neq 0 (
set /a "framerate=(frames * 1000) / duration_ms"
rem Use remainder to improve rounding of framerate (if necessary)
set /a "remainder=(frames * 1000) %% duration_ms"
set /a "half_duration=duration_ms / 2"
if !remainder! geq !half_duration! (set /a framerate+=1)
)

Rem Set initial values
set "add_warning_suffix=false"
set "duration_is_unknown=false"

rem In a partially corrupted file, the read duration in milliseconds may be too low or negative
rem In a partially corrupted file, the calculated frame rate may be too low or negative
if !duration_ms! lss 8 (
set "result=problem"
) else if !framerate! lss 3 (
set "result=problem"
) else (
set "result=no_problem"
)
if "!result!"=="problem" (
set "framerate=14"  rem Default to 14 fps, if duration is less than 8 ms or framerate is less than 3 fps
set "add_warning_suffix=true"
set "duration_is_unknown=true"
set "warning_suffix= [[[WARNING - Duration may differ from original]]]"
)

rem END OF: DETERMINE DURATION AND FRAMERATE

rem Notify file being processed
if /i "!duration_is_unknown!"=="true" (
echo !current! of !count! - WEBP - !filename! - !frames! ... !framerate! [frames, duration in ms, framerate] - Duration could not be determined, framerate set to default
) else (
echo !current! of !count! - WEBP - !filename! - !frames! !duration_ms! !framerate! [frames, duration in ms, framerate]
)

rem If add_suffix is true and filename does not already end with ani_suffix, add it
if /i "!add_suffix!"=="true" (
if "!filename:~-!ani_suffix_length!!" neq "!ani_suffix!" (
set "output_filename=!filename!!ani_suffix!"
) else (
set "output_filename=!filename!"
)
) else (
set "output_filename=!filename!"
)

rem Add warning_suffix for exceptional cases (see above)
if /i "!add_warning_suffix!"=="true" (
set "output_filename=!output_filename!!warning_suffix!"
)

rem Check whether to always automatically skip overwriting of existing files in the output directory
set "skip_automatically=false"
if exist "!output_dir!\!output_filename!.avif" (
if /i "!always_skip_overwriting!"=="true" (
echo       Automatically skipping existing file "!output_dir!\!output_filename!.avif"
set "skip_automatically=true"
)
)

rem Decide whether to convert a series of png files to an animated avif
rem If conditions are met, do the conversion
call :decide_whether_to_convert_png_files_to_animated_avif

rem Delete the temporary png files
del "!temp_dir!\!filename!_frame_*.png"

) else (

rem END OF: PROCESS WEBP ANIMATION

rem BEGINNING OF: PROCESS STATIC WEBP IMAGE

rem Here, the number of frames is equal to one
rem If the number of frames is equal to one, then the file is a static image

rem Get width and height of the current static webp using ImageMagick
for /f "tokens=1,2 delims= " %%d in ('magick identify -format "%%w %%h" "!input_file!"') do (
set "width=%%d"
set "height=%%e"
)

rem Set initial values
set "reduce_image_area_size=false"
set "new_width=!width!"
set "new_height=!height!"

rem Call the subroutine to decide whether to reduce the area (width x height) of the current image, and if so, calculate new dimensions (new_width, new_height)
call :calculate_dimensions

rem Notify file being processed
if /i "!reduce_image_area_size!"=="true" (
echo !current! of !count! - WEBP - !filename! - !width!x!height! - Reduced to !new_width!x!new_height!
) else (
echo !current! of !count! - WEBP - !filename! - !width!x!height!
)

rem Set filename for output
set "output_filename=!filename!"

rem Add reduction_suffix if size of image area will be reduced
if /i "!reduce_image_area_size!"=="true" (
set "reduction_suffix= [REDUCED]"
set "output_filename=!filename!!reduction_suffix!"
)

rem Check whether to always automatically skip overwriting of existing files in the output directory
set "skip_automatically=false"
if exist "!output_dir!\!output_filename!.avif" (
if /i "!always_skip_overwriting!"=="true" (
echo       Automatically skipping existing file "!output_dir!\!output_filename!.avif"
set "skip_automatically=true"
)
)

rem Convert static image to avif if conditions are met
call :decide_whether_to_convert_static_image_to_avif
)
)
)

rem END OF: PROCESS STATIC WEBP IMAGE

rem Remove the temporary subdirectory
rmdir /s /q !temp_dir!

rem BEGINNING OF: PROCESS JPG, JPEG, AND PNG IMAGE

rem Check if also to convert jpg, jpeg, and png files to avif files
if /i "!convert_jpg_jpeg_png!"=="true" (

rem Loop through all jpg, jpeg, and png files in the input directory
for %%f in ("!input_dir!\*.jpg" "!input_dir!\*.jpeg" "!input_dir!\*.png") do (
set "filename=%%~nf"
set "extension=%%~xf"
set /a current+=1
set "input_files_found=true"

rem Store the input file name and path in a new variable
set "input_file=%%f"

rem Get width and height of the current image file using ImageMagick
for /f "tokens=1,2 delims= " %%d in ('magick identify -format "%%w %%h" "!input_file!"') do (
set "width=%%d"
set "height=%%e"
)

rem Set initial values
set "reduce_image_area_size=false"
set "new_width=!width!"
set "new_height=!height!"

rem Call the subroutine to decide whether to reduce the area (width x height) of the current image, and if so, calculate new dimensions (new_width, new_height)
call :calculate_dimensions

rem Notify file being processed
if /i "!reduce_image_area_size!"=="true" (
if /i "!extension!"==".jpg" (
echo !current! of !count! - JPG - !filename! - !width!x!height! - Reduced to !new_width!x!new_height!
) else if /i "!extension!"==".jpeg" (
echo !current! of !count! - JPEG - !filename! - !width!x!height! - Reduced to !new_width!x!new_height!
) else if /i "!extension!"==".png" (
echo !current! of !count! - PNG - !filename! - !width!x!height! - Reduced to !new_width!x!new_height!
)
) else (
if /i "!extension!"==".jpg" (
echo !current! of !count! - JPG - !filename! - !width!x!height!
) else if /i "!extension!"==".jpeg" (
echo !current! of !count! - JPEG - !filename! - !width!x!height!
) else if /i "!extension!"==".png" (
echo !current! of !count! - PNG - !filename! - !width!x!height!
)
)

rem Set filename for output
set "output_filename=!filename!"

rem Add reduction_suffix if size of image area will be reduced
if /i "!reduce_image_area_size!"=="true" (
set "reduction_suffix= [REDUCED]"
set "output_filename=!filename!!reduction_suffix!"
)

rem Check whether to always automatically skip overwriting of existing files in the output directory
set "skip_automatically=false"
if exist "!output_dir!\!output_filename!.avif" (
if /i "!always_skip_overwriting!"=="true" (
echo       Automatically skipping existing file "!output_dir!\!output_filename!.avif"
set "skip_automatically=true"
)
)

rem Convert static image to avif if conditions are met
call :decide_whether_to_convert_static_image_to_avif
)

rem END OF: PROCESS JPG, JPEG, AND PNG IMAGE

rem BEGINNING OF: ADD ERROR MESSAGE TO ZERO BYTES FILES

rem Add an error message to the names of files in the output directory that are zero bytes in size (if the two conditions below are met)
rem Note: The conversion of partially corrupted files, may result in files that are zero bytes in size
for %%A in ("!output_dir!\*.*") do (
if %%~zA == 0 (
set "filename=%%~nA"
set "extension=%%~xA"

rem [1] Do not add " [[[[[[ERROR]]]]]]" to filenames that already end with " [[[[[[ERROR]]]]]]"
rem Check if the last 18 characters of the filename are " [[[[[[ERROR]]]]]]"
set "last18=!filename:~-18!"
if "!last18!" neq " [[[[[[ERROR]]]]]]" (
set "testname=!filename! [[[[[[ERROR]]]]]]!extension!"

rem [2] Do not add " [[[[[[ERROR]]]]]]" to a filename, if this results in a filename that already exists
rem Check if the new name already exists in the output directory
if not exist "!output_dir!\!testname!" (

rem Rename the file only if both conditions are met, but first remove a warning if one exists
rem Check if the filename ends with " [[[WARNING - Duration may differ from original]]]"
set "last50=!filename:~-50!"
if "!last50!"==" [[[WARNING - Duration may differ from original]]]" (
rem Remove " [[[WARNING - Duration may differ from original]]]" from the filename
set "filename=!filename:~0,-50!"
)

rem Rename the file only if both conditions are met
set "newname=!filename! [[[[[[ERROR]]]]]]!extension!"
rem Renaming "%%A" to "!newname!"
ren "%%A" "!newname!"

) else (
rem echo File "!newname!" already exists. Skipping renaming.
)
) else (
rem echo File "%%A" already contains the error message. Skipping renaming.
)
)
)

rem END OF: ADD ERROR MESSAGE TO ZERO BYTES FILES

if "!input_files_found!"=="false" (
echo.
echo No relevant image/animation files found in the input directory. Please set the input directory with your image/animation files in this script [do this after "BEGINNING OF: USER SETTINGS" in this script], and/or place relevant image/animation files in the input directory.
echo.
)

goto :end

rem BEGINNING OF: Subroutine to decide whether to convert a series of png files to an animated avif
:decide_whether_to_convert_png_files_to_animated_avif
rem Convert the series of png files to an animated avif using the calculated framerate
rem Only write file if it doesn't exist or if permission to write is given and existing file is not read-only
if /i "!skip_automatically!"=="false" (
rem ----------------------------------------
rem Check if file exists in output directory
if exist "!output_dir!\!output_filename!.avif" (
set "confirm=N"   rem Initialize confirm
<nul set /p="_____ File "!output_filename!.avif" already exists. Overwrite? [y/N] "
set /p confirm=
if /i "!confirm!"=="" set "confirm=N"    rem Pressing Enter is here equivalent to entering "N".
if /i "!confirm!"=="y" (
rem ----------------------------------------
rem Check if file is read-only
call :is_readonly "!output_dir!\!output_filename!.avif"
if defined readonly (
echo       Permission to overwrite is denied, because existing file is read-only
echo       Skipping file "!output_filename!.avif"
rem ----------------------------------------
) else (
echo       Overwriting file "!output_filename!.avif"
rem Convert a series of png files to an animated avif using FFmpeg
call :convert_png_files_to_animated_avif
)
rem ----------------------------------------
) else (
echo       Skipping file "!output_filename!.avif"
)
rem ----------------------------------------
) else (
rem Convert a series of png files to an animated avif using FFmpeg
call :convert_png_files_to_animated_avif
)
)
exit /b
rem END OF: Subroutine to decide whether to convert a series of png files to an animated avif

rem BEGINNING OF: Subroutine to convert a series of png files to an animated avif using FFmpeg
:convert_png_files_to_animated_avif
rem Convert series of png files to avif animation using FFmpeg
ffmpeg -y -framerate !framerate! -i "!temp_dir!\!filename!_frame_%%03d.png" -c:v libaom-av1 -cpu-used 8 -pix_fmt yuv420p -loglevel error "!output_dir!\!output_filename!.avif"
rem ----------------------------------------
if /i "!keep_file_date_time!"=="true" (nircmdc clonefiletime "!input_file!" "!output_dir!\!output_filename!.avif" >nul 2>&1)
if /i "!make_output_files_read_only!"=="true" (attrib +r "!output_dir!\!output_filename!.avif")
exit /b
rem END OF: Subroutine to convert a series of png files to an animated avif using FFmpeg

rem BEGINNING OF: Subroutine to decide whether to convert static image to avif
:decide_whether_to_convert_static_image_to_avif
rem Convert static image to avif if conditions are met
rem Only write file if it doesn't exist or if permission to write is given and existing file is not read-only
if /i "!skip_automatically!"=="false" (
rem ----------------------------------------
rem Check if file exists in output directory
if exist "!output_dir!\!output_filename!.avif" (
set "confirm=N"   rem Initialize confirm
<nul set /p="_____ File "!output_filename!.avif" already exists. Overwrite? [y/N] "
set /p confirm=
if /i "!confirm!"=="" set "confirm=N"    rem Pressing Enter is here equivalent to entering "N".
if /i "!confirm!"=="y" (
rem ----------------------------------------
rem Check if file is read-only
call :is_readonly "!output_dir!\!output_filename!.avif"
if defined readonly (
echo       Permission to overwrite is denied, because existing file is read-only
echo       Skipping file "!output_filename!.avif"
rem ----------------------------------------
) else (
echo       Overwriting file "!output_filename!.avif"
rem Convert a static image to avif using ImageMagick
call :convert_static_image_to_avif
)
rem ----------------------------------------
) else (
echo       Skipping file "!output_filename!.avif"
)
rem ----------------------------------------
) else (
rem Convert a static image to avif using ImageMagick
call :convert_static_image_to_avif
)
)
)
)
exit /b
rem END OF: Subroutine to decide whether to convert static image to avif

rem BEGINNING OF: Subroutine to convert static image to avif using ImageMagick
:convert_static_image_to_avif
if /i "!reduce_image_area_size!"=="true" (
rem Convert static image to avif and resize using ImageMagick
magick "!input_file!" -resize !new_width!x!new_height! -define heic:speed=5 "!output_dir!\!output_filename!.avif"
rem ----------------------------------------
) else (
rem Convert static image to avif using ImageMagick
magick "!input_file!" -define heic:speed=5 "!output_dir!\!output_filename!.avif"
rem ----------------------------------------
)
if /i "!keep_file_date_time!"=="true" (nircmdc clonefiletime "!input_file!" "!output_dir!\!output_filename!.avif" >nul 2>&1)
if /i "!make_output_files_read_only!"=="true" (attrib +r "!output_dir!\!output_filename!.avif")
exit /b
rem END OF: Subroutine to convert static image to avif using ImageMagick

rem BEGINNING OF: Subroutine to check if a file is read-only
:is_readonly
set "readonly="
rem echo Checking file: "%~1"
for /f "tokens=*" %%A in ('attrib "%~1"') do (
set "attrib_output=%%A"
)
set "attrib_output=%attrib_output:~0,13%"
rem echo Attrib output: "%attrib_output%"
if "%attrib_output%"=="%attrib_output:R=%" (
rem echo No "R" found in attrib output.
) else (
set "readonly=true"
)
exit /b
rem END OF: Subroutine to check if a file is read-only

rem BEGINNING OF: Subroutine to decide whether to reduce the area (width x height) of the current image, and if so, calculate new dimensions (new_width, new_height)
:calculate_dimensions

rem BEGINNING OF: DECIDE WHETHER TO REDUCE AREA OF IMAGE (IF SIGNIFICANT DIFFERENCE)
rem Check whether to reduce the area (width x height) of large images during conversion to avif

rem Calculate the area of the image
set /a area=!width!*!height!

rem Compare area to max_area
if !area! gtr !max_area! (
set "area_greater_than_max_area=true"
) else (
set "area_greater_than_max_area=false"
)

rem Check whether to reduce the area (width x height) of the image
rem Will only reduce area of image, if relative difference with max_area is significant
set "reduce_image_area_size_if_difference_is_significant=false" rem Initial value
if /i "!reduce_size_of_large_static_images!"=="true" (
if /i "!area_greater_than_max_area!"=="true" (
set "reduce_image_area_size_if_difference_is_significant=true"
)
)

rem END OF: DECIDE WHETHER TO REDUCE AREA OF IMAGE (IF SIGNIFICANT DIFFERENCE)

rem BEGINNING OF: CALCULATE SCALE FACTOR TIMES THOUSAND

if /i "!reduce_image_area_size_if_difference_is_significant!"=="true" (
rem The intention is to calculate this: scale_factor_times_thousand=1000*sqrt(!max_area!/!area!)
rem However, a Windows batch script cannot calculate a square root directly, so it cannot be calculated in this way
rem Therefore, the square root will calculated by an iterative approximation method
rem The multiplication by 1000 and extra calculation steps, are due to the limited computational capabilities of a Windows batch script

rem First preparation step to calculate square root
set /a thousand_times_division=1000*!max_area!/!area!

rem Second preparation step to calculate square root
set /a million_times_division=1000*!thousand_times_division!

rem Note: The scale_factor_times_thousand to be calculated, is equal to the square root of million_times_division

rem This is the number for which to calculate the square root
set /a number=million_times_division

rem Initialize variables for iterative method
set /a guess=number / 2
set /a epsilon=1
set /a diff=number

rem Calculate the square root using an iterative method
call :iterate

rem Store the iteratively calculated square root in scale_factor_times_thousand
set /a scale_factor_times_thousand=guess

rem END OF: CALCULATE SCALE FACTOR TIMES THOUSAND

rem Very small relative differences between area and max_area will be ignored
rem Only reduce image, if value of scale_factor_times_thousand is less than 985
rem Note: A value of scale_factor_times_thousand of 985, equals 98.5% of max_area
if !scale_factor_times_thousand! lss 985 (

set "reduce_image_area_size=true"

rem BEGINNING OF: CALCULATE NEW WIDTH AND NEW HEIGHT

rem Note: Later the area of the image will be reduced to become smaller than the maximum area
rem Use scale factor (x 1000) to calculate new_width (x 1000) and new_height (x 1000)
set /a new_width_times_thousand=!width!*!scale_factor_times_thousand!
set /a new_height_times_thousand=!height!*!scale_factor_times_thousand!

rem Calculate new_width
set /a new_width=!new_width_times_thousand!/1000
rem Use remainder to improve rounding of new_width (if necessary)
set /a remainder=!new_width_times_thousand! %% 1000
set /a half_divisor=1000 / 2
if !remainder! geq !half_divisor! (set /a new_width+=1)

rem Calculate new_height
set /a new_height=!new_height_times_thousand!/1000
rem Use remainder to improve rounding of new_height (if necessary)
set /a remainder=!new_height_times_thousand! %% 1000
set /a half_divisor=1000 / 2
if !remainder! geq !half_divisor! (set /a new_height+=1)
)
)

rem END OF: CALCULATE NEW WIDTH AND NEW HEIGHT
exit /b
rem END OF: Subroutine to decide whether to reduce the area (width x height) of the current image, and if so, calculate dimensions (new_width, new_height)

rem BEGINNING OF: Subroutine to calculate the square root of a number using an iterative method
rem This code has been placed outside of any loop to prevent unwanted changes to the value of variables in a loop.
:iterate
set /a guess_plus_number_devided_by_guess = guess + number / guess
set /a new_guess=guess_plus_number_devided_by_guess / 2
set /a diff=new_guess - guess
if !diff! lss 0 set /a diff=-!diff!
set /a guess=new_guess
if !diff! gtr !epsilon! goto iterate
exit /b
rem END OF: Subroutine to calculate the square root of a number using an iterative method

:end

endlocal
echo Conversion complete
pause

Closing as this issue is attracting LLM slop. Please see the issues linked to in https://github.com/lovell/avif-cli/issues/5#issuecomment-1471494867


Typing error: onError is now required instead of optional

Hi there,
looks like the tying now requires an onError callback. Is this intentional? Looking at he code it seems that it should be optional?
Best regards,
CK

Thank you for fixing :-)

Thank you for the heads-up, 2.0.8 contains the fix.


*** Bundle has failed to load

Hello, John.
Please, help.
During injection I'm receiving such error:
Xcode console log:
Compiling /Users/sergei.prikhodko/PR/Project/ProjectLive/Code/Managers/NetworkStatusManager.m         0.05 real         0.03 user         0.01 sys 2016-10-10 10:20:48.948 Project[9481:140869] Error loading /Users/sergei.prikhodko/PR/Project/iOSInjectionProject/build/Debug-iphonesimulator/InjectionBundle5.bundle/InjectionBundle:  dlopen(/Users/sergei.prikhodko/PR/Project/iOSInjectionProject/build/Debug-iphonesimulator/InjectionBundle5.bundle/InjectionBundle, 265): Library not loaded: @rpath/AFNetworkActivityLogger.framework/AFNetworkActivityLogger   Referenced from: /Users/sergei.prikhodko/PR/Project/iOSInjectionProject/build/Debug-iphonesimulator/InjectionBundle5.bundle/InjectionBundle   Reason: image not found 2016-10-10 10:20:48.949 Project[9481:140869] Bundle Load Error 2016-10-10 10:20:48.949 Project[9481:141010] *** Bundle has failed to load. If this is due to symbols not found, this may be due to symbols being hidden from dynamic libraries. *** 4 injections performed so far.
Where shall I have a look?

You’ll likely have to adjust the “Runpath Search Paths” of your injection bundle.

John, could you give an example of how to modify those paths?  I've installed Google/SignIn via cocoapods and can no longer inject with the error:
ld: framework not found GGLSignIn for architecture x86_64

clang: error: linker command failed with exit code 1 (use -v to see invocation)

** BUILD FAILED **

The following build commands failed:
	Ld build/Debug-iphonesimulator/InjectionBundle.bundle/InjectionBundle normal x86_64
(1 failure)

*** Build Failed with status: 65. You may need to open and edit the bundle project to resolve issues with either header include paths or Frameworks the bundle links against. ***

@johnno1962 I works inly when I "patch project for injection".
otherwerwise xcode console says:
Error loading /Users/user/Library/Application Support/Developer/Shared/Xcode/Plug-ins/InjectionPlugin.xcplugin/Contents/Resources/InjectionLoader.bundle/InjectionLoader:  dlopen(/Users/user/Library/Application Support/Developer/Shared/Xcode/Plug-ins/InjectionPlugin.xcplugin/Contents/Resources/InjectionLoader.bundle/InjectionLoader, 265): no suitable image found.  Did find:
	/Users/user/Library/Application Support/Developer/Shared/Xcode/Plug-ins/InjectionPlugin.xcplugin/Contents/Resources/InjectionLoader.bundle/InjectionLoader: required code signature missing for '/Users/user/Library/Application Support/Developer/Shared/Xcode/Plug-ins/InjectionPlugin.xcplugin/Contents/Resources/InjectionLoader.bundle/InjectionLoader'

How can I use plugin without modification in main.m and .pch files?

Hi @dollar2048, you still have a problem where the InjectionLoader boostrap bundle isn’t signed correctly. Please build from the sources and alter the codesigning identity in the Build Phase until it is not giving an error. A build from Alcatraz will not report this error. Then you shouldn’t need to patch.

Hi, @johnno1962 . Yes. you are right.
I have deleted from alcatraz then build the injection project it manually (with evident signing instead of "iPhone Developer").
and it fixed the problem. I don't have to patch my project to apply injection :)
Thank you again.

@johnno1962
It came back again (it doesn't work even if I build the plugin using xcode):
2016-11-09 14:35:03.102 Project[1642:10843608] Error loading /Users/user/PR/Project/iOSInjectionProject/build/Debug-iphonesimulator/InjectionBundle2.bundle/InjectionBundle:  dlopen(/Users/user/PR/Project/iOSInjectionProject/build/Debug-iphonesimulator/InjectionBundle2.bundle/InjectionBundle, 265): Library not loaded: @rpath/XCTest.framework/XCTest
  Referenced from: /Users/user/PR/Project/iOSInjectionProject/build/Debug-iphonesimulator/InjectionBundle2.bundle/InjectionBundle
  Reason: image not found
2016-11-09 14:35:03.102 Project[1642:10843608] Bundle Load Error
2016-11-09 14:35:03.102 Project[1642:10843757] *** Bundle has failed to load. If this is due to symbols not found, this may be due to symbols being hidden from dynamic libraries. ***
82 injections performed so far.

what could I do?

Alcatraz may have rebuilt your rebuilt your plugin? Try building it manually again?

@johnno1962  I have removed it from Alcatraz.
Then build it manually.
Still this problem.
what does this mean?
Library not loaded: @rpath/XCTest.framework/XCTest

I’m sorry, but I don’t know. Which Xcode is this??

xcode 8.1 and I think it happens after upgrading to it.
I'll check on another machine if it works there.

same problem
Library not loaded: @rpath/XCTest.framework/XCTest

Details:
2016-11-27 00:53:36.034 xxy[51329:3837555] Error loading /Users/ooops/Documents/Xcode/Projects/xxx/iOSInjectionProject/build/Debug-iphonesimulator/InjectionBundle8.bundle/InjectionBundle:  dlopen(/Users/ooops/Documents/Xcode/Projects/xxx/iOSInjectionProject/build/Debug-iphonesimulator/InjectionBundle8.bundle/InjectionBundle, 265): Library not loaded: @rpath/XCTest.framework/XCTest
  Referenced from: /Users/ooops/Documents/Xcode/Projects/xxx/iOSInjectionProject/build/Debug-iphonesimulator/InjectionBundle8.bundle/InjectionBundle
  Reason: image not found
2016-11-27 00:53:36.034 xxy[51329:3837555] Bundle Load Error
2016-11-27 00:53:36.035 xxy[51329:3838456] *** Bundle has failed to load. If this is due to symbols not found, this may be due to symbols being hidden from dynamic libraries. ***
5 injections performed so far.

After several hours' investigation, I finally make it works for the Library not loaded: @rpath/XCTest.framework/XCTest issue inspired by another similar dyld question.
All you need to do is clear all the derived data (not just your current project). At least this works for me.
Derived data was causing the problem.
Thank you so much!!!
p.s. but on Sierra since  8.1 unsigned xcode doesn't work good(( it freezes on saving.

Hey @dollar2048
Try this and see if that doesn't solve the freezes -<EMAIL_ADDRESS>

@zenangst Thank you so much! it helped me! Xcode is great again.

@johnno1962 I don't know if I should close this issue or leave it as it is now.

Let’s leave it open for now there is a lot of useful information here, particularly the contacts fix.

@johnno1962 您好，这个视图没有被删除当我删除一个视图。


How can I use this module to recognize cat faces?

face_recognition version: 1.3.0
Python version: 3.8.10
Operating System: Ubuntu

Description
I am trying to make this example work with cat faces
https://github.com/ageitgey/face_recognition/blob/master/examples/facerec_from_webcam_faster.py
What I Did
I tried to replace the two pictures with two different cats, but I get this output:
Traceback (most recent call last):   File "reco.py", line 19, in <module>     cat1_face_encoding = face_recognition.face_encodings(cat1_image)[0] IndexError: list index out of range
I think that the module doesn't recognize cats. Is there a way to make it works?
Thanks!

Models have been trained to detect human faces, not cats. You get the Index error because you're trying to index an empty list, because it didn't detect any (human) face.

Models have been trained to detect human faces, not cats. You get the Index error because you're trying to index an empty list, because it didn't detect any (human) face.

Exactly

face_recognition version: 1.3.0
Python version: 3.8.10
Operating System: Ubuntu

Description
I am trying to make this example work with cat faces https://github.com/ageitgey/face_recognition/blob/master/examples/facerec_from_webcam_faster.py
What I Did
I tried to replace the two pictures with two different cats, but I get this output: Traceback (most recent call last): File "reco.py", line 19, in <module> cat1_face_encoding = face_recognition.face_encodings(cat1_image)[0] IndexError: list index out of range I think that the module doesn't recognize cats. Is there a way to make it works?
Thanks!

If you want the model to recognise cats, you have to train the model with cat pictures, since the model is not aware of cats existence, you have that error.

Thanks for helping me. I will try to train my own model, as you suggested.

Thanks for helping me. I will try to train my own model, as you suggested.

Hello, have you solve this new model training?

Thanks for helping me. I will try to train my own model, as you suggested.

Hello, have you solve this new model training?

No but you can use opencv for cats detection and tensor flow for recognition.

Thanks for helping me. I will try to train my own model, as you suggested.

Hello, have you solve this new model training?

No but you can use opencv for cats detection and tensor flow for recognition.

Thanks.
After finding hard to train new model for this solution,  I'm trying to use tensorflow instead.


Fix lost links in webportal's doc and pylon's doc

Coverage decreased (-22.2%) to 51.476% when pulling 2faeda3792556e8b8038dee0c63bcd802a7c20fa on hwuu/webportal-doc into 17075deed9bc18d8fb19e528473e5f1e1ea23cac on master.


[GLUTEN-2733][CORE][CH] Implementation of the automatic-resource-management pattern

What changes were proposed in this pull request?
This PR impelments automatic-resource-management pattern. For SQLMetric, we can use the following code to reduce duplicated codes:
val result = GlutenTimeMetric.millis(longMetric("collectTime")) { _ =>
 ...
}

I also implement a convenient utility GlutenTimeMetric.withMillisTime and GlutenTimeMetric.withNanoTime
other improvement
I removed duplicate codes of postColumnarTransitions and preColumnarTransitions, extracting into a common method transformPlan, rewriting the following foreach
      preOverrides().foreach {
        r =>
          overridden = r(session)(overridden)
          planChangeLogger.logRule(r(session).ruleName, plan, overridden)
      }

Actually, The above foreach  is fold
    val overridden = getRules.foldLeft(plan) {
      (p, getRule) =>
        val rule = getRule(session)
        val newPlan = rule(p)
        planChangeLogger.logRule(rule.ruleName, p, newPlan)
        newPlan
    }

(Fixes: #2733)
How was this patch tested?
Using Existed UT.

@baibaichen looks great!


[ENH] Add generate module

closes #308

Changes proposed in this pull request:

add generate.py to create coordinate based datasets for meta-analysis
add test_generate.py
add test_estimator_performance.py for example on how to test different datasets with different estimators

Open Questions/Thoughts

Does the general approach for generating coordinate based datasets look sound (in generate.py)?

one weird thing I'm leveraging is ma_compute to create a probability function to select foci for specific studies
other weird things?

What kinds of outputs should be measured/observed to see if the estimator is "working" and performing adequately?

not all multiple correction transformers have a p-value output.
some measure of a false positive rate

How should I generalize to image-based meta-analysis?
How should I profile time/memory usage?

potentially pytest-profiling

create a method to generate random foci between studies (false positives)?

I'll be editing this text to provide more context.
This pull request is a WIP, I'm opening early so @tsalo and @tyarkoni (and anyone else interested) can provide feedback so I do not stray too far away from what this pull request should do.
use cases being supported:

[x] generating arbitrary numbers of foci for an arbitrary number of studies
[x] generating a focus only present in some of the studies.
[x] having 2 (or more) foci with different probabilities of being selected in a study
[x] have 1 study with a large number of reported foci and other studies without
[x] have 1 study with a large sample size and other studies without
[x] varying the number of noise foci to be reported in each study
[x] creating only noise foci (kind of)
[x] varying the spread of study foci around the "ground truth focus"

use cases not supported:

[ ] reading/permuting input datasets to generate simulated coordinates from observed values
[ ] user-generated masks to select a portion of the brain
[ ] specifying coordinates for noise_foci
[ ] creating coordinates in non-MNI space
[ ] image-based simulations

@jdkent This looks great! Comments below.

Does the general approach for generating coordinate based datasets look sound (in generate.py)?

LGTM, with the caveat that it's hard to get a qualitative sense of what the data generated by create_coordinate_dataset() actually look like. Maybe you could create a demo notebook (@tsalo, any thoughts on where you want it to go?) that plots (a) the generated data (or at least a subset of it) and then (b) the resulting statistical maps when run through a CBMA estimator? I think this would help us get a better sense of whether everything is kosher, though from the looks of the code (and the test), it seems sound.

one weird thing I'm leveraging is ma_compute to create a probability function to select foci for specific studies
other weird things?

Seems fine to me.

What kinds of outputs should be measured/observed to see if the estimator is "working" and performing adequately?

In addition to the existing sensitivity test at the ground truth location, we could add some specificity checks to the test—i.e., to verify that there isn't a significant result almost anywhere else. An expedient way to do this would be to just select a few random coordinates that are not very close to the ground truth sources, and verify that p > .05. A more robust approach would be to look at the p-values for all voxels in the brain that are not spatially close to the ground truth sources. I.e., you could generate a large-ish ROI mask around the true sources, and then verify that the proportion of voxels outside the mask that show a significant effect is roughly .05 (i.e., the method controls the false positive rate approximately as expected). For this test, you might want to repeatedly generate datasets and calculate the false positive rate over all datasets. Plotting a histogram in the notebook suggested above might also help give us a sense of how an estimator is performing (though the critical thing is that the mean be close to 0.05).

not all multiple correction transformers have a p-value output.

That should be fine as long as they still output an image of voxels that survive correction; then you can just mask that image with the set of voxels that are far from true sources, and the proportion of significant voxels in the result is your false positive rate.

some measure of a false positive rate

Yup, see above.

How should I generalize to image-based meta-analysis?

Good question. Eventually we'll probably want to test these with more realistic ground truth maps where each voxel has some non-zero population value and SD, but for now I think it's fine to stick with the sparse sources. A reasonable first pass would be to add a create_maps() function to complement create_foci(), where the former would return entire maps rather than study-specific peaks. You could probably use the same general approach—i.e., convolve the study-specific peak with a gaussian, and then return the resulting image (where most of the voxels will have a value of zero). Because the voxel values will now have magnitude information, you may want to add a parameter or two that control the variance in the peak value (i.e., some study maps should contain stronger effects than others).
I wouldn't spend a lot of time at this point trying to come up with a realistic generation procedure; let's first get a general workflow for evaluation and testing (particularly, quantification of error rates) figured out, and then we can iterate and figure out how to produce IBMA meta-analysis datasets that look more realistic.

How should I profile time/memory usage?

Let's start simple. I think just recording the total time taken to run each combination of estimator/corrector, and maybe max memory consumption, would be great. You could lean heavily on packages like line_profiler and memory-profiler—which would also make it easier to build fancier reports later on.

potentially pytest-profiling

Haven't used or looked at it before, but if it does what you need it to, sure!

create a method to generate random foci between studies (false positives)?

Great idea. But I'd probably defer that until we're satisfied with the basic framework for quantifying and reporting error rates. At that point we can iterate on the data-generating utilities.
Nice work!

Maybe you could create a demo notebook (@tsalo, any thoughts on where you want it to go?) that plots (a) the generated data (or at least a subset of it) and then (b) the resulting statistical maps when run through a CBMA estimator?

I would definitely like to see this used in testing. Not just unit testing of the module itself, but also integration testing with the CBMA estimators. If memory usage is not too high, an example would be nice. RTD is limited though, so if memory usage is too high then I think a notebook is a good place for it.

use cases being supported:

[x] generating arbitrary numbers of foci for an arbitrary number of studies
[x] generating a focus only present in some of the studies.
[x] having 2 (or more) foci with different probabilities of being selected in a study
[x] have 1 study with a large number of reported foci and other studies without
[x] have 1 study with a large sample size and other studies without
[x] varying the number of noise foci to be reported in each study
[x] creating only noise foci (kind of)
[x] varying the spread of study foci around the "ground truth focus"

use cases not supported:

[ ] user-generated masks to select a portion of the brain
[ ] specifying coordinates for noise_foci
[ ] creating coordinates in non-MNI space

This now tests many combinations of meta-analysis estimators, kernels, and correctors for CBMA (48 combinations total) on two simulated datasets resulting in 96 additional tests.
There are several combinations of these objects that fail, and they are noted with explicit if/then clauses within the test.
related to #344

Alright, in this latest sequence of updates I:

renamed variables from review
made it more straightforward to generate a dataset with no converging foci

and added a section in the example notebook demonstrating this functionality

use pytest.xfail in test_estimator_performance.py to make it more obvious which meta/kernel/corrector combinations have potential issues
minor bugfixes/reformatting

There are other items on the wishlist for this module, but I think it has reached a point where it is useful as is.
@tyarkoni and @tsalo, if you believe I've responded to your reviews adequately, this is ready to go.

@tyarkoni, I think after your review (and potential fixes) we can merge this.

summary of findings:

errors

meta.fit()

any Meta + Peaks2MapsKernel does not work
ALE + KDAKernel sometimes does not work

corr.transform(res)

MKDA + any kernel + any corrector except montecarlo simulations
KDA + any kernel + any corrector except montecarlo simulations
ALE + KDAKernel + montecarlo sometimes does not work

poor performance

ALE + MKDAKernel + any corrector demonstrates inadequate performance

once #362 and #365 are merged, I believe all tests will pass, and we can have another round of reviews since I changed the code somewhat significantly after Tal's review.

Now that tests are passing, can I get another look through @tsalo and @tyarkoni?
If it looks good to you both then I think we can merge.

Sorry, @tsalo—I only saw you self-requested a review after I hit merge. Feel free to revert if you like, or we can just apply any new changes as a new PR.


Feature/data integrity test

To note, there are some updates in the feature/update-manifest branch that haven't been implemented yet since the pipeline isn't being rerun yet, which means this script will need to be updated when that time comes.
This script will generate a Markdown report for each team and an output log. To test:

One team: util/promote_staging_data -t team-lee
Multiple teams: util/promote_staging_data -t team-lee,team-jakobsson

@kfang4 can you post a screenshot of an example report (paths can be made up, just want to see the format)

All comments make sense- will implement these changes. One thing, the order operations would instead be:

Test data
Write report locally; include paths to the previous and current manifests (even if you haven't yet written the current manifest to that loc)
Copy report and manifest to the metadata/${timestamp} directory.
Promote data

If data is promoted, the prod manifests will be overwritten.

To note:

Changes in feature/update-manifest include renaming workflow to just harmonzied_pmdbs rather than preprocess and cohort_analysis in manifest files
Changes in feature/update-manifest include adding hash (md5) in manifest files
Changes in feature/update-manifest include adding the GitHub release link in manifest files
"Previous manifest gs location" referred in this script does not exist yet

Dave (MJFF) reviewed report.

@hkeward Added a description for data integrity test result tables as per Dave's request. Merging.


Allow to navigate to an organisation from the dashboard

For now we just have the possibility to perform an action on a given event of an organisation.

Already done while developing the tours: https://github.com/kalisio/aktnmap/commit/9f55dc908c5cf1f6743a8d25323778a9a410bcd7


Add PoolInfo to implement discovery-time pool-specific logic

This PR is intended to demonstrate #212 and propose a possible implementation (still work in progress).
Firstly, it splits the current PciNetDevice interface into two separate interfaces:

PciNetDevice for low level device information (used for filtering)
PoolDevice for providing the information that the ResourcePool needs (i.e: DeviceSpecs, Mounts, Envs, etc)

That way, InfoProviders could be selected based on ResourcePool information.
This is part of my efforts to cleanly implement vdpa support for the Device Plugin.
It can be observed how, with this preliminary rework, an experimental vdpa support can be implemented with little change in existing DP code:
https://github.com/amorenoz/sriov-network-device-plugin/tree/vdpa_pooldev

/retest

I'm waiting for some feedback regarding the high level design before starting to work on the tests. Could you ptal @ahalim-intel @zshi-redhat ?

After adding poolInfo, we will have two places to get device related info, one is in discover phase where ResourceConfig is not passed in so we only do device filter based on device class, the other is when generating ResourcePool where ResourceConfig is passed in so advanced filtering is possible, for example, getting rdma spec only if isRdma is configured (see https://github.com/intel/sriov-network-device-plugin/pull/226).
Do we have a clear definition in mind on when will each path be used? or shall we consider combining them to one, for example, use poolInfo for device discover?

@zshi-redhat You're right, with this PR isRdma can be reinterpreted as a "Pool Flag" instead of a "pciNetDevice Filter", their correspondent meanings being:

Pool Flag: If the flag is set, this pool will provide Rdma deviceSpecs. If any of the filtered devices does not have valid Rdma device specifications, the pool creation will fail.
NetDevice Filter: RdmaSpecs will be retrieved from all NetDevices (it's a DeviceType property). If the flag is set, only devices with valid RdmaSpecs will be added to the pool.

In general, I think that all the properties that are not valid across all devices within a DeviceType could be interpreted as "Pool Flags" and only be retrieved if the pool flag is specified. With the future introduction of vDPA, both "isVdpa" and "isRdma" would fit in this definition since they are mutually exclusive. (Note: I don't mean to say isRdma devices cannot be exposed via vDPA, which they surely can. But the rdma devices will be handled by the vDPA Framework and only a vhost socket will be exposed via DevicePlugin API).
Regarding the other alternative you propose, IIUC it would entail:

Having a DeviceProvider per ResourceConfig (instead of per DeviceType)
Postpone the call to DeviceProvider.AddTargetDevices() to ResourceManager.initServers() so that the ResourceConfig is available.
Perform device discovery  + filtering + pool information gathering with the ResourceConfig handy

I think your proposal would enable the usecase being tackled here (Same device info, different pool info). We would basically implement all the logic inside NewPciNetDevice(), that would now accept a ResourceConfig as parameter. We would still be combining both device-specific and pool-specific information in the same structure (hence I still think the API split proposed is technically valid) but it would make a smaller PR to enable this usecase.

I think your proposal would enable the usecase being tackled here (Same device info, different pool info). We would basically implement all the logic inside NewPciNetDevice(), that would now accept a ResourceConfig as parameter. We would still be combining both device-specific and pool-specific information in the same structure (hence I still think the API split proposed is technically valid) but it would make a smaller PR to enable this usecase.

makes sense to me, let's make the small change.

@amorenoz please close this PR if #231 satisfies this usecase

@amorenoz please close this PR if #231 satisfies this usecase

Right. It does. Thanks


iptables: ignore loopback traffic

Please describe the Improvement and/or Feature Request
iptables rules for sidecar interception should ignore loopback traffic.
Scope (please mark with X where applicable)

New Functionality          [X]
Install                    [ ]
SMI Traffic Access Policy  [ ]
SMI Traffic Specs Policy   [ ]
SMI Traffic Split Policy   [ ]
Permissive Traffic Policy  [ ]
Ingress                    [ ]
Egress                     [ ]
Envoy Control Plane        [ ]
CLI Tool                   [ ]
Metrics                    [ ]
Certificate Management     [ ]
Sidecar Injection          [ ]
Logging                    [ ]
Debugging                  [ ]
Tests                      [ ]
CI System                  [ ]
Demo                       [ ]
Project Release            [ ]

Possible use cases
Loopback traffic should not be proxied to the sidecar because such traffic does not require a physical connection to a network.

<IP_ADDRESS>/32 is ignored.


CLI Tool for static/dynamic analysis (Prototype)

Find all the Deprecated Attribute assignments and list them.
There should be possibility to include non-SimpUtils Deprecated attributes for analysis, like the JetBrains one: https://github.com/JetBrains/phpstorm-attributes/blob/master/src/Deprecated.php

Created https://github.com/PandaHugMonster/php-simputils-dev-tools repo for this functionality.


Add k8srequireingressclass together with OPA/library upgrades

Update OPA to 3.8.1, gatekeeper-library to 0.12.1 and add k8srequireingressclass constraint.

Related to https://github.com/XenitAB/gatekeeper-library/pull/76

@phillebaba would it be worth moving https://github.com/XenitAB/terraform-modules/blob/9698261a3684db318d3b6f6b0524e151d9496484/modules/kubernetes/aks-core/modules.tf#L51-L61 and same in EKS in to the OPA module instead?

Makes sense moving it but I would do it in a different PR.


Allow renderStackFrameBox to be overridden

Allows the function that renders individual stack frame boxes to be overridden so applications using d3-fg can style their boxes however they want. This includes setting fills, strokes, drawing any additional lines and applying and removing any custom styles to hovered nodes.
It requires exposing quite a lot of d3-fg global variables, so these have been hoovered up and passes as an object alongside the object containing the node-specific arguments.

At this point a renderFrame() option would be the best, so only the positioning calculations are always done in d3-fg and all of the drawing can be overridden by an option. The boxes, heat, and labels could all be done in there.
That is a bigger change though, this LGTM code wise as is

Maybe, though that change would just mean that it's not possible to have custom text without overriding the boxes, or custom boxes without overriding the text, etc. Don't really see much benefit there.


Do not push main binary into cachix cache

What type of PR is this?
/kind cleanup
What this PR does / why we need it:
The new release v9 of the cachix-action now supports filtering pushed
derivations. We now exclude the main binary for the project. This
prevents from flooding the cache with built derivations from the
security-profiles-operator binaries, since we usually want to cache only
its dependencies.
Which issue(s) this PR fixes:

Ref https://github.com/cachix/cachix-action/releases/tag/v9
Does this PR have test?

None
Special notes for your reviewer:
None
Does this PR introduce a user-facing change?

None

Rebased on the latest master branch.

/test pull-security-profiles-operator-test-e2e


Adding imshow function to directly display image/video without moving tensor off GPU.

🚀 Feature
Looking for a imshow function similar to that of cv2 to directly display tensor without moving it off GPU.
Motivation
The main use case is for displaying video frames for real-time applications (such as webcam demo etc).
If the image is already on the graphics card, it should be a way to directly display it. I currently cannot find other tools available to achieve this and I don't know how to implement it. I think it would be a useful function to include in torchvision.
Alternatives
Moving tensor to CPU every frame, but this is very slow for real-time.

Thanks for the proposal @PeterL1n , but I doubt this will ever be in scope for torchvision: at the moment we don't even offer a basic imshow util for CPU images, let alone videos


Cleanup JobCreator_t.py

including changing to set literals for @amaltaro

wonderful! :)

test this please

@ericvaandering I guess there are still a few warnings/errors that you could get rid of? :)

Three of these tests are just commented out which is maybe the biggest source of warnings. But it seems they might be useful at some point?

Hmm, yes, I think so. How about decorating them with performance then? If performance unittests are picked up by jenkins, then we could use integration, right?

Ok, this is as good as it will get. @ticoann please merge when Jenkins comes back.

test this please

test this please

test this please

test this please

test this please

@ticoann please merge


Update Youtube Search for Sinusbot 0.10

First of all, thanks for this awesome script!
Are you planning to update it for Sinusbot 0.10? Do you have any ETA?

The intention is keep at the latest sinus version, also if you want to help make a pull request! all of you are welcome.
PS. What do you mean with ETA?


X11: VSync not working unless it's activated in code

This issue might be related to #25377
Godot version:
3.1 beta 3
OS/device including version:
OS: MX Linux 17(based on Debian 9)
GPU: Intel HD Graphics
Issue description:
VSync only seems to work when activated from code. This doesn't happen on windows.
Steps to reproduce:
Activate VSync from the editor in any project and the option will be ignored.

I can confirm that vsync is still not working with 3.1 Beta 5 (mono build but I don't think that is very relevant but needed mentioning) I am running a Radeon R7 360 with the RadeonSi (amdgpu) Mesa drivers (Mesa version is 18.3.2)
This project has vsync enabled and prints the fps to the console.
vsync.zip

Does vsync work otherwise on your system, e.g. running glxgears?

glxgears is not running with vsync

However I cannot tell if it is even trying to sync with the vblank

I'll go check some gl based games and see if vsync works in them on this rig.

I just tested an when running TF2 with vsync enabled the fps still exceeds my monitor's refresh rate by a great deal. Very odd....

I think this was a system issue, I added

to my .drirc and the issue  was fixed, I don't know if anything related to vsync changed from beta 3 to beta 5 but I can't replicate the issue anymore even if I remove this from my .drirc


Is it possible to change a pallet's constant?

Hey, folks!
I'm working on adapting Kusama's interfaces to reflect the changes done to the Society pallet (see Gav's PR)
I'm experimenting with Chopsticks lately to make my life easier and I think it's great!
But I think I hit a roadblock... I'm trying to create my own config to set some initial state for the society, I managed to change the storage fairly easily but now I need to change some constants too, both VotingPeriod and ChallengePeriod, but they seem to be hardcoded on the runtime ☹
What are my options here?

Is there an easy way to change constants through Chopsticks without a runtime upgrade?
If no, is there an easy way to do a runtime upgrade?
I thought about adding Sudo, but when I tried this I got this error: Error: Cannot find pallet Sudo. Is there a way to load the Sudo pallet again on Kusama?

Thanks!

unfortunately constants are part of the runtime so can't be changed
you can use option --wasm-override to run your custom runtime
you can't added storage for a module that doesn't exists

@laurogripa you don't really need sudo when you can modify storage or use --mock-signature-host

You can use scheduler to dispatch root origin call https://hackmd.io/6bKd532rTdaxVX0hmGXX7g#Dispatch-Call

@ermalkaleci @xlc
Oh, perfect!
I read the docs and the code looking for something with runtime but ended up overlooking the wasm-override parameter. It worked like a charm, just had to build my own custom runtime and point to it.

you can't added storage for a module that doesn't exists

I know, I know, what I meant is if there was a way to use Chopsticks to load a module that was removed, but wasm-override did the trick anyway :)
I'm going to close this issue as it's solved for me.
Thanks a lot for you attention in such a short time.
This project is awesome! 💟


[SPARK-50510][CONNECT][3.5] Fix sporadic ReattachableExecuteSuite failure

What changes were proposed in this pull request?

ReattachableExecuteSuite detected a rare data race issue where ExecuteThreadRunner may send the client the wrong error code before the SparkConnect service sends the correct error code.

The test fails if ExecuteThreadRunner is finished before the SparkConnect service sends the correct error code and after the session is invalidated; to be specific, the event manager throws an illegal state exception (SPARK-49688) that is translated into an unknown error.
The whole problem was addressed under https://github.com/apache/spark/pull/48208 for Spark 4.0.

Why are the changes needed?

Clients may get the wrong error message: expect session-closed or the like, but get unknown.
To fix the ReattachableExecuteSuite failure.

Does this PR introduce any user-facing change?

No.
How was this patch tested?

ReattachableExecuteSuite.
Was this patch authored or co-authored using generative AI tooling?

No.

@LuciferYang Hi, I'm a little bit late, but anyway, I found the cause of the test failure and fixed it.

so this is not a test only issue ?

Not a test only issue, but a "rare" data race between the SparkConnect service and ExecuteThreadRunner.
-> The client may get the wrong error code.

Why are the changes needed?
To fix the ReattachableExecuteSuite failure.

@changgyoopark-db This description appears somewhat misleading, leading me to believe it's a testOnly issue.
also cc @HyukjinKwon @hvanhovell

@dongjoon-hyun I'll think about writing a dedicated test case -> it is a concurrency issue, so it will not be trivial, but definitely possible.

Overall, given the number of sporadic test failures, I think we (or I?) need to spend significant time dealing with all the data race problems in the SparkConnect code.

Created https://issues.apache.org/jira/browse/SPARK-50535 and https://issues.apache.org/jira/browse/SPARK-50534 for tracking known issues. I'll fix them "hopefully" by the end of this year.

Yep, it looks good to go to me, @HyukjinKwon . Thank you, @changgyoopark-db and all.

Merged into branch-3.5 for Apache Spark 3.5.4 RC2, thanks @changgyoopark-db @dongjoon-hyun and @HyukjinKwon

Just for the record, CI proves that this PR (SPARK-50510) isn't enough to mitigate ReattachableExecuteSuite failures in branch-3.5 by showing that ReattachableExecuteSuite fails again at the merge commit.

https://github.com/apache/spark/actions/runs/12269953616/job/34234396166

[info] *** 1 TEST FAILED ***
[error] Failed tests:
[error] 	org.apache.spark.sql.connect.execution.ReattachableExecuteSuite

@dongjoon-hyun thanks for informing this. I precisely know what exactly went wrong with the test by looking at the stack trace - it‘s now a test-only problem; unfortunately, I’m not available this week (no laptop access) so I’ll try to make a PR next Monday.

Unfortunately, it fails again today. Apparently, this backport has a chance to increase flakiness instead of reducing.

https://github.com/apache/spark/actions/runs/12296721038/job/34316344940

[info] *** 1 TEST FAILED ***
[error] Failed tests:
[error] 	org.apache.spark.sql.connect.execution.ReattachableExecuteSuite


Optimize Inference Performance on CPU

Description
the news in https://github.com/dmlc/gluon-nlp/releases/tag/v0.8.1 shows BERT int8 quantization is presented in blog
https://medium.com/apache-mxnet/optimization-for-bert-inference-performance-on-cpu-3bb2413d376c
But the blog only shows some results of BERT quantization test,

The work on low precision deployment is still ongoing and involves un-released SW, the reproduction instructions will be available later.

When will this work be released and can we apply this quantization method on GPT2?
Thanks a lot for the great work!

@TaoLv

Sorry for missing the message. We're working on cleaning the code and solution. Hope we can have a PR soon. I'm not familiar with the status of GPT2 in GluonNLP. Could you please point me to the scripts and whether it can be exported as a static model?

Yes, recently static GPT2 model is supported: https://github.com/dmlc/gluon-nlp/pull/1010

Thanks for the replies. @leezu @TaoLv
Looking forward to try int8 bert and gpt2 soon~

@carter54 FYI, here is the PR for BERT quantization: #1080

@TaoLv Thx for the work, can this method applied to GPT 2 model?


⚠ Samsung has degraded performance

In a87db64, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2331 ms

Resolved: Samsung performance has improved in 08bfbe7.


Fix PR #6 (bump connect-nodejs dependency)

PR #6 depended on a simultaneous PR to connect-nodejs (which was just published as v0.4.0).
Bumping the dependency to reflect this.

thanks!


Added LMCOKillChain class

This addresses #158.
This PR adds an LMCOKillChain class to the stix.common.kill_chains module. This class initializes a KillChain instance with the phases, ids, references etc. set to the values found in the STIX definition here: http://stix.mitre.org/language/version1.1.1/stix_v1.1.1_lmco_killchain.xml.
@gtback, let me know if this isn't what you were thinking for #158 or if there is anything I missed. Feel free to merge it otherwise :)

This is exactly what I was looking for. A few questions:

what is temp_id_namespace used for? I see it defined but not used anywhere in this PR.
The more I look at it, I'm wondering if it would be better to have an LMCOKillChain be an module-level instance of KillChain rather than a subclass. I don't see a need for people to instantiate their own copy of the class, which can only lead to problems. The phases could similarly be defined as module-level variables, so you could easily add references to them to TTPs (an example in the documentation would be great as well).

I think this is awesome and definitely needed. I pushed one typo fix as well.
I would like for this to work, but currently it isn't. I'm not sure if the best approach is to make the constants into KillChainPhaseReferences, or add new constants that can be used for indicators. An even better approach might be to redefine  KillChainPhaseReference.__init__  so that if you pass it a KillChainPhase as the first argument (currently phase_id), it builds a reference to that type.
from stix.common.kill_chains import PHASE_RECONNAISSANCE
from stix.indicator import Indicator
i = Indicator()
i.add_kill_chain_phase(PHASE_RECONNAISSANCE)
print i.to_xml()

Output snippet:
<indicator:Kill_Chain_Phases>
    <stixCommon:Kill_Chain_Phase phase_id="<stix.common.kill_chains.KillChainPhase object at 0x1094bc1d0>"/>
</indicator:Kill_Chain_Phases>

I like the __init__() idea! I had been thinking about your use case as well but hadn't thought of that. The ones I was fixated on were:

Implement the PHASE_FOO constants as KillChainPhaseReference objects
Add an as_reference() method to KillChainPhase

I am a bit torn between the first bullet and your __init__() idea. Hmm. I think changing the constants to KillChainPhaseReference is less obtrusive since modifying __init__() could break code that isn't using keyed args, but its also less powerful. Thoughts?

It gets a bit crazy and messy, but you could do:
if isinstance(phase_id, KillChainPhase):
   <build a reference>
else:
    <what's currently there>

I'm not sure how I feel about that, but that's what I was imagining. It would keep backward compatibility, at the cost of some additional complexity and confusion.

I'm happy with this now. If the documentation looks good to you, @bworrell, go ahead and merge this.

Looks good! Thanks @gtback! Merging.