Alembic 1.11.2 fixes

Closes #
Changes proposed in this pull request

Instructions for Reviewers
[All PRs] - Confirm PR template filled
[Feature Branches] - Review code
[Production Merges to main]
    - Check story numbers included
    - Check for debug code
    - Check version

This is no longer required.


More than one logfile created when using multiprocessing ( exe built with pyinstaller)

Hello.
I am using logger_tt module with python code having multiprocessing approach in windows environment. While running the code from python interpreter , one logfile is created as expected.
But when I build the exe ( using pyinstaller) and execute it using a windows command prompt , number of logfile created are more than one. I am creating three processes, and the four logfiles are created.
Though only one logfile contains all the logs (including print, etc..). The other logfiles created just contains following text (with different time stamps) :
[2022-11-02 10:58:44] [root:190 DEBUG] New log started_
[2022-11-02 10:58:44] [root:191 DEBUG] Log config file: configs\log_config.json

Hi, could you include your log config and the name of the others 3 log files?

Hi , yes sure.
I have attached the file here with changed extension (csv, .config is not allowing)
The name of the files:
LOG_2022-11-02_10-57-58.log ( complete content)
LOG_2022-11-02_10-58-48.log (extra file)
LOG_2022-11-02_10-58-44.log (extra file)
LOG_2022-11-02_10-58-40.log (extra file)
log_config.csv
Thanks

Hi, can you also include the code that you config the logger? setup_logging(...)
Did you put it out or inside the if __name__ == "__main__"?
Are you running on Linux or Windows?
Just a minimal reproducible code would be very helpful.

hi @81piyush, any update on your input?
If you are not using the latest logger_tt please upgrade and try again.
From your log filename, you are creating the name dynamically by a function.
You can reference issue #5 for an alternative solution with the same problem if it still persists.

Hi @Dragon2fly , apologies for the delayed reply.
The setup_logging function is getting called in the following manner:
setup_logging(config_path=r"./configs/log_config.json", log_path=str_file_path, use_multiprocessing=True,
full_context=1, capture_print=True)
It is outside the if name == "main"
I am using it on Windows pc. It is updated to the version 1.7.0
I will go though the mentioned issue (https://github.com/Dragon2fly/logger_tt/issues/5 ) to check the changes accordingly.
If it still persists , I will share the reproducible code as well. Thanks for the suggestions.
Will keep updated about the same.

Hey @Dragon2fly ,
I have tried to integrate logger_tt into our project, oceancabbage/RimSort
If you have a moment, do you think you could take a look at my implementation of your logger?
On Linux, it behaves as expected both when run from python interpreter, and PyInstaller directly.
On Windows, it behaves as expected up until I try to launch a process, and then some really wonky stuff happens:

On MacOS, I have similar behavior to Windows, except I was actually able to get a stacktrace out of it since running a Unix executable in a terminal is actually apparently much more informative than a Windows exe from PyInstaller ;)

Perhaps it is a misconfiguration on my end?
https://github.com/oceancabbage/RimSort/blob/ba71c6d746e9c9892900902de74cce6e33c57993/RimSort.py

Hi @twstagg,
On Windows, you can open a command prompt and run your exe from there to get the error message if there is any.
Could you try and provide that messages too, if any?
Anyway, from your MacOS log, I suspect that logger-tt didn't recognize that it was running in the child process.
Thus it tried to launch the logger server again and failed as the server was already launched in the main process.
This needs further digging. But first, could you print out the child process name by using print(multiprocessing.current_process().name)?
Put it in your child process's code and see if it is different before and after pyinstallered

Steamworks wrapper: <_MainProcess name='MainProcess' parent=None started>
Main script: <_MainProcess name='MainProcess' parent=None started>
[2023-04-01 18:19:34] WARNING: Running using Python interpreter
Steamworks wrapper: <_MainProcess name='Process-1' parent=None started>
Main script: <_MainProcess name='Process-1' parent=None started>

@twstagg Thanks for your logs.
What OS did you collect the log from?
I'm seeing the below line repeated 2 times in the output after PyInstallered.
This is different than the pure interpreter's log.

Main script: <_MainProcess name='MainProcess' parent=None started>

Then the Process-1 appears in [2023-04-01 18:29:36] WARNING: Process-1 Unable to initiate Steamworks API call. Steam is not running!.
So, we see that the spawned child process still had the name MainProcess when logger-tt was invoked, which causes the problem. Then there are 2 questions that need to be answered.

Is the text Process-1 output by the multiprocessing.current_process().name too?
If yes, at which point the process name changed?

@twstagg Thanks for your logs.
What OS did you collect the log from?

That was from Windows. I will post some additional output from MacOS for you soon, with these new debug prints added.

I'm seeing the below line repeated 2 times in the output after PyInstallered. This is different than the pure interpreter's log.

Main script: <_MainProcess name='MainProcess' parent=None started>

Then the Process-1 appears in [2023-04-01 18:29:36] WARNING: Process-1 Unable to initiate Steamworks API call. Steam is not running!.
So, we see that the spawned child process still had the name MainProcess when logger-tt was invoked, which causes the problem. Then there are 2 questions that need to be answered.
1. Is the text `Process-1` output by the `multiprocessing.current_process().name` too?

Yes it is. I went ahead and put the print in both of the files that are being used here:

The main script (where setup_logging is called)

https://github.com/oceancabbage/RimSort/blob/c0f86bd55c2219afd8e7c0c43fbbb7b0f914a18f/RimSort.py#L20

The module which contains the functions that passed to the multiprocessing.Process

https://github.com/oceancabbage/RimSort/blob/c0f86bd55c2219afd8e7c0c43fbbb7b0f914a18f/util/steam/steamworks/wrapper.py#L16

See where we are actually launching the processes (This method is triggered by PySide2 GUI element from the MainProcess:

https://github.com/oceancabbage/RimSort/blob/c0f86bd55c2219afd8e7c0c43fbbb7b0f914a18f/view/main_content_panel.py#L781-L813

2. If yes, at which point the process name changed?

I am not sure how to answer this question exactly. It appears that when the application is PyInstaller'd and we are using spawn() - the MainProcess is copied before the Process-1 is created by the user action in the GUI.
If I do not have multiprocessing.freeze_support() run (this is conditionally run if platform.system() returns "Windows") then the GUI window will actually fully duplicate when the multiprocessing.Process is created.

@twstagg

If I do not have multiprocessing.freeze_support() run (this is conditionally run if platform.system() returns "Windows") then the GUI window will actually fully duplicate when the multiprocessing.Process is created.

From your description above, I understand that it opened the second GUI exactly the same as the first one even though the code is guarded with: if __name__ == "__main__".
This suggests that the __name__ of the child process is still __main__ as opposed to __mp_main__ when running with the interpreter. So the multiprocessing.freeze_support() must have a different way to know if it is in a child process or not.
Following the freeze_support() function, I added a similar check into:
https://github.com/Dragon2fly/logger_tt/blob/4cdbe73ff254121a4ed0a9b710239e8a31810af4/logger_tt/core.py
Could you replace the logger_tt/core.py in your local environment and test again?

@Dragon2fly

From your description above, I understand that it opened the second GUI exactly the same as the first one even though the code is guarded with: if __name__ == "__main__".

Correct.

This suggests that the __name__ of the child process is still __main__ as opposed to __mp_main__ when running with the interpreter. So the multiprocessing.freeze_support() must have a different way to know if it is in a child process or not.

Ahh! This makes sense. A quick test with similar debug prints confirms this as well.

Following the freeze_support() function, I added a similar check into: https://github.com/Dragon2fly/logger_tt/blob/4cdbe73ff254121a4ed0a9b710239e8a31810af4/logger_tt/core.py
Could you replace the logger_tt/core.py in your local environment and test again?

In my Windows env, I have replaced the Pypy  distributed logger-tt (latest release) with a locally cloned version of logger-tt. I did git checkout pyinstaller_test before I built the module. Sadly, I don't think your change had any effect on my issue:

@81piyush
Can you post the result of ‘pyinstaller’ed rimsort with the cloned logger-tt in the macOS env?
I want to know if it can run on macOS now or not.
As I understanding, there are 2 problems we are facing:

It runs on windows but output a bunch of ‘Null’ when starting the child process.
It doesn’t run on macOS

We will have to solve it one by one.
As for the windows case, can you provide an instruction to build the environment and create an exe with pyinstaller?

@twstagg Can you post the result of pyinstallered rimsort with the cloned logger-tt in the macOS env? I want to know if it can run on macOS now or not.

Sure thing. For science, here is output when running from Python interpreter. The process launches properly and multiprocessing works as intended.  Steam is not running!
[2023-04-05 15:44:28] WARNING: Process-1 Unable to launch game with Steamworks API. Please close your game and try to launch again after opening Steam.

On MacOS, here is the PyInstaller'd output. It does not launch the process because of the Unhandled exception. Super weird compared to what we see on Windows. With this I also added an additional print for sys.argv:
$ ./RimSort
RimSort.py: <_MainProcess name='MainProcess' parent=None started>
__name__: __main__
sys.argv: ['./RimSort']
steamworks.wrapper: <_MainProcess name='MainProcess' parent=None started>
__name__: util.steam.steamworks.wrapper
sys.argv: ['./RimSort']
main_content_panel.py: <_MainProcess name='MainProcess' parent=None started>
__name__: view.main_content_panel
sys.argv: ['./RimSort']
[2023-04-05 15:55:30] WARNING: Running using PyInstaller bundle
[2023-04-05 15:55:30] WARNING: Non-Linux platform detected: using multiprocessing.freeze_support() & setting 'spawn' as MP method
QPixmap::scaled: Pixmap is a null pixmap
RimSort.py: <_MainProcess name='MainProcess' parent=None started>
__name__: __main__
sys.argv: ['/Users/user/Desktop/RimSort/dist/RimSort.app/Contents/MacOS/RimSort', '-B', '-S', '-E', '-s', '-c', 'from multiprocessing.resource_tracker import main;main(32)']
RimSort.py: <_MainProcess name='MainProcess' parent=None started>
__name__: __main__
sys.argv: ['/Users/user/Desktop/RimSort/dist/RimSort.app/Contents/MacOS/RimSort', '--multiprocessing-fork', 'tracker_fd=33', 'pipe_handle=35']
steamworks.wrapper: <_MainProcess name='MainProcess' parent=None started>
__name__: util.steam.steamworks.wrapper
sys.argv: ['/Users/user/Desktop/RimSort/dist/RimSort.app/Contents/MacOS/RimSort', '-B', '-S', '-E', '-s', '-c', 'from multiprocessing.resource_tracker import main;main(32)']
steamworks.wrapper: <_MainProcess name='MainProcess' parent=None started>
__name__: util.steam.steamworks.wrapper
sys.argv: ['/Users/user/Desktop/RimSort/dist/RimSort.app/Contents/MacOS/RimSort', '--multiprocessing-fork', 'tracker_fd=33', 'pipe_handle=35']
main_content_panel.py: <_MainProcess name='MainProcess' parent=None started>
__name__: view.main_content_panel
sys.argv: ['/Users/user/Desktop/RimSort/dist/RimSort.app/Contents/MacOS/RimSort', '--multiprocessing-fork', 'tracker_fd=33', 'pipe_handle=35']
main_content_panel.py: <_MainProcess name='MainProcess' parent=None started>
__name__: view.main_content_panel
sys.argv: ['/Users/user/Desktop/RimSort/dist/RimSort.app/Contents/MacOS/RimSort', '-B', '-S', '-E', '-s', '-c', 'from multiprocessing.resource_tracker import main;main(32)']
Traceback (most recent call last):
Traceback (most recent call last):
  File "RimSort.py", line 36, in <module>
  File "RimSort.py", line 36, in <module>
  File "logger_tt/__init__.py", line 225, in setup_logging
  File "logger_tt/__init__.py", line 225, in setup_logging
  File "logger_tt/core.py", line 88, in from_dict
  File "logger_tt/core.py", line 88, in from_dict
  File "logger_tt/core.py", line 111, in _set_mode
  File "logger_tt/core.py", line 111, in _set_mode
  File "logger_tt/core.py", line 158, in _replace_with_socket_handler
  File "logger_tt/core.py", line 158, in _replace_with_socket_handler
  File "logger_tt/core.py", line 355, in __init__
  File "logger_tt/core.py", line 355, in __init__
  File "socketserver.py", line 452, in __init__
  File "socketserver.py", line 452, in __init__
  File "socketserver.py", line 466, in server_bind
  File "socketserver.py", line 466, in server_bind
OSError: [Errno 48] Address already in use
OSError: [Errno 48] Address already in use
[7582] [7583] Failed to execute script 'RimSort' due to unhandled exception: [Errno 48] Address already in use
Failed to execute script 'RimSort' due to unhandled exception: [Errno 48] Address already in use
[7582] [7583] Traceback:
Traceback (most recent call last):
  File "RimSort.py", line 36, in <module>
  File "logger_tt/__init__.py", line 225, in setup_logging
  File "logger_tt/core.py", line 88, in from_dict
  File "logger_tt/core.py", line 111, in _set_mode
  File "logger_tt/core.py", line 158, in _replace_with_socket_handler
  File "logger_tt/core.py", line 355, in __init__
  File "socketserver.py", line 452, in __init__
  File "socketserver.py", line 466, in server_bind
OSError: [Errno 48] Address already in use

Traceback:
Traceback (most recent call last):
  File "RimSort.py", line 36, in <module>
  File "logger_tt/__init__.py", line 225, in setup_logging
  File "logger_tt/core.py", line 88, in from_dict
  File "logger_tt/core.py", line 111, in _set_mode
  File "logger_tt/core.py", line 158, in _replace_with_socket_handler
  File "logger_tt/core.py", line 355, in __init__
  File "socketserver.py", line 452, in __init__
  File "socketserver.py", line 466, in server_bind
OSError: [Errno 48] Address already in use

As I understanding, there are 2 problems we are facing:

It runs on windows but output a bunch of Null when starting the child process.

Pretty much. I don't know why this happens when the app is frozen. When the app is running from interpreter, the log file just continues business as usual. It does not get cutoff and the process logs to the file with 0 issue using the handler.

It doesn’t run on macOS

Yeah, logger_tt crashes my Process on MacOS seemingly. The process is not spawned and subsequently the game is not launched, when the app is frozen. No issues when run from Python interpreter.

We will have to solve it one by one. As for the windows case, can you provide an instruction to build the environment and create an exe with pyinstaller?

Yep! It is a mod manager for RimWorld game however, but you might be able to trick it kinda. Let me know if you have any questions:

https://github.com/oceancabbage/RimSort/wiki/Development-Guide

You can skip building SteamworksPy. The libs are already pre-built and included in-repo. You do need the submodule however, so make sure you follow that part. Make sure you copy ./SteamworksPy/steamworks/ to the project root so that the app has the actual python module used to interface with those libs.

@Dragon2fly
Interesting development. I tried out compiling using Nuitka again, with my latest on Windows.
Frozen with Nuitka, it doesn't properly detect that the executable is frozen with sys.frozen and _MEIPASS checks? I had to just tab the freeze_support() back a bit to be outside of that check, as the check was failing (you can see it still thinks it is running from Python interpreter...???)
You can see below, the __name__ of the child process is __parents_main__ which is better than PyInstaller, because at least it's properly changing the name? I definitely like this behavior with Nuitka a lot more than the Pyinstaller'd behavior, as it made this a lot more functional in terms of the multiprocessing aspects:
PS C:\Users\localadmin\Desktop\RimSort\RimSort.dist> .\RimSort.exe
RimSort.py: <_MainProcess name='MainProcess' parent=None started>
__name__: __main__
sys.argv: ['C:\\Users\\localadmin\\Desktop\\RimSort\\RimSort.dist\\RimSort.exe']
steamworks.wrapper: <_MainProcess name='MainProcess' parent=None started>
__name__: util.steam.steamworks.wrapper
sys.argv: ['C:\\Users\\localadmin\\Desktop\\RimSort\\RimSort.dist\\RimSort.exe']
main_content_panel.py: <_MainProcess name='MainProcess' parent=None started>
__name__: view.main_content_panel
sys.argv: ['C:\\Users\\localadmin\\Desktop\\RimSort\\RimSort.dist\\RimSort.exe']
Failed to load libEGL (The specified module could not be found.)
QWindowsEGLStaticContext::create: Failed to load and resolve libEGL functions
Failed to load opengl32sw.dll (The specified module could not be found.)
 Steam is not running!
[2023-04-06 14:16:48] WARNING: Process-1 Unable to launch game with Steamworks API. Please close your game and try to launch again after opening Steam.

Ultimately, the child process appears to properly spawn when compiled with Nuitka. The issue with the logger_tt log being overwritten is still present however:

This makes me think there is more of an issue/limitation with Pyinstaller than I initially thought. Logger_tt is still having issue as well.
What are your thoughts on this?

@Dragon2fly
Yeah, officially, I think we are ditching PyInstaller for Nuitka in light of these developments.
Seeing the same improvement in frozen multiprocessing spawn() behavior when using Nuitka to "build" on MacOS:
$ ./RimSort
RimSort.py: <_MainProcess name='MainProcess' parent=None started>
__name__: __main__
sys.argv: ['./RimSort']
steamworks.wrapper: <_MainProcess name='MainProcess' parent=None started>
__name__: util.steam.steamworks.wrapper
sys.argv: ['./RimSort']
main_content_panel.py: <_MainProcess name='MainProcess' parent=None started>
__name__: view.main_content_panel
sys.argv: ['./RimSort']
[2023-04-06 21:08:49] WARNING: Running using Python interpreter
QPixmap::scaled: Pixmap is a null pixmap
RimSort.py: <_MainProcess name='MainProcess' parent=None started>
__name__: __parents_main__
sys.argv: ['/Users/user/Desktop/RimSort/RimSort.app/Contents/MacOS/RimSort', '--multiprocessing-fork', 'tracker_fd=35', 'pipe_handle=37']
steamworks.wrapper: <_MainProcess name='MainProcess' parent=None started>
__name__: util.steam.steamworks.wrapper
sys.argv: ['/Users/user/Desktop/RimSort/RimSort.app/Contents/MacOS/RimSort', '--multiprocessing-fork', 'tracker_fd=35', 'pipe_handle=37']
main_content_panel.py: <_MainProcess name='MainProcess' parent=None started>
__name__: view.main_content_panel
sys.argv: ['/Users/tstagg/Desktop/RimSort/RimSort.app/Contents/MacOS/RimSort', '--multiprocessing-fork', 'tracker_fd=35', 'pipe_handle=37']
Traceback (most recent call last):
  File "/Users/user/Desktop/RimSort/RimSort.app/Contents/MacOS/RimSort.py", line 36, in <module __parents_main__>
  File "/Users/user/Desktop/RimSort/RimSort.app/Contents/MacOS/logger_tt/__init__.py", line 225, in setup_logging
  File "/Users/user/Desktop/RimSort/RimSort.app/Contents/MacOS/logger_tt/core.py", line 88, in from_dict
  File "/Users/user/Desktop/RimSort/RimSort.app/Contents/MacOS/logger_tt/core.py", line 111, in _set_mode
  File "/Users/user/Desktop/RimSort/RimSort.app/Contents/MacOS/logger_tt/core.py", line 158, in _replace_with_socket_handler
  File "/Users/user/Desktop/RimSort/RimSort.app/Contents/MacOS/logger_tt/core.py", line 355, in __init__
  File "/Users/user/Desktop/RimSort/RimSort.app/Contents/MacOS/socketserver.py", line 452, in __init__
  File "/Users/user/Desktop/RimSort/RimSort.app/Contents/MacOS/socketserver.py", line 466, in server_bind
OSError: [Errno 48] Address already in use

Unfortunately, logger_tt still crashes the process. But it has a different __name__ at least!

Hi @twstagg
I don't have RimWorld installed so my environment is not the same as yours. I replaced the call to the RimWorldWin64.exe with notepad.exe and tested it with pyinstaller on Windows. But I cannot replicate the log file as yours.
Can you replicate it on another machine?
Can you provide a minimal reproducible code that doesn't require installing RimWorld?
As for the MacOS with Nuitka, from the logs you provided, it doesn't contain the line Logging server started!.
This indicates that the main process/main thread wasn't able to start the log server.
From OSError: [Errno 48] Address already in use message, the port that it tried to bind was being in use by other applications.
Could you please make sure the 9020 port is not in use, or change the port in the config file and test again?

Hi @twstagg
I added some modifications to detect child processes and print out some debug logs to logger-tt.
I would like to see the logs from your environment using the below package:
logger_tt-1.7.4-py3-none-any.whl.zip

unzip the above file
from within your venv, pip install -U [path_to_the_unzipped_file.whl]
do pyinstaller or nuitka as normal then run the exe to get the log

Hi @twstagg
I don't have RimWorld installed so my environment is not the same as yours. I replaced the call to the RimWorldWin64.exe with notepad.exe and tested it with pyinstaller on Windows. But I cannot replicate the log file as yours.
Can you replicate it on another machine? Can you provide a minimal reproducible code that doesn't require installing RimWorld?
As for the MacOS with Nuitka, from the logs you provided, it doesn't contain the line Logging server started!. This indicates that the main process/main thread wasn't able to start the log server. From OSError: [Errno 48] Address already in use message, the port that it tried to bind was being in use by other applications. Could you please make sure the 9020 port is not in use, or change the port in the config file and test again?

Hi @twstagg
I added some modifications to detect child processes and print out some debug logs to logger-tt. I would like to see the logs from your environment using the below package: logger_tt-1.7.4-py3-none-any.whl.zip
1. unzip the above file

2. from within your venv, `pip install -U [path_to_the_unzipped_file.whl]`

3. do pyinstaller or nuitka as normal then run the exe to get the log

Heyo. I think that this was more of an issue with my implementation. I've made a zillion changes to the structure of my application and am now doing things more intelligently. I just circled back to test this issue out again (with your 1.7.4 version provided above) and I don't have any of the previous issues.
Thanks for your response and discussion! I'll let you know if I run into more issues.

Ah, yes, on Windows the issue is still present with the log starting over and zeroing.
I don't see the same debug prints that you added on Windows that I do in MacOS though...

Hi @twstagg. Have you had any progress on Windows yet?
Since the problem was resolved on MacOS, I'm going to officially release v1.7.4 to PyPI.
Of course, it includes other fixes and improvements too.

 The logger setup was getting executed on re-import. Idk what i was thinking before. Thank you very much for your time and for the awesome piece of software btw!


Order of arguments in NewDispatcher event is backwards

Severity: Code Quality
Description
The convention followed in the rest of the system is event VarChanged(var oldVal, var newVal) but when updating the dispatcher, the values are reversed: event NewDispatcher(address Dispatcher, address oldDispatcher);

Will unify the naming.


Forms don't have dark mode support

https://metamask.io/institutions

The Request Access and Download Brochure forms in the Institutions page don't have the correct formatting for dark mode. The text is difficult to read with the dark background.
Reproducible on desktop and mobile browsers

I would love to make the CSS right for this form.

@jlazoff can I start on this?

@jlazoff can I start on this?

Not yet; we need to make internal changes to the form. I have already submitted the request. Once complete, aiming for EOW, the form should be fixed. 🚀

How can I contribute to Metamask?

All forms have been set to raw HTML for styling purposes. Over to you.

Thank you!On Feb 10, 2023, at 7:44 AM, SunnySian @.***> wrote:
All forms have been set to raw HTML for styling purposes. Over to you.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

Thank you all; we have resolved this issue. Closing ticket 🚀


Composter Fills with no items

Seems to be a bug. To recreate just use a composter with an empty hand and it fills up like an item is being used on it.
Tried on Cultural Delights 1.0.4 & 1.0.5 (1.20.1)

(Translation used)
This bug has been found since version 1.0.3 and is still present in the latest version. We need a way to fix this.

hopefully this gets fixed soon, it's a very strange bug and it makes it very difficult to use composters without accidentally getting extra free bone meal, hoppers being the only consistent option.

If someone needs this mod now
https://github.com/north82/Cultural-Delights-Fabric/releases/tag/1.20.1-1.0.5-fix

#34

Temp Fix with another mod
I had this issue even with the fixed version, especially on fabric.
If you install https://modrinth.com/mod/more-composting
and add to the config
this line "-minecraft:air"
it should fix it
it worked for me and I hope for others as well


use of aws-sdk

Hey ! i have query
What's the use of aws-sdk ?
https://github.com/geist-org/contributors/blob/8c71e1adeb923680d124d8b3e059aa9887d09a0e/package.json#L13

Well, in the original release, this forwarding service was hosted on the AWS service, and then was removed and make open source. (I forgot to delete the expired deps 😢 )


UnicodeDecodeError: 'charmap' codec can't decode byte

When running on Windows 11 with Python 3.10.6 I get the following error:
C:\Python\cpe_search-master>python cpe_search.py -q "Steam 2.91" -v 2.3
[+] Running initial setup (might take a couple of minutes)
[+] Downloading NVD's official CPE dictionary (might take some time)
[+] Unzipping dictionary
[+] Creating a custom CPE database for future invocations
Traceback (most recent call last):
  File "C:\Python\cpe_search-master\cpe_search.py", line 541, in <module>
    update(args.version)
  File "C:\Python\cpe_search-master\cpe_search.py", line 81, in update
    content = fin.read()
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 1599017: character maps to <undefined>

This was fixed by changing line 80 in cpe_search.py from:
with open(cpe_dict_filepath) as fin:

to:
with open(cpe_dict_filepath,encoding="utf8") as fin:

I can now use your awesome searcher :)
C:\Python\cpe_search-master>python cpe_search.py -q "Steam 2.91" -v 2.3
[+] Running initial setup (might take a couple of minutes)
[+] Downloading NVD's official CPE dictionary (might take some time)
[+] Unzipping dictionary
[+] Creating a custom CPE database for future invocations
[+] Cleaning up
cpe:2.3:a:ponsoftware:explzh:2.91:*:*:*:*:*:*:*
[('cpe:2.3:a:ponsoftware:explzh:2.91:*:*:*:*:*:*:*', 0.42640140575829194),
 ('cpe:2.3:a:valvesoftware:steam_client:-:*:*:*:*:*:*:*', 0.42640140575829194),
 ('cpe:2.3:a:encodable:filechucker:2.91:*:*:*:*:*:*:*', 0.40824812725634746)]

Hi @arbitmcdonald,
thanks for describing this issue and creating a pull request to resolve it :)


Undefined behavior

Many tests use the pattern https://github.com/DMTF/SPDM-Responder-Validator/blob/f83abe7cf7d5fbaa30b963bd321bca069e294eb8/library/spdm_responder_conformance_test_lib/spdm_responder_test_7_measurements.c#L1578-L1586 when searching for invalid slot IDs. test_buffer->slot_mask and slot_id are both uint8_ts. If slot_id is greater than 7 then the left-shift invokes undefined behavior.

From the code logic, if (slot_id < SPDM_MAX_SLOT_COUNT) is false, the (0x1 << slot_id) wont be executed.

We agree to close it at this moment. If we have real issue later, we can open that.


Add new PES SKU to Healthbot

ARM API Information (Control Plane)
Changelog
Add a changelog entry for this PR by answering the following questions:

What's the purpose of the update?

[ ] new service onboarding
[x] new API version
[ ] update existing version for new feature
[ ] update existing version to fix swagger quality issue in s360
[ ] Other, please clarify

When are you targeting to deploy the new service/feature to public regions? Please provide the date or, if the date is not yet available, the month. - End of May/Start oh June
When do you expect to publish the swagger? Please provide date or, the the date is not yet available, the month. End of May
By default, Azure SDKs of all languages (.NET/Python/Java/JavaScript for both management-plane SDK and data-plane SDK, Go for management-plane SDK only ) MUST be refreshed with/after swagger of new version is published. If you prefer NOT to refresh any specific SDK language upon swagger updates in the current PR, please leave details with justification here.

Contribution checklist (MS Employees Only):

[ ] I commit to follow the Breaking Change Policy of "no breaking changes"
[x] I have reviewed the documentation for the workflow.
[x] Validation tools were run on swagger spec(s) and errors have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.
ARM API Review Checklist

Applicability: :warning:
If your changes encompass only the following scenarios, you should SKIP this section, as these scenarios do not require ARM review.

Change to data plane APIs
Adding new properties
All removals

Otherwise your PR may be subject to ARM review requirements. Complete the following:

[x] Check this box if any of the following apply to the PR so that the label "ARMReview" and "WaitForARMFeedback" will be added by bot to kick off ARM API Review.  Missing to check this box in the following scenario may result in delays to the ARM manifest review and deployment.

Adding a new service
Adding new API(s)
Adding a new API version
-[x] To review changes efficiently, ensure you copy the existing version into the new directory structure for first commit and then push new changes, including version updates, in separate commits. You can use OpenAPIHub to initialize the PR for adding a new version. For more details refer to the wiki. Note that this doesn't apply if you are trying to merge a PR that was previously in the private repository.

[x] Ensure you've reviewed following guidelines including ARM resource provider contract and REST guidelines. Estimated time (4 hours). This is required before you can request review from ARM API Review board.

[x] If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist
If you have any breaking changes as defined in the Breaking Change Policy, request approval from the Breaking Change Review Board.
Action: to initiate an evaluation of the breaking change, create a new intake using the template for breaking changes. Additional details on the process and office hours are on the Breaking Change Wiki.
NOTE: To update API(s) in public preview for over 1 year (refer to Retirement of Previews)
Please follow the link to find more details on PR review process.

@Wzb123456789 Hi. There are some Checks that failed. Is this blocking my PR?

@guy-microsoft Please check this PR so that I can check bugs with the SDK owner.

@Wzb123456789 This looks good to me. Please proceed.

Hi @Wzb123456789 can you proceed to approve?

@raych1. Should I be able to merge the PR? It seems that I don't have authorization to merge.

@raych1. Should I able to merge the PR? It seems that I don't have authorization to merge.

@Wzb123456789  Can you help me with merging?

@raych1. Should I be able to merge the PR? It seems that I don't have authorization to merge.

@arieschwartzman , can you resolve the comment from @qiaozha ?

@raych1. Comments resolved


Adding documentation for debugging with Docker

To address #515, the CSS comment may belong better somewhere else, not sure where though

👏 @mjsisley


Select Yorkie API endpoint by JEKYLL_ENV

Currently, while testing the yorkie-homepage, we have to change the API endpoint manually.
So I introduced JEKYLL_ENV and added a condition statement with jekyll.environment.

@lqez Thanks for the contribution.


Initialize adding the Chinese translation of the fpm documents

[x] add the Chinese translation.

CN Pages: https://zoziha.github.io/fpm-docs/cn/

I think cn is not matching any language sphinx-intl is aware of, however it has two entries for Chinese:

zh_CN – Simplified Chinese
zh_TW – Traditional Chinese

Using either of those would also use translations for the blog and table of contents. If you prefer cn as path for the URL I can have a look how we can keep this path and still get the right translation from the sphinx-intl for the theme.

Thanks, I'll replace cn with zh_CN.


[Event Request] codeunit 333 "Req. Wksh.-Make Order" - Event for procedure CheckInsertFinalizePurchaseOrderHeader

Can we get the following events for the "Req. Wksh.-Make Order" Codeunit: OnBeforeCheckInsertFinalizePurchaseOrderHeader
    local procedure CheckInsertFinalizePurchaseOrderHeader(RequisitionLine: Record "Requisition Line"; var PurchOrderHeader: Record "Purchase Header"; UpdateAddressDetails: Boolean): Boolean
    var
        CheckInsert: Boolean;
    begin
        with RequisitionLine do
            CheckInsert :=
              (PurchOrderHeader."Buy-from Vendor No." <> "Vendor No.") or
              (PurchOrderHeader."Sell-to Customer No." <> "Sell-to Customer No.") or
              (PrevShipToCode <> "Ship-to Code") or
              (PurchOrderHeader."Order Address Code" <> "Order Address Code") or
              (PurchOrderHeader."Currency Code" <> "Currency Code") or
              (PrevPurchCode <> "Purchasing Code") or
              CheckAddressDetails("Sales Order No.", "Sales Order Line No.", UpdateAddressDetails);

        ///
        OnBeforeCheckInsertFinalizePurchaseOrderHeader(RequisitionLine, PurchOrderHeader, CheckInsert, OrderCounter, PrevShipToCode, PrevPurchCode, UpdateAddressDetails); ///NEW
        ///-
        exit(CheckInsert);
    end;

   ///
    // [IntegrationEvent(false, false)]
    //local procedure OnBeforeCheckInsertFinalizePurchaseOrderHeader(RequisitionLine: Record "Requisition Line"; var PurchaseHeader: Record "Purchase Header"; var CheckInsert: Boolean; var OrderCounter: Integer)
    [IntegrationEvent(true, false)] ///NEW
    local procedure OnBeforeCheckInsertFinalizePurchaseOrderHeader(RequisitionLine: Record "Requisition Line"; var PurchaseHeader: Record "Purchase Header"; var CheckInsert: Boolean; var OrderCounter: Integer; var PrevShipToCode: Code[10]; var PrevPurchCode: Code[10]; var UpdateAddressDetails: Boolean) ///NEW
    ///-
    begin
    end;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


[DO NOT MERGE] Unpublish special doc

This pull request was originally opened on specialist-publisher https://github.com/alphagov/specialist-publisher/pull/695
There was no comment on the original pull request at the time of this migration.
Original comment:
Trello Card
##This is a PR opened to get more comments and is not feature-complete.
##Questions

Shall we call it 'withdraw' or 'unpublish' ?

master branch uses withdraw
trello card has been updated to call it 'unpublish'

By sending a 'gone' format, the document is now lost and unretrievable in publishing-api. I believe this requires an extension of publishing-api.

This PR will need more development when publishing-api has been extended to support withdrawals.


fix: taos-optin api entry default

Codecov Report

Merging #263 (e671561) into main (4c7c348) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #263   +/-   ##
=======================================
  Coverage   58.29%   58.29%
=======================================
  Files         102      102
  Lines       25264    25264
=======================================
  Hits        14728    14728
  Misses      10536    10536

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Dubbing: Receiving target language empty even though sending a target language

Issue
Using the dubbing api i'm receiving a 400 error that my request is missing the target_lang even though it is provided
Expected
pretty straightforward, the targetlanguage should work
Example code
const response = await elevenlabs.dubbing.dubAVideoOrAnAudioFile({
      mode: "automatic",
      source_lang: "en",
      target_lang: "de",
      file: video,
    });

and the response:

@asafchn do you mind trying the request via cURL and confirming that the API doesn't throw? We would love to narrow whether or not the error is specific to the SDK

Same issue here:
Error during dubbing request: {
detail: {
status: 'empty_target_language',
message: 'Target language must be set in automatic dubbing mode. Either set the target language or set dubbing studio to true.'
}
}
POST /api/tools/video-dubbing 500 in 220ms
Not using the SDK, but direct API call in NextJS. This only happens on "URL" mode, not on the file-upload.

@lharries it looks like the user is saying that this doesn't work in the raw API

Same error here, also using source_url. Curl works perfect but all of this are failing with this error for me:

Javascript SDK
Native fetch (form-data and JSON)
npm node-fetch (form-data and JSON)

In form-data (multipart/form-data) the error is different: "Missing boundary in multipart"
Hope it help with the debug because it looks like is an API issue.

Thanks for bearing with us folks, we've tested and things are looking good on version 0.8.2.


Changes default file extension for screenshots to .png

Solution for #54

Thanx! Was unsure about that so I guessed at jpg


Update how to build flax docs

Codecov Report

Merging #2740 (6f76a27) into main (4960e96) will increase coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #2740      +/-   ##
==========================================
+ Coverage   81.19%   81.22%   +0.03%
==========================================
  Files          53       53
  Lines        5626     5625       -1
==========================================
+ Hits         4568     4569       +1
+ Misses       1058     1056       -2

Impacted Files
Coverage Δ

flax/linen/spmd.py
84.68% <0.00%> (-0.91%)
:arrow_down:

flax/serialization.py
94.56% <0.00%> (+0.09%)
:arrow_up:

flax/core/frozen_dict.py
96.33% <0.00%> (+0.10%)
:arrow_up:

flax/training/train_state.py
83.33% <0.00%> (+11.11%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

The doc build fails, could you try rebasing to main? I think it should be fixed by now.


Fix comments on 'get bindings'

Trivial Fix: add namespace into supported list
actually namespace is supported when retrieve the bindings

/ok-to-test

/retest

@jichenjc Thanks for the fix! Here are the instructions for how to regenerate the golden files, those files are out-of-date with your recent change, and that is why the unit tests are failing.
https://svc-cat.io/docs/devguide/#golden-files

@jichenjc this looks good but you need to regenerate the golden files used by unit tests so it will pass.  See https://github.com/kubernetes-incubator/service-catalog/pull/2355#issuecomment-424375914 for details.

/lgtm


Create a sample app with the jspm-git approach

It could even be a skeleton-navigation clone

I believe (but am not sure) that if you define the "bower based endpoint" in the package.json, then jspm will add the reference to jspm-git to the (jspm generated) config.js file
This nearly killed me trying to get rid of references to jspm-git - only when I replaced the intent to use bower based Kendo distribution with GitHub KendoUI-core, I got rid of jspm-got

So, I have just confirmed that with the current setup (having kendo-core as dep in the plugin's package.json), you have to override this dependency when installing the plugin.
Turns out it's easier to do this by using map than overriding dependencies, because if you override dependencies you have to list repeat all dependencies of the plugin and then change one (kendo-ui). if you override map then you only need to specify the dependency you wish to override.
jspm install github:aurelia-ui-toolkits/aurelia-kendoui-plugin -o "{ map: { 'kendo-ui'<EMAIL_ADDRESS>} }"

I am assuming that<EMAIL_ADDRESS>is a reference to the PRO version?
It's really good that you are pushing the "smooth" KendoUI PRO plugin installation as this is really where it matters the most.

You are correct in assuming that it is the PRO version of kendo. This was installed by running jspm install kendo-ui=kendo:bower-kendo-ui after installing  jspm-git and registering the kendo endpoint with jspm
Perhaps we can remove the version number from the override :open_mouth:

The sample repository can be found here. The README contains some clues on how to get the app running and how the steps it took to include our plugin (using jspm-git) in a new skeleton-navigation app

Like the README.md a lot!

I have moved the sample app here


Update jeremy-likness.yml

Refresh bio, add video and update location.

Hey @JeremyLikness,  Extremely sorry for delay this time. Got some issues from other PRs and tried to get in all at one go, as it involves lots of validation before go live.
Finally , all bits are tested live...Happy weekend!
Let me know if you dont see your changes.


when using qianfan chat streaming occurs TypeError: Additional kwargs key sentence_id already exists in left dict and value has unsupported type <class 'int'>.

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code
qianfan_model = QianfanChatEndpoint(model='ERNIE-Bot', qianfan_ak=...... ,
                                    qianfan_sk=......)

for chunk in qianfan_model.stream(input_prompt):
    all_output += chunk.content
    print(chunk.content, end="", flush=True)

Error Message and Stack Trace (if applicable)
Traceback (most recent call last):
File "E:\JMIT\JMIT3\code\ans.py", line 247, in
print(retrieve_ans_chat_stream('我想離婚，孩子該怎麼判？', history=[('你好', '你好呀')]))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\JMIT\JMIT3\code\ans.py", line 236, in retrieve_ans_chat_stream
for chunk in qianfan_model.stream(input_prompt):
File "E:\envs\JMIT\Lib\site-packages\langchain_core\language_models\chat_models.py", line 250, in stream
raise e
File "E:\envs\JMIT\Lib\site-packages\langchain_core\language_models\chat_models.py", line 241, in stream
generation += chunk
File "E:\envs\JMIT\Lib\site-packages\langchain_core\outputs\chat_generation.py", line 57, in add
generation_info = merge_dicts(
^^^^^^^^^^^^
File "E:\envs\JMIT\Lib\site-packages\langchain_core\utils_merge.py", line 38, in merge_dicts
raise TypeError(
TypeError: Additional kwargs key created already exists in left dict and value has unsupported type <class 'int'>.
Description
This bug occurs when the langchain_core>0.1.12
Downgrading to langchain_core==0.1.12 is bug-free
System Info
langchain==0.1.10
langchain-community==0.0.25
langchain-core==0.1.28
langchain-text-splitters==0.0.1
windows
Python 3.11.7

The logic of chunk merging is relatively complicated. I think it is possible to formulate some keys that need to be merged.

@baskaryan Please give me some suggestions and I will fix it.

Same issue. any update on this error?

Same issue. any update on this error?

It has been solved, please see the associated PR.https://github.com/langchain-ai/langchain/pull/18889

Same issue. any update on this error?

It has been solved, please see the associated PR.#18889

Thanks. Update to the latest langchain can fix this issue.


cephfs: add Fsync and Sync calls for open files

Add wrappers for ceph_fsync. The Fsync call directly wraps ceph_fsync
which provides options to behave more like fsync or more like fdatasync.
Add Sync, a simpler wrapper over Fsync, to match any interfaces that
make use of File.Sync from Go's os package.
Fixes: #250
The tests here are a little on the weak side because we don't really do the extra heavy lifting to verify the flags do what they say on tin, but I trust this part of ceph to work and our wrapper is pretty trivial.
Checklist

[x] Added tests for features and functional changes
[x] Public functions and types are documented
[x] Standard formatting is applied to Go code

@nixpanic @ansiwen any blockers here? The conversation on this seems to have petered out a bit.

@nixpanic @ansiwen any blockers here? The conversation on this seems to have petered out a bit.

I would like to see the change in name(s) for SyncChoice values.

@nixpanic @ansiwen any blockers here? The conversation on this seems to have petered out a bit.

I would like to see the change in name(s) for SyncChoice values.

Ack. Funny, I thought I had changed it but I guess I forgot.

I think I finally managed to push everything I previously forgot to push.
@nixpanic , @ansiwen, @agarwal-mudit PTAL

Not sure what changed, but the PR was modified after @agarwal-mudit reviewed it, so he should have a look again.


Add a bunch of super interesting projects to the wiki

Thanks @utilForever ❤ for sharing:

https://github.com/utilForever/CubbyTower
https://github.com/utilForever/Runeterra
https://github.com/CubbyTeam/Cubby
https://github.com/utilForever/PokeMaster

Moreover, these ones added the #entt tag 🙂 as suggested in the readme:

https://github.com/Electro-Technologies/Electro
https://github.com/Mathieu-Lala/Kawaii_Engine
https://github.com/Doctor-Foxling/Becketron

Updated the wiki. @utilForever do Cubby and PokerMaster use EnTT already or is it only something planned for the future?


[usecase][XWALK-2620] Update usecase for Vehicle API

Impacted tests(approved): new 0, update 1, delete 0
Unit test platform: [IVI]
Unit test result summmary: pass 1, fail 0, block 0

Please prefix the bug id in commit.

@zqzhang ,I have updated it following your comments


using same ASP.net core identity for multiple web applicaitons

Hello
I want to have 3 different web applications but want to use same identity back end application so i have SSO. im using Individual user accounts
thanks

You can check the eShopOnContainers template, there is an centalized Identity service : https://github.com/dotnet-architecture/eShopOnContainers

I was hoping its something that could be used with already made identity individual user authenticaton

You could use Identity Server as the login point, and then connect your web sites to that via OIDC

thats one thing i was looking at but for now i was still looking at something that could work with what i already have(i have on onee web app accounts already and i dont want migrating accounts if it isnt neccecery)

You don't need to migrate your user data to use IdentityServer.

didnt know that

@brockallen just a quick question for information if it is needed in future is there a way to use identityserver for logins in lets say console app, (basicly login in console app and for backend it uses identity server)

Check the docs page: http://docs.identityserver.io/


[Bug]: Issue with cloning code commit repo - Argo CD automation

Installation method
Workshop Studio
What happened?
Initial Setup:
Step:
git clone $(cat ~/environment/argocd_repo_url) ~/environment/argocd

Error:
Cloning into '/home/ec2-user/environment/argocd'...
Access denied: User: arn:aws:iam::XXXXXXX:user/eks-workshop-gitops is not authorized to perform: codecommit:GitPull on resource: arn:aws:codecommit:us-west-2:xxxxxxxx:eks-workshop-argocd because no identity-based policy allows the codecommit:GitPull action
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

What did you expect to happen?
Git clone should have been successful
How can we reproduce it?
Following the steps will recreate the issue
Anything else we need to know?
IAM User "arn:aws:iam::xxxxxx:user/eks-workshop-gitops" has permissions to code commit repo eks-workshop-argocd but the code commit repo is created with eks-workshop-gitops name
{
    "Statement": [
        {
            "Action": [
                "codecommit:GitPush",
                "codecommit:GitPull"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:codecommit:us-west-2:xxxxx:eks-workshop-argocd",
            "Sid": ""
        }
    ],
    "Version": "2012-10-17"
}

EKS version
1.23

Hi @ramasamy-seranthaiya were you trying to use eksworkshop.com content with Workshop Studio? Or the specific content referenced by the Workshop Studio content, which is currently https://build-ee0efb6.eksworkshop.com/docs/introduction/ide

Hi @niallthomson Yes, I was trying to use eksworkshop.com content with Workshop studio. I missed the build version of the content

Verified the workshop with the Workshop Studio Content and the labs worked fine. Closing the issue as non-issue


Adds brkt-env option

Currently metavisors determine what yeti to
contact by reading an s3 bucket. The brkt-env
option will tell the encryptor what address to
write into the metavisor image.
Testing: launched encryptor instance and verified
userdata was set properly.

Is this for developers or end users?  It's an internal option, we should make it hidden.

it is for developers. I hid the brkt-env arg.

My bad.  I wasn't paying attention.

Nice.. lgtm


The story for distributed caching is strictly better for volumes than layers. Multiple machines building containers can't share layers at all. Or, maybe they can through a third party wrapper, but that's a huge amount of config, too. It doesn't work out of the box - I'm pretty sure about that.

I don’t understand how it’s better with volumes, with volume layers you just need --cache-from and a registry (that you probably already have if you’re to the point where you have multiple build machines)

depending on where your builder are hosted you might not have the option to use this feature

How is that different from needing nightly cargo? I expect it would be harder to build caching through cargo than through buildkit for a while. Buildkit is not really experimental anymore, as it's been around for years, is widely supported, is the officially blessed way forward for all of Docker, is shipped with standard Docker distros, and will soon be enabled by default for most distros.

it’s different because it’s often easier to have rust nightly inside a build container than being able to tinker with the docker daemon configuration and/or build process

if you do use this feature, it’s not necessarily easy to configure it properly (think about concurrency or different build configurations (features, architecture, etc.?)

Cache layers have absolutely nothing they can do to avoid invalidating the entire layer when a single file used as input is touched in any way. Cache volumes never perform worse than that and often perform much better.

--only-deps would only be useful when there is no changes, true. using --only-deps does not prevent to use volume cache for when there is a cache invalidation.

Concurrent builds - yes it could be a problem, as I think Cargo just acquires a lockfile while building, but it's nothing that cache layers solve. They just "solve" the issue by never sharing work between machines. It's easy to do that for cache volumes, and is the default behavior anyway.

it’s not sharing work across machine that’s an issue, concurrent build could happen on the same machine as well

the build will then make an entirely new image layer, most likely a big one, and that will cost a lot of storage and/or bandwidth.

Hmm. This might be an actual thing. I believe that it would only create a new layer this in cases where a cache layer would have been rebuilt, anyway, but I'm only about 50% sure of that.

I’m a 100% sure :)

it will still not be as quick as docker detecting that the layer already exists in cache

Why not? Cargo and Docker have to do essentially the same op when scanning for cache invalidation. If there's a time difference at all, it's negligible, order of ms.

docker will only have to see that the Cargo.toml and Cargo.lock files are unchanged. ie: the last dockerfile commands are cached too), it won’t look into anything else at all.
no matter how many dependencies (and workspaces supposing those are handled too) docker won’t have anything to scan

here is an exemple of how it looks like:

the full line 12 is :

RUN --mount=type=cache,id=yarn-$APP-build,target=/usr/local/share/.cache/yarn
yarn install --frozen-lockfile --no-progress --non-interactive

the same lines appears at line 14 because this is a multi stage build and it detected that the same layer from the builder-prod can be reused
you can notice that I’m also using volume cache for when the lockfile changes
when the whole image is already in cache it takes 0.4 second to build from scratch:

otoh I understand very well that the cargo works might make it difficult to implement. but there is

That copies a single Cargo.toml file doesn't it? In a workspace with 10 sub-packages, you need to copy 11 Cargo.toml files.

Please stop pushing for BUILDKIT until docker actually starts to enable it by default. It's not relevant for a majority of docker users yet and I think it's offtopic, as well as the discussion about build clusters and sscache.

I'm afraid that command copies all the Cargo.toml files in the same destination

My post might get berried again, but did anyone have any comments for the workaround (when at least needing to use docker) I detailed in https://github.com/rust-lang/cargo/issues/2644#issuecomment-544820077 . It simply caches cargo files via multistage build layers, no BUILDKIT or advanced volume juggling required.
One could even break up the /tmp folder into /tmp{1,2,3,...} to cluster packages into breakable docker cache layers, depending on the degree of volatility of the upstream packages in your workspace to help save more build time for just the downstream packages in active development.
I feel like a 3rd party rust cli could help shuffle the skeleton of the workspace around so one doesn't need to lean on bash. In other other build tools, like colcon in the ROS2 ecosystem, one can specify --packages-up-to to get a list of upstream package dependencies in the current workspace with respect to a specific package. That could then be used to programmatically filter the workspace into a corresponding hierarchy of docker image layers.
That said, having something like cargo build --dependencies-only that didn't need to check for any main.rs or lib.rs files could help simplify the last RUN directive in the workspace_cache stage.
# Filter or glob files to cache upon
RUN mkdir ./cache && cd ./ws && \
    find ./ -name "Cargo.*" -o \
      -name "lib.rs" -o -name "main.rs" | \
      xargs cp --parents -t ../cache

@ruffsl I don’t quite ‘get’ it. How does the caching work? I would think as soon as someone changes a source file, docker would invalidate that cache layer as all source files were loaded into it.
On the buildkit caching front I have ~0 probability that docker will hit the same machine. Only by caching in an base image layer do I get reuse but that’s good enough (my docker is spelled kaniko, which does have its own way of caching but whether I can get to it is another question)
https://github.com/nacardin/cargo-build-deps looks promising...
Maybe we should all help with cargo-build-deps and continue the discussion over there?
Sent with GitHawk

https://github.com/nacardin/cargo-build-deps looks promising...

I tried it and it doesn't work.
And besides, it still solves the wrong problem. It still involves manually recreating the workspace skeleton in the Dockerfile, which is cumbersome at best and not feasible for more complex projects.
The problem is not building dependencies but realiably and automatically recreating a cacheable workspace skeleton, taking into account all cargo features & complexities.
This issue should just be closed, it's going nowhere.

For me personally, the buildkit cache trick works, but there is now a big documentation issue: So many Dockerfile examples on the web still use the old hacky way with fake lib.rs etc.
My searches led me to this issue, but I didn't bother searching through hundreds of comments to find out the better solution.

It's been said already:

PHP's Composer allows me to just copy composer.json and composer.lock and have composer install be cached by Docker.
JavaScript's NPM allows me to just copy package.json and package-lock.json and have npm ci be cached by Docker.

Rust not having something palatable for a standard docker layer cache solution makes it a less good tool for many real world cases.

This issue should just be closed, it's going nowhere.

If you don't see this as a problem worth thinking about, then maybe not partake in the discussion?
Maybe we can solve the simpler problem first: single project builds maybe also just those without path dependencies. Maybe we can find some 80/20 solution that solves most cases, but not all of them?

If you don't see this as a problem worth thinking about, then maybe not partake in the discussion?

See what as a problem, that's the question. This issue is about the --deps-only option. The problem you talk about is integrating cargo in docker builds efficiently. The --deps-only option is presented here as a solution, which is pretty confusing, because it's at best a second step and it might not actually even be necessary (in case a skeleton is generated in a clever way with perhaps adjusted mtimes etc.).
A new issue with explicitly a topic of docker integration and not necessarily involving --deps-only would make more sense to me.

Maybe we can solve the simpler problem first: single project builds maybe also just those without path dependencies. Maybe we can find some 80/20 solution that solves most cases, but not all of them?
Perhaps. If you're inclined that way, feel free to go ahead and implement something like that. I am personally not motivated to do that because a solution that only works for simple cases wouldn't solve problems for me.

@ruffsl I don’t quite ‘get’ it. How does the caching work? I would think as soon as someone changes a source file, docker would invalidate that cache layer as all source files were loaded into it.

@gilescope , the approach in https://github.com/rust-lang/cargo/issues/2644#issuecomment-544820077 only busts the docker build cache layer that cargo fetchs/builds the dependencies if any of the Cargo .toml or .lock files change; any other source file changes would not break that particular layer in the build stage of the multistage Dockerfile.
Your still correct in that any source file change not ignored by .dockerignore would break docker build caches given the scope of the COPY ./ ./ directive, but only for the cache stage of the multistage Dockerfile. The cache stage is where one can invoke arbitrary logic on what should or shouldn't be used to bust cache layers in later stages based on how you'd like to shuffle and prioritized files.
In my example approach, the cache stage is where I make a copy the workspace using the same parent directory hierarchy, but that only includes Cargo.* files, and substitute any main.rs or lib.rs files with skeleton place holders. This skeleton workspace is what is first copied from the cache stage into the build stage, and will be byte-for-byte the same in terms of file content (docker seems to ignore file metadata like timestamps) as long as no files used in the making the skeleton workspace are changed (or added or removed).
Later in the build stage, the rest of the source files are copied over, and is where other source files changes would end up breaking the docker build cache as desired. This can of course be customized by daisy-chaining or parallelizing stages to how ever complex you'd want your multistage build to be.

https://github.com/nacardin/cargo-build-deps is not working because of Cargo.lock parsing, so I created this repository https://github.com/errmac-v/cargo-build-dependencies based on cargo-build-deps. I think it would be useful until there is no general solution.

I'de like to give my perspective.
One thing that draws people to rust is the whole integrated ecosystem around the language. Installing a cross compiler toolchain used to be a dependency nightmare between compilers, libraries and scripts. It now is a simple rustup command.
So you want unit tests? Find a fitting framework, include it in your development pipeline, teach the developers to use this specific framework... A whole kludge. With rust, it comes included cargo test
A trade-off of the language design is compilation takes longer in favour of no-cost abstractions and guarantees. So at some point any rust developer will run into the need to optimise the build times.
Within this which integrated ecosystem, where lots of used-to-be hard problems are taken care of, and a significant portion of build time spent on building dependencies it almost feels natural and expected that something like cargo build --dependencies-only exists.
I now understand , by following this thread, that there are quite some corner cases and different explantations of what a 'dependency' is and what such flag should do. Yet such flag should exists, because it fit's the expectation of the ecosystem and because of the language design any project will at some point look at and try to optimise their compilation times.
The alternative presented in this thread, create a dummy project and copy cargo.toml and .lock over, is exactly the cludge we used to have and which shouldn't exists in the ruste ecosystem.
The flag should cover most common use cases (only external, versioned, dependencies) and have very precise and specific wording in what the flag does. This to make it clear which cases are and aren't covered. And also signal future maintainers what the intention is and prevent feature creep in the future.

And in which a significant portion of build time spent on building dependencies it almost feels natural and expected that something like cargo build --dependencies-only exists.

What do you need the flag for? What would be the point? Let me reiterate a key point here which I originally also didn't realize and implemented  --dependencies-only in vain:
The --dependencies-only flag does NOT solve the problem with dependencies build caching in Docker images
... unless it only needs the project's skeleton like in @errmac-v 's implementation.
If you run cargo build --dependencies-only in Docker on the full project, the resulting build, even if only containing dependencies, will be tied to the exact state of the project's sources and will be trashed as soon as the sources change. This is why you need a dummy project / skeleton - you need a representation of the project independent of the current state of the sources.

The alternative presented in this thread, create a dummy project and copy cargo.toml and .lock over, is exactly the cludge we used to have and which shouldn't exist in the rust ecosystem.

Maybe, but there's nothing that cargo can do about this, it's just the way docker works. Without some change in docker, you could only alleviate this by having a more automated way of generating the skeleton, but that's about it.

Well, it would still let us implement the cludge without all the problems associated with having to remove the correct fingerprint files, partial bulds, etc.
I can't count how many times I've set up a build pipeline only to have the production image just print Hello world and exit because there was some fingerprint file I forgot to delete. This has also broken because of Rust upgrades where new files were being emitted but not cleaned up by my build steps.
Right now you need to

Create a dummy project (simple, and just like most other things)
Copy the Cargo.toml file over (simple, and just like most other things)
Build test and release builds, to make sure all dependencies are downloaded, delete the fingerprints for release and debug builds without removing the compiled dependencies, delete the partial build files for the debug and release files, and make sure this works even after Rust/Cargo is upgraded, without really having a way to test it. (Brittle and hard)
Copy sources over and continue normally (simple, and just like most other things)

What do you need the flag for? What would be the point? Let me reiterate a key point here which I originally also didn't realize and implemented --dependencies-only in vain:

I gave an example earlier in the thread: I'd like this for profiling the compilation times of projects.

@Mange

3. Build test and release builds, to make sure all dependencies are downloaded, delete the fingerprints for release and debug builds without removing the compiled dependencies, delete the partial build files for the debug and release files, and make sure this works even after Rust/Cargo is upgraded, without really having a way to test it. (Brittle and hard)

I've also ran into this and I believe this is an issue with docker whereby docker doesn't update the mtime to what the source file has and instead leaves mtime from a cached layer, which confuses cargo. Right now I just use touch to update mtimes of all source files, which seems easy enough. IMO the best solution would be to have an option to update mtime in docker.
I feel like docker's shortcomings are constantly being shifted to cargo here, with demands to fix them by hacking up things in cargo.
@lnicola

I gave an example earlier in the thread: I'd like this for profiling the compilation times of projects.
Okay, but in that case would it be better to have an option to output profiling info directly? That would probably entail a lot more useful profiling detail compared to running time cargo build --dependencies and subtracting that from cargo build ...

I've also ran into this and I believe this is an issue with docker whereby docker doesn't update the mtime to what the source file has and instead leaves mtime from a cached layer, which confuses cargo.

I think the actual issue is that I wrote the source files a few minutes ago, while the generated project file was a few seconds ago. Ergo the Hello World file is newer and an older version is copied to the image.

Right now I just use touch to update mtimes of all source files, which seems easy enough. IMO the best solution would be to have an option to update mtime in docker.

It hurts when doing partial compilation, though. I usually want to share images between dev and release builds by using the same "dependencies are built" layers. So even when doing local "compiled on every save" I expect to be able to mount the correct directory and have things work.
Which they do, unless you consider someone checking out the code for the first time and then compiling the dev build. Then the dummy becomes newer and the project is not recompiled.

I feel like docker's shortcomings are constantly being shifted to cargo here, with demands to fix them by hacking up things in cargo.

While I really don't disagree with this sentiment, I also understand why people expect things to behave this way since Cargo is the black sheep. Almost every other kind of project separates dependencies from the project itself: Ruby, JS, C, C++, Python, and so on all have a "install dependencies" step and then you compile/build/test the project.
This is probably why Docker was even designed they way it was. And sure, Cargo has no obligation to work well with Docker, but the thought that we're sooo close to it working like most other langagues (e.g. "work with Docker's workflow") really makes you wish for a simple step to get rid of the hacks.

What if cargo build could work without a src dir? Then we won't need to run cargo new and we only have to copy the Cargo.toml file and build. No mtime issues either. This would also bring us even closer to some other languages' flow.
(Assuming that if you have something more exotic in Cargo.toml, you'll have to set it up anyway)

I've also ran into this and I believe this is an issue with docker whereby docker doesn't update the mtime to what the source file has and instead leaves mtime from a cached layer, which confuses cargo.

I think the actual issue is that I wrote the source files a few minutes ago, while the generated project file was a few seconds ago. Ergo the Hello World file is newer and an older version is copied to the image.

I thought so too, the first thing I tried was setting the timestamp to an artificial old date when the skeleton was generated. But it solved nothing. I think the issue is that docker's COPY command updates the file contents but will re-use some weird cached mtime if the file is already there from the previous layer. Maybe I don't understand docker enough, not sure.

And sure, Cargo has no obligation to work well with Docker, but the thought that we're sooo close to it working like most other langagues (e.g. "work with Docker's workflow") really makes you wish for a simple step to get rid of the hacks.

I don't understand what makes you think we're close, we're not. Solving this in cargo is hard both design-wise and implementation-wise (I know because unlike those downvoters I tried). Especially if you're going for a general solution, which I believe is the only approach that could actually be merged into cargo. Why ask for an 80% semi-hacky solution, which can't be accepted into cargo? That sort of thing belongs in a 3rd party crate...

[...] but the thought that we're sooo close to it working like most other langagues [...]

I don't understand what makes you think we're close, we're not.

I meant from the end-user's perspective. We're "only" missing a single command to cover the majority (I assume) of cases - small binary apps for use internally at companies or in people's homes.

Why ask for an 80% semi-hacky solution, which can't be accepted into cargo?

I'm not asking for a hack. I've been patient here for a long time now and would prefer a solid solution.
I just wanted to adress the seeming confusion on why so many people want a command to build all dependencies. It's very expected from a lot of developers coming from this side of the ecosystem; and developers coming from embedded, driver dev, GUI, etc. might not see the perspective here.
I'm not assuming your background, but at least I can share mine to perhaps help with the understanding.
I'm also a fan of being pragmatic. Rust 1.0 was not perfect but it was shipped anyway because it was better to release something than to stay in 0.x forever. I like that perspective. If we could solve a problem for 90% of cases and then work towards the other 10% if still needed, I think that's better than not doing anything. I also think that most of the exceptional cases here don't even assume or need this capability. If you have a monorepo or build a while bunch of crates from the same codebase, then you are probably unlikely about to package it as a Docker container to ship it to production in the same way as you build it for dev.
Right now we have people that seem to want to build dependencies because they are working on smaller projects where setting up custom CI software, or cache layers for Docker, or custom storage drivers is just too much to ask. I don't see comments from large teams with infra teams that work only with Rust and can afford to invest time in building their own CI services, so right now we don't have enough information on how to design for them. I don't think it's misguided to design a solution for the people with the problem and letting the others be a known unknown until more information is received.
No matter what, I appreciate that you are thinking about this problem and raising very valid objections. We can best avoid the XY problem if we keep asking questions. I don't think we should blindly accept solution X because people ask for solution X; better we describe problem Y and solve it with either X or Z.
I think you are mostly right; there's a risk that the solution doesn't cover 90%, but only like 40% while being a large foot-gun for the other 60%. We don't want that shipped anywhere near Cargo.

@Mange Ok, thanks for writing that and for being patient here.

It's very expected from a lot of developers coming from this side of the ecosystem; and developers coming from embedded, driver dev, GUI, etc. might not see the perspective here.

Well, that's true, but also I think it's that, as far as I know, tools like npm or composer are simpler in principle compared to cargo, which is why people may not realize that the request to add a --dependencies-only option to cargo requires a lot more consideration compared to those tools. At least that's my impression...

We can best avoid the XY problem if we keep asking questions. I don't think we should blindly accept solution X because people ask for solution X; better we describe problem Y and solve it with either X or Z.

Yeah. Asking for a not-perfect-but-still-viable solution is fine and the point about Rust 1.0 is valid, but it is really unclear to me where the line should be drawn, ie. what should (not) be supported and in particular how to do that such that the user is not confused (having a build flag that works 80% of the time may be pretty confusing).
I'm thinking that maybe a more straightforward approach would be to have a command or option to serialize the bulid of dependencies into a file (which could be checked in VCS), kind of like Cargo.lock, but for building dependencies, ie. there would be more information (such as build flags etc). You could then copy a single file (even for a large workspace or so) and run cargo based on that. Cargo already has an (ustable) option to output build plan, this could essentially be a subset of the build plan.

I'm thinking that maybe a more straightforward approach would be to have a command or option to serialize the bulid of dependencies into a file (which could be checked in VCS), kind of like Cargo.lock, but for building dependencies, ie. there would be more information (such as build flags etc). You could then copy a single file (even for a large workspace or so) and run cargo based on that. Cargo already has an (ustable) option to output build plan, this could essentially be a subset of the build plan.

I was about to suggest something similar - I this this could be a good way to go. Even if build plan would not be tracked in VCS, it is possible to generate it cheaply thanks to Docker staged builds (i.e. a preparatory stage could be used to generate it - it would run fast, so people could copy entire source tree into it).

Edit: Or maybe Cargo.lock could be enriched to contain enough information to run a dependencies build.

That would be the best option from Docker PoV, but I fear it would be controversial for everybody else: it would cause a lot (more) of information from Cargo.toml is duplicated into Cargo.lock (feature flags, build profiles, ...). For projects with VCS-tracked Cargo.lock, people would have to remember check it in after feature flag changes etc (a small change of workflow).

I've been lurking on this thread for quite a while, and might have a helpful tidbit for those looking for a docker workaround. I did a scan and couldn't see anywhere this has been suggested, but the following seems to work for caching deps for me in a docker build:
# Create a fake main file
RUN echo "fn main() {println!(\"if you see this, the build broke\")}" > src/main.rs
# copy over your manifests
COPY ./Cargo.toml ./Cargo.toml

# this build step will cache your dependencies
RUN cargo build --release

COPY ./src ./src
COPY build.rs build.rs

RUN cargo build --release

RUN cp target/release/pepsi /usr/local/cargo/bin/pepsi

Pepsi is just the codename for a project of mine, and build.rs is essentially a single println in a main function. For some reason the build.rs is necessary in order to do an actual cache bust for the binary.
Anyway, not looking to comment either way, just thought I'd leave this here for docker folks looking for a quick workaround!

@trezm I have been playing with the same docker commands but ran into some strange thing where it is not rebuilding the binary. Does it have to do with timestamps of the main.rs file? Why did the build rs fix the problem?

@camerondavison do be 100% honest, I'm really not sure why the build.rs affects the result! I came across it as a solution by accident. I had a project that I have a dockerfile for that uses tonic, so I needed a build.rs to build proto files.
I noticed that I could super simplify the Dockerfile for that project, but when I copied the same Dockerfile to a new project (sans build.rs) it no longer worked because of what you mentioned. I added the build.rs file back with a single dummy println and it seemed to work again!
Really not sure why this would change anything, maybe something to do with the presence of a build.rs file forcing cargo to look at the actual timestamp on all the files? This is me wildly speculating, but I encourage you to try adding a dummy build.rs and see if you get the same results!

@trezm  adding a build.rs file with
fn main() {}

does fix the problem. so crazy. I am going to keep trying to figure out why.

@camerondavison adding to your work-around, you can replace pre-last line with: find target/release/ -type f -executable -maxdepth 1 -delete

hey @trezm doesn't work if there are multiple workspaces. 😢

Bummer! @vn971's solution works? I don't claim to have any substantial knowledge about cargo, I was just hopefully helping with a workaround :)

@anitnilay20 just copy the needed Cargo.* files only, and otherwise repeat the steps? Should work?

@trezm Solution does work for single workspace cargo projects but it doesn't work for multiple workspaces..

Let's better raise a StackOverflow question or something. This ticket is to add support for the flag. I think multiple workspaces work if done correctly, in the right order.

@vn971 it worked.. issue was bad Dockerfile. anyway thanx for your help.

Does anyone have a working version that can cache deps for workspaces too?

@dessalines You can check this https://github.com/sha-el/Vouch/blob/master/Dockerfile

@anitnilay20 Thank you so much, that did it!
It does re-fetch the workspace deps when anything changes in there, but any changes in the main project don't re-fetch the workspace deps!

I have been dealing with the issue over a year now, both in libs, bins, and workspaces, with projects with and without examples and benches, mostly in docker images. Here are my 2 cents:
The underlying consequence is the too high compilation time of slowly-varying code in the context of CI/CD. Specifically, there is a number of rust projects whose majority of their CI/CD time is spent compiling non-changing code, which can be cached (e.g. check logs of arrow/rust or its issues within docker/CI.
The core issue is that, in the context of containerization (e.g. docker), engines have no way of knowing that the resulting artifacts in target/*/deps are independent of src/ after RUN cargo build. As such, they must invalidate all artifacts in target/*/deps when src/ changes (e.g. via a COPY . . before RUN). This is the reason why cargo build is not sufficient: we need a command to

build the stuff in target/*/deps using the dependants of target/*/deps

Cargo supports this locally by not recompiling dependencies that are already compiled. However, it does this because it knows that target/*/deps is independent of src/; a generic technology like containerization has no way of knowing this.
In theory, solving this issue requires two different API endpoints:

a command to get the list of dependants (which we have been assuming to be only Cargo.toml, src/[lib.rs/main.rs] and optionally build.rs)
a command to build target/*/deps

IMO this is issue a sub-part of one of the top 5 requested improvements to rust - compilation time. IMO slow compilation is not only addressed by producing better plans for the LLVM, but also providing the tools to efficiently cache slowly varying code, such as external dependencies.
Finally, why the current status quo and "replacing lib.rs/main.rs for a dummy" is not enough.
non-workspace projects
This approach works well for simple projects, but does not work for projects with examples or benches. Essentially, given a Cargo.toml with a [[bench]]/[[example]], cargo build [--lib] [--release] requires the respective file benches/examples to be there. This implies that we also need to copy the benches and examples (which causes a recompilation on every change of example), or parse the Cargo.toml and touch a file for every [[bench]]/[[example]] on it (high maintenance).
workspace projects
When there are inter-workspace dependencies, say w1 <- w2, we can't compile w2's dependencies because it requires compiling w1 dependencies, like @shepmaster mentioned.
One current "solution" for this is to repeat the step for non-workspace projects on each workspace including examples/benches (cumbersome).
In the end, a command like cargo build --dependencies-only would collect the necessary files required to build targets/*/deps based on Cargo.tomls and would execute the download/compile to achieve that final state, in the same way that the hacks aforementioned currently achieve with different levels of "hackiness".

@jorgecarleitao

The underlying consequence is the too high compilation time of slowly-varying code in the context of CI/CD

Why you can not use sccache for that?
It also automatically handle changes of versions of dependencies, compiler version changes and so on.

@jorgecarleitao

The underlying consequence is the too high compilation time of slowly-varying code in the context of CI/CD

Why you can not use sccache for that?
It also automatically handle changes of versions of dependencies, compiler version changes and so on.

We have:

This is complex, and doesn't seem to be helping IRL, so I'm closing this

@jorgecarleitao
Why you can not use sccache for that?
It also automatically handle changes of versions of dependencies, compiler version changes and so on.

In our case we don’t go for sccache because:

it adds yet another tool to our stack, which comes with a maintenance and knowledge cost.
it isn’t supported natively by our cloud CI (Bitbucket)

Our product consists of multiple components written in different languages. We support this with a small team, so we stick to native tooling and defaults as much as possible.

for those who have the benefit of a buildkit based docker CI process... this is how i've been able to achieve < 1 minute test builds and < 2 minute release builds
For Testing:
# define dependencies for temporary build
COPY src/Cargo.toml src/Cargo.toml
COPY test/Cargo.toml test/Cargo.toml
COPY Cargo.lock Cargo.lock
COPY Cargo.toml Cargo.toml

# dummy files so we can compile and build depenencies
RUN echo "fn main(){}" > src/lib.rs
RUN echo "fn main(){}" > test/lib.rs

# cache dependency compilation
RUN --mount=type=cache,target=/root/.cargo/registry/ RUSTFLAGS="-C target-feature=-crt-static" cargo test --package test

# remove dummy files and compilation cache (not dependency cache)
RUN rm -rf src test
RUN rm -rf target/debug/**/libsrc*
RUN rm -rf target/debug/**/libtest*

# copy actual src over that tends to change more often than dependencies
COPY build/cdn build/cdn
COPY test test
COPY src src

# run another build using our actual source code
RUN RUSTFLAGS="-C target-feature=-crt-static" cargo test --package test

For release/deployment
# dummy files so we can compile and build depenencies
RUN mkdir src && echo "fn main(){}" > src/lib.rs
RUN echo "[workspace]" > Cargo.toml && echo "members = [\"src\"] " >> Cargo.toml

# define dependencies for temporary build
COPY src/Cargo.toml src/Cargo.toml
COPY Cargo.lock src/Cargo.lock

# cache dependency compilation
RUN --mount=type=cache,target=/root/.cargo/registry/ RUSTFLAGS="-C target-feature=-crt-static"  cargo build --release --lib

# remove dummy files and compilation cache (not dependency cache)
RUN rm -rf src
RUN rm -rf target/release/**/libsrc*

# copy actual src over that tends to change more often than dependencies
COPY build/cdn build/cdn
COPY src src

# run another build using our actual source code
RUN RUSTFLAGS="-C target-feature=-crt-static" cargo build --release

FROM release as bin1

WORKDIR /rust

COPY src/cfg src/cfg
COPY --from=release /rust/target/release/bin1 bin1

ENTRYPOINT ["/rust/bin1"]

FROM release as bin2

WORKDIR /rust

COPY src/cfg src/cfg
COPY --from=release /rust/target/release/bin2 bin2

ENTRYPOINT ["/rust/bin2"]

Having lost more of my life than I care to mention trying to get non-trivial incremental compilation working in something docker-like, I have come to the conclusion that only by removing mtime from the picture can we achieve stable caching. I think we need https://github.com/rust-lang/cargo/issues/6529 to be solved before there's any point in having a command that builds just the dependencies (for which we have reasonable workarounds). Currently there's no way to work around the mtime dependency.

I figured out how to get this also working with cargo workspaces, using romac's fork of cargo-build-deps.
This example has my_app, and two workspaces: utils and db.
FROM rust:nightly as rust

# Cache deps
WORKDIR /app
RUN sudo chown -R rust:rust .
RUN USER=root cargo new myapp

# Install cache-deps
RUN cargo install --git https://github.com/romac/cargo-build-deps.git

WORKDIR /app/myapp
RUN mkdir -p db/src/ utils/src/

# Copy the Cargo tomls
COPY myapp/Cargo.toml myapp/Cargo.lock ./
COPY myapp/db/Cargo.toml ./db/
COPY myapp/utils/Cargo.toml ./utils/

# Cache the deps
RUN cargo build-deps

# Copy the src folders
COPY myapp/src ./src/
COPY myapp/db/src ./db/src/
COPY myapp/utils/src/ ./utils/src/

# Build for debug
RUN cargo build

In python there is a file called dependencies.txt, So in your dockerfile you do something like this:
WORKDIR /my-stuff
COPY dev.dependencies.txt dependencies.txt ./

# in dependencies i declared fastapi that install uvicorn
RUN  pip install -r dev.dependencies.txt -r dependencies.txt

COPY . .

CMD ["uvicorn", "app.main:app", "--host", "<IP_ADDRESS>", "--port", "8080", "--reload"]

So you dont have to download all dependencies everytime you changes the code. With Rust Im unable to generate compilation cache and build without creating a dummy project. It would be great something like:
WORKDIR /my-stuff

COPY Cargo.toml Cargo.lock ./
# this runs cargo fetch, cargo install, and finally cargo build
RUN cargo cache build

COPY . .
RUN cargo build

CMD ["cargo", "watch", "-x", "run"]

Then cargo cache can contain clean and build subcommands (or generate, or something that describes the action of building the cache)
Look at https://github.com/rust-lang/cargo/issues/5120

@uselessscat This is why I created #8362 , I believe that's more or less what you mean...

As mentioned in #8362, I believe cargo-chef is a MVP of what this could look like from an API perspective.
A sample Dockerfile looks like this:
FROM rust as planner
WORKDIR app
RUN cargo install cargo-chef
COPY . .
RUN cargo chef prepare  --recipe-path recipe.json

FROM rust as cacher
WORKDIR app
RUN cargo install cargo-chef
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json

FROM rust as builder
WORKDIR app
COPY . .
COPY --from=cacher /app/target target
RUN cargo build --release --bin app

FROM rust as runtime
WORKDIR app
COPY --from=builder /app/target/release/app /usr/local/bin
ENTRYPOINT ["./usr/local/bin/app"]

It does not require Buildkit.
I still believe this functionality should somehow find its way into cargo, but hopefully we can explore the space a bit with cargo-chef outside of it to understand the tradeoffs and actual usage patterns.

@LukeMathWalker this is much cleaner to have the recipe planning inside docker as well... you should make this the default example in the cargo-chef readme.

I thought I did, but it turns I had not push the commit to remote :sweat_smile: Updated! @dessalines

Workaround with empty lib.rs not working perfectly for example for wasm-bindgen. I couldn't correctly separate deps build and lib build. As a result applying this workaround I have empty .d.ts file after wasm-pack package generating.

This issue has been known for 5 years, can you estimate if/when cargo will be able to download and build all deps only without dirty hacks?
Virtually every other modern language has the same problem and solved it somehow (dotnet restore, npm install, pip install -r, bundle install, ...).

I came here looking for the same use-case of wanting to properly layer my Docker images to decrease build times.
I ended up with the following solution, in case this helps anyone else:
FROM ekidd/rust-musl-builder AS builder

RUN sudo chown -R rust:rust /home/rust

RUN mkdir src && touch src/lib.rs
COPY Cargo.lock .
COPY Cargo.toml .
RUN cargo build --release

ADD . .
RUN cargo build --release

FROM scratch
COPY --from=builder \
  /home/rust/src/target/x86_64-unknown-linux-musl/release/my-app \
  /my-app
ENTRYPOINT ["/my-app"]

Here's the result of the first run, vs a second run with only source code changes, no dependency changes:
collapsed wall of text

Absolute life saver, was using cargo init to create a dummy bin, however this week all my containers started failing and the only thing in the logs was "Hello, world!". Switching to a lib fixed everything. Thank you <3
It's a mystery to me why COPY stopped overwriting the dummy project as it used to.

It's a mystery to me why COPY stopped overwriting the dummy project as it used to.

It does not, you just have to RUN touch src/main.rs and cargo will build the new sources (otherwise the timestamp from the file is  older then the last build).

The workaround with empty lib.rs doesn't work properly, for example, with wasm-bindgen. I couldn't correctly separate the deps build and the lib build. As a result of using this workaround I have an empty .d.ts file after wasm-pack package generation.
Docker layers

@zarix908 Building two separate images (eg, wasm-base and binary-base) in a multistage build may work for you. I'm doing this in a project that is using both wasm-bindgen and cargo build and it's caching both builds properly.

This issue has been known for 5 years, can you estimate if/when cargo will be able to download and build all deps only without dirty hacks?
Virtually every other modern language has had the same problem and solved it somehow (dotnet restore, npm install, pip install -r, bundle install, ...).

I really don't understand why there's so much friction with this...

For now, this development set up is working great for me

Dockerfile.dev
FROM rust:1.54-buster
WORKDIR /usr/src/app

# Installing entr for hot reload
RUN apt-get update && apt-get upgrade && apt-get install -y entr

# Creating a new dummy project
RUN cargo init
# Replacing with the desired dependencies
COPY ./Cargo.toml ./Cargo.toml
# Dependencies built and cached, will be reuilt when Cargo.toml is changed
RUN cargo build
# Deleting the dummy project, leaving an environment ready to `run`
RUN rm src/* -rf

docker-compose.dev.yml
version: '2'

services:
  api:
    build:
      context: .
      dockerfile: Dockerfile.dev
    env_file: .env
    ports:
      - 8080:8080
    volumes:
      - ./:/usr/src/app/
    command: sh -c ./run-env.sh

run-env.sh
while sleep 1 ; do
  find ./src/ | entr -r -s 'cargo run';
done

This set up builds all of my dependencies, and waits changes in the src folder to rebuild the code. In theory it should support hot-dependency-reload, but I haven't tested it yet. Please bear in mind that I've worked with rust for about 8 entire hours, so I don't really know what I'm doing :laughing:
I'm sure someone more experienced than me will, eventually, build an extension for cargo to build the dependencies without this much overhead

This already exists, its called cargo-chef, just a few comments up. It works really well and I have all my projects using it.

My desire for --dependencies-only is for use in a Dockerfile. So far, I am happy with using BuildKit's --mount=type=cache feature to cache the target directory instead.

If anyone ever adds this feature, you can reward yourself by answering my Stackoverflow question 😋

I have a big folder full of cool examples with about twenty different Rust crates to help me learn Rust more. Some of these crates don't support certain targets or just don't get along, so being able to build a single binary and have a way to only build the crates it actually needs would be great.


Feature - Terminal - Part 3: Terminal renderer

This picks up the common model / renderer utilities from https://github.com/revery-ui/revery-terminal and picks up to be used for Onivim 2's terminal rendering.
Note that Onivim 2 supports terminals on Windows as well, because we leverage the extension host (node-pty) for the PTY layer.
Progress:

This wires up the $onProcessData from our extension host and pipes it into ReveryTerminal to be handled. Most of the heavy lifting is handled by Revery Terminal for the state management - this mainly plumbs through the $onProcessData into a revery terminal instance (managed by the terminal subscription) - augments the state with ReveryTerminal.Screen.t and ReveryTerminal.Cursor.t, and then hands it off to ReveryTerminal.render to actually draw.
TODO:

[ ] Factor terminal renderer out
[x] Fix revery-terminal compilation on Windows
[x] Validate and test on Windows

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.2 out of 3 committers have signed the CLA.:white_check_mark: glennsl:white_check_mark: bryphe:x: leematosYou have signed the CLA already but the status is still pending? Let us recheck it.