flash_attn ImportError

Please check that this issue hasn't been reported before.

[X] I searched previous Bug Reports didn't find any similar reports.

Expected Behavior
I'm trying to fine-tune the Mistral model using the example script on the free tier of Google Colab.
Current behaviour
Getting the following error related to flash attention
The following values were not passed to `accelerate launch` and had defaults used instead:
	`--num_processes` was set to a value of `1`
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2023-10-25 15:38:42.945005: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-10-25 15:38:42.945068: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-10-25 15:38:42.948363: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-10-25 15:38:45.475489: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/content/axolotl/axolotl/src/axolotl/cli/__init__.py", line 21, in <module>
    from axolotl.common.cli import TrainerCliArgs, load_model_and_tokenizer
  File "/content/axolotl/axolotl/src/axolotl/common/cli.py", line 11, in <module>
    from axolotl.utils.models import load_model, load_tokenizer
  File "/content/axolotl/axolotl/src/axolotl/utils/models.py", line 10, in <module>
    from optimum.bettertransformer import BetterTransformer
  File "/usr/local/lib/python3.10/dist-packages/optimum/bettertransformer/__init__.py", line 14, in <module>
    from .models import BetterTransformerManager
  File "/usr/local/lib/python3.10/dist-packages/optimum/bettertransformer/models/__init__.py", line 17, in <module>
    from .decoder_models import (
  File "/usr/local/lib/python3.10/dist-packages/optimum/bettertransformer/models/decoder_models.py", line 38, in <module>
    from transformers.models.llama.modeling_llama import LlamaAttention
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py", line 45, in <module>
    from flash_attn import flash_attn_func, flash_attn_varlen_func
  File "/usr/local/lib/python3.10/dist-packages/flash_attn/__init__.py", line 3, in <module>
    from flash_attn.flash_attn_interface import (
  File "/usr/local/lib/python3.10/dist-packages/flash_attn/flash_attn_interface.py", line 8, in <module>
    import flash_attn_2_cuda as flash_attn_cuda
ImportError: /usr/local/lib/python3.10/dist-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEi
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 33, in <module>
    sys.exit(load_entry_point('accelerate==0.24.0.dev0', 'console_scripts', 'accelerate')())
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 986, in launch_command
    simple_launcher(args)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 628, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'axolotl.cli.train', '/content/axolotl/examples/mistral/config.yml']' returned non-zero exit status 1.

Steps to reproduce
Run the example script of the Mistral model
!accelerate launch -m axolotl.cli.train /content/axolotl/examples/mistral/config.yml

Config yaml
Same as the one at examples/mistral/config.yml
Possible solution
Saw someone else with an issue with flash attention and the solution was to use Docker but this is not available on Colab.
Which Operating Systems are you using?

[ ] Linux
[ ] macOS
[ ] Windows

Python Version
3.10.12
axolotl branch-commit
main/20aa4b5
Acknowledgements

[X] My issue title is concise, descriptive, and in title casing.
[X] I have searched the existing issues to make sure this bug has not been reported yet.
[X] I am using the latest version of axolotl.
[X] I have provided enough information for the maintainers to reproduce and diagnose the issue.

same

Please check that this issue hasn't been reported before.

[x]  I searched previous Bug Reports didn't find any similar reports.

Expected Behavior
I'm trying to fine-tune the Mistral model using the example script on the free tier of Google Colab.
Current behaviour
Getting the following error related to flash attention
The following values were not passed to `accelerate launch` and had defaults used instead:
	`--num_processes` was set to a value of `1`
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2023-10-25 15:38:42.945005: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-10-25 15:38:42.945068: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-10-25 15:38:42.948363: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-10-25 15:38:45.475489: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/content/axolotl/axolotl/src/axolotl/cli/__init__.py", line 21, in <module>
    from axolotl.common.cli import TrainerCliArgs, load_model_and_tokenizer
  File "/content/axolotl/axolotl/src/axolotl/common/cli.py", line 11, in <module>
    from axolotl.utils.models import load_model, load_tokenizer
  File "/content/axolotl/axolotl/src/axolotl/utils/models.py", line 10, in <module>
    from optimum.bettertransformer import BetterTransformer
  File "/usr/local/lib/python3.10/dist-packages/optimum/bettertransformer/__init__.py", line 14, in <module>
    from .models import BetterTransformerManager
  File "/usr/local/lib/python3.10/dist-packages/optimum/bettertransformer/models/__init__.py", line 17, in <module>
    from .decoder_models import (
  File "/usr/local/lib/python3.10/dist-packages/optimum/bettertransformer/models/decoder_models.py", line 38, in <module>
    from transformers.models.llama.modeling_llama import LlamaAttention
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py", line 45, in <module>
    from flash_attn import flash_attn_func, flash_attn_varlen_func
  File "/usr/local/lib/python3.10/dist-packages/flash_attn/__init__.py", line 3, in <module>
    from flash_attn.flash_attn_interface import (
  File "/usr/local/lib/python3.10/dist-packages/flash_attn/flash_attn_interface.py", line 8, in <module>
    import flash_attn_2_cuda as flash_attn_cuda
ImportError: /usr/local/lib/python3.10/dist-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEi
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 33, in <module>
    sys.exit(load_entry_point('accelerate==0.24.0.dev0', 'console_scripts', 'accelerate')())
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 986, in launch_command
    simple_launcher(args)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 628, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'axolotl.cli.train', '/content/axolotl/examples/mistral/config.yml']' returned non-zero exit status 1.

Steps to reproduce
Run the example script of the Mistral model
!accelerate launch -m axolotl.cli.train /content/axolotl/examples/mistral/config.yml

Config yaml
Same as the one at examples/mistral/config.yml
Possible solution
Saw someone else with an issue with flash attention and the solution was to use Docker but this is not available on Colab.
Which Operating Systems are you using?

[ ]  Linux
[ ]  macOS
[ ]  Windows

Python Version
3.10.12
axolotl branch-commit
main/20aa4b5
Acknowledgements

[x]  My issue title is concise, descriptive, and in title casing.
[x]  I have searched the existing issues to make sure this bug has not been reported yet.
[x]  I am using the latest version of axolotl.
[x]  I have provided enough information for the maintainers to reproduce and diagnose the issue.

flash_attn can only be used on certain versions of the gpu. Check your environment.


Allow tab navigation to be used for external links

I tried using the tab navigation with a mix between panels on the same page as well as external links. External links didn't work because the clicks get intercepted and the code then tries to match the href to a panel name which is then displayed. I changed the code so the panel logic is only used when the href starts with a hash symbol.

What exactly is the use-case here?
I find this to be confusing UX which should be avoided. Tabs are tabs, they dynamically change the content displayed in some fashion. So having tabs then a few external links mixed in that don't do what the rest of the items in the same are do seems very odd.

With "external links" I do not mean links to other sites, but links to a different HTML document on the same site.
I use it to ease integration of legacy code where I need to switch to an entirely different document because I can't load the content right away inside a hidden panel.
So, except for a missing transition, the user won't notice anything different.

External/Internal links, same diff in the case. They are links going to an external page (i.e. not one the user is currently on) which causes a full page refresh.
I find this case annoying UX overall if allowed upstream. Seems like a case where you should maintain your own clone of the component during the transition period if it is needed that badly. I don't like the idea of allowing the core code to allow confusing UX to be implemented so easily.

Well, in my case, the app runs on a local server, so you won't even notice the screen refresh.
IMHO, the fix gives developers more choice over what they want to do without any other side effects. And believe me, there are easier ways to screw up UX ;-)

@garbee not everyone has a fully client-rendered application.  In fact, that is only a small minority.  Using tabs for navigation and having the server render the result should be supported.

@nickretallack Then that isn't a good use-case for tabs, you should be using normal navigation menus.

Tabs are used to switch between "related" content. As @nickretallack says, this does not necessarily have to be client-rendered content.
e.g. Google Search uses tabs to switch between web search, image search, news search, etc ...

Wait, I'm a little confused.  Look at the navigation on getmdl.com -- it's tabs!  And those tabs are links!  So what are we talking about here?
I see that the classes that are actually applied there are mdl-navigation, but all the CSS that makes them look like tabs is custom.  Why not just use mdl-tabs?

Because tabs aren't the proper type of component for that. Yes we need an
active state for navigation in the header. That is something I'm working on
separately.
On Jul 17, 2015 5:47 PM, "Nick Retallack"<EMAIL_ADDRESS>wrote:

Wait, I'm a little confused. Look at the navigation on getmdl.com -- it's
tabs! And those tabs are links! So what are we talking about here?
I see that the classes that are actually applied there are mdl-navigation,
but all the CSS that makes them look like tabs is custom. Why not just use
mdl-tabs?
—
Reply to this email directly or view it on GitHub
https://github.com/google/material-design-lite/pull/929#issuecomment-122427742
.

The header navigation on http://www.getmdl.io/ also supports horizontal scrolling (like tabs). It would be great if MDL navigation in the header could provide that too.

I'm using React and Flux, so I need the Tab implementation to set state via script, not toggle between components on the page.  I'm having to use material-ui for a tab implementation because of this limitation.

@Garbee Is filtering a good use of tabs.  If not, what component would be correct?

Leaving MDL so that you can't use tabs for direct linking isn't going to stop people from using it that way, it's just something they will write a hack around (i.e. intercept the onClick event and e.preventdefault and window.navigate if the href doesen't start with #).
Traditional url navigation from tabs is done all over the web and is not confusing at all to users.
Isn't a big point of MDL to let people use components without needing a framework?  Because this is treading on opinionated framework territory.

For the record, I can definitely see a case for merging tabs and navigation into a single component, and I think this is something the team should discuss further for future BC-breaking versions. We don't have much guidance from the MD spec in this regard, sadly, but we can probably come up with a good approach.
In the meantime, I like this PR in that it doesn't break backwards-compatibility and doesn't really make any promises in making tabs work like nav; it simply makes it easier to adopt @mweimerskirch's use case. I vote for merging, thanks @mweimerskirch!

Hey folks. This is one of the areas of the Material Design spec that unfortunately doesn't yet have strong guidance when it comes to the flexibility around tab behaviour. We're somewhat left to consult the UX team and then make a call based on what we believe makes the most sense for both our users and the project. @Garbee's take is trying to make sure what lands upstream makes sense long-term.
That said, I've seen a few upcoming Google projects that do use tabs in the manner described by @mweimerskirch and personally wouldn't mind seeing this land as long as we consult UX beforehand - I can do this. When it comes to areas like this where we've got obvious strong opinions about what we think makes sense and letting actual designers help us make the call when we're in doubt seems like a good balance to me.
Is that reasonable?

Now, that my Q has been resolved: LGTM :)

LGTM. Requires a rebase against master.

Sorry for the delayed answer, I was on vacation.
I rebased the patch.

@mweimerskirch Thanks a lot. Merging. :)


Setting up coverage reporting via CI / linting?

Are you interested on having coverage reports via CI?

I suggest using codecov for that (with a config that disables its comments, so that only PR/commit statuses are reported).
I also suggest having a tox configuration for this then, so that tox -e py38-coverage could be used on CI, but also locally in the same way.

Are you interested in having some linting via CI / tox also?
For that using flake8 might be good enough, but would require some fixes / running black/autopep8 once.

I'd be happy to contribute whatever you prefer.

Yeah, I plan on setting up coverage reporting soon. Once I have more coverage to report, its currently at 52%
At the moment I'm relying on Mypy, but I'll think about some additional linting.
Thanks for the offer of help, but I can set that up.


ERROR in ./node_modules/sanitize-html/node_modules/postcss/lib/input.js

PLEASE NOTE: make sure the bug exists in the latest patch level of the project. For instance, if you are running a 2.x version of Apostrophe, you should use the latest in that major version to confirm the bug.
To Reproduce
Step by step instructions to reproduce the behavior:
After instaling the next package:
"@types/sanitize-html": "1.27.0",
"sanitize-html": "2.2.0",
And in the Angular component import
import sanitizeHtml from 'sanitize-html';
ngOnInit(): void {
super.ngOnInit();
const html = 'hello world';
console.log('sanitize', sanitizeHtml(html));
}
ERROR in ./node_modules/sanitize-html/node_modules/postcss/lib/input.js
Module not found: Error: Can't resolve 'path' in '\node_modules\sanitize-html\node_modules\postcss\lib'
ERROR in ./node_modules/sanitize-html/node_modules/postcss/lib/previous-map.js
Module not found: Error: Can't resolve 'path' in '\node_modules\sanitize-html\node_modules\postcss\lib'
ERROR in ./node_modules/sanitize-html/node_modules/postcss/lib/map-generator.js
Module not found: Error: Can't resolve 'path' in '\node_modules\sanitize-html\node_modules\postcss\lib'
Describe the bug
v10.20.1
Version of Node.js:
v10.20.1
Server Operating System:
Windows
Additional context:
Chrome
Expected behavior:
Not throw the exception.

This looks to be a duplicate of https://github.com/apostrophecms/sanitize-html/issues/450. Please review that and comment if you have additional information.

This looks to be a duplicate of https://github.com/apostrophecms/sanitize-html/issues/450. Please review that and comment if you have additional information.


Assign requires shapes of both tensors to match.

Hello, I try to train the granite model and get an error with unmatched tensors:
env USE_BAZEL_VERSION=6.4.0  ../bazelisk-linux-amd64   run //gematria/granite/python:run_granite_model -- --gematria_action=train --gematria_checkpoint_dir=/tmp/test_model/ --gematria_training_num_epochs=10  --gematria_input_file=/tmp/basic_blocks_with_throughput.tfrecord  --gematria_tokens_file=/tmp/tokens.txt

(env) ~/gematria$ env USE_BAZEL_VERSION=6.4.0  ../bazelisk-linux-amd64   run //gematria/granite/python:run_granite_model -- --gematria_action=train --gematria_checkpoint_dir=/tmp/test_model/ --gematria_training_num_epochs=10  --gematria_input_file=/tmp/basic_blocks_with_throughput.tfrecord  --gematria_tokens_file=/tmp/tokens.txt
INFO: Analyzed target //gematria/granite/python:run_granite_model (0 packages loaded, 85 targets configured).
INFO: Found 1 target...
Target //gematria/granite/python:run_granite_model up-to-date:
  bazel-bin/gematria/granite/python/run_granite_model
INFO: Elapsed time: 0.168s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/gematria/granite/python/run_granite_model '--gematria_action=train' '--gematria_checkpoint_dir=/tmp/test_model/' '--gematria_training_num_epochs=10' '--gematria_input_file=/tmp/basic_blocks_with_throughput.tfrecord' '--gematria_tokens_file=/tmp/tokens.txt'
2024-06-07 11:42:19.573765: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-06-07 11:42:19.575373: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2024-06-07 11:42:19.603053: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-06-07 11:42:19.603109: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-06-07 11:42:19.604100: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-06-07 11:42:19.609623: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2024-06-07 11:42:19.609868: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-06-07 11:42:20.147141: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
WARNING:tensorflow:From /home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/compat/v2_compat.py:108: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
W0607 11:42:21.881521<PHONE_NUMBER>19904 model_base.py:683] ModelBase._output_tensor has invalid name. Expected ModelBase.output_tensor, found concat/concat:0.
W0607 11:42:22.258888<PHONE_NUMBER>19904 model_base.py:902] ModelBase._synchronous_training is True with a single worker.
I0607 11:42:23.650717<PHONE_NUMBER>19904 timer.py:61] Creating model: TokenGraphBuilderModel: 2.525757s
I0607 11:42:23.650949<PHONE_NUMBER>19904 timer.py:61] Loading basic blocks: 0.000080s
WARNING:tensorflow:From /home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/model/python/main_function.py:545: StopAtStepHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
W0607 11:42:23.651067<PHONE_NUMBER>19904 deprecation.py:50] From /home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/model/python/main_function.py:545: StopAtStepHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
WARNING:tensorflow:From /home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py:579: StepCounterHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
W0607 11:42:23.882867<PHONE_NUMBER>19904 deprecation.py:50] From /home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py:579: StepCounterHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
WARNING:tensorflow:From /home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/basic_session_run_hooks.py:686: SecondOrStepTimer.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
W0607 11:42:23.883046<PHONE_NUMBER>19904 deprecation.py:50] From /home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/basic_session_run_hooks.py:686: SecondOrStepTimer.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
WARNING:tensorflow:From /home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py:586: SummarySaverHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
W0607 11:42:23.883132<PHONE_NUMBER>19904 deprecation.py:50] From /home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py:586: SummarySaverHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
WARNING:tensorflow:From /home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py:597: CheckpointSaverHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
W0607 11:42:23.883212<PHONE_NUMBER>19904 deprecation.py:50] From /home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py:597: CheckpointSaverHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
INFO:tensorflow:Create CheckpointSaverHook.
I0607 11:42:23.883271<PHONE_NUMBER>19904 basic_session_run_hooks.py:557] Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
I0607 11:42:25.717828<PHONE_NUMBER>19904 monitored_session.py:240] Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/test_model/model.ckpt-0
I0607 11:42:25.724106<PHONE_NUMBER>19904 saver.py:1413] Restoring parameters from /tmp/test_model/model.ckpt-0
2024-06-07 11:42:25.754281: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:388] MLIR V1 optimization pass is not enabled
Traceback (most recent call last):
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/client/session.py", line 1402, in _do_call
    return fn(*args)
           ^^^^^^^^^
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/client/session.py", line 1385, in _run_fn
    return self._call_tf_sessionrun(options, feed_dict, fetch_list,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/client/session.py", line 1478, in _call_tf_sessionrun
    return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [53,128] rhs shape= [9,128]
         [[{{node save/Assign_12}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/saver.py", line 1418, in restore
    sess.run(self.saver_def.restore_op_name,
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/client/session.py", line 972, in run
    result = self._run(None, fetches, feed_dict, options_ptr,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/client/session.py", line 1215, in _run
    results = self._do_run(handle, final_targets, final_fetches,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/client/session.py", line 1395, in _do_run
    return self._do_call(_run_fn, feeds, fetches, targets, options,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/client/session.py", line 1421, in _do_call
    raise type(e)(node_def, op, message)  # pylint: disable=no-value-for-parameter
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:

Detected at node 'save/Assign_12' defined at (most recent call last):
    File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/granite/python/run_granite_model.py", line 109, in <module>
    File "/home/hrong1/gematria/env/lib/python3.11/site-packages/absl/app.py", line 308, in run
    File "/home/hrong1/gematria/env/lib/python3.11/site-packages/absl/app.py", line 254, in _run_main
    File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/granite/python/run_granite_model.py", line 48, in main
    File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/model/python/main_function.py", line 893, in run_gematria_model_from_command_line_flags
    File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/model/python/main_function.py", line 565, in _monitored_training_session_from_flags
Node: 'save/Assign_12'
Assign requires shapes of both tensors to match. lhs shape= [53,128] rhs shape= [9,128]
         [[{{node save/Assign_12}}]]

Original stack trace for 'save/Assign_12':
  File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/granite/python/run_granite_model.py", line 109, in <module>
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/absl/app.py", line 308, in run
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/absl/app.py", line 254, in _run_main
  File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/granite/python/run_granite_model.py", line 48, in main
  File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/model/python/main_function.py", line 893, in run_gematria_model_from_command_line_flags
  File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/model/python/main_function.py", line 565, in _monitored_training_session_from_flags
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/saver.py", line 934, in __init__
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/saver.py", line 946, in build
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/saver.py", line 974, in _build
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/saver.py", line 543, in _build_internal
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/saver.py", line 383, in _AddRestoreOps
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/saving/saveable_object_util.py", line 86, in restore
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/ops/state_ops.py", line 353, in assign
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/framework/op_def_library.py", line 796, in _apply_op_helper
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/framework/ops.py", line 2652, in _create_op_internal
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/framework/ops.py", line 1160, in from_node_def

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/granite/python/run_granite_model.py", line 109, in <module>
    app.run(main)
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
             ^^^^^^^^^^
  File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/granite/python/run_granite_model.py", line 48, in main
    main_function.run_gematria_model_from_command_line_flags(
  File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/model/python/main_function.py", line 893, in run_gematria_model_from_command_line_flags
    session = _monitored_training_session_from_flags(model, is_chief)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/model/python/main_function.py", line 571, in _monitored_training_session_from_flags
    return tf.train.MonitoredTrainingSession(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py", line 606, in MonitoredTrainingSession
    return MonitoredSession(
           ^^^^^^^^^^^^^^^^^
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py", line 1050, in __init__
    super(MonitoredSession, self).__init__(
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py", line 753, in __init__
    self._sess = _RecoverableSession(self._coordinated_creator)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py", line 1259, in __init__
    _WrappedSession.__init__(self, self._create_session())
                                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py", line 1264, in _create_session
    return self._sess_creator.create_session()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py", line 906, in create_session
    self.tf_sess = self._session_creator.create_session()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py", line 665, in create_session
    return self._get_session_manager().prepare_session(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/session_manager.py", line 320, in prepare_session
    sess, is_loaded_from_checkpoint = self._restore_checkpoint(
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/session_manager.py", line 254, in _restore_checkpoint
    _restore_checkpoint_and_maybe_run_saved_model_initializers(
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/session_manager.py", line 71, in _restore_checkpoint_and_maybe_run_saved_model_initializers
    saver.restore(sess, path)
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/saver.py", line 1454, in restore
    raise _wrap_restore_error_with_msg(
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Graph execution error:

Detected at node 'save/Assign_12' defined at (most recent call last):
    File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/granite/python/run_granite_model.py", line 109, in <module>
    File "/home/hrong1/gematria/env/lib/python3.11/site-packages/absl/app.py", line 308, in run
    File "/home/hrong1/gematria/env/lib/python3.11/site-packages/absl/app.py", line 254, in _run_main
    File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/granite/python/run_granite_model.py", line 48, in main
    File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/model/python/main_function.py", line 893, in run_gematria_model_from_command_line_flags
    File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/model/python/main_function.py", line 565, in _monitored_training_session_from_flags
Node: 'save/Assign_12'
Assign requires shapes of both tensors to match. lhs shape= [53,128] rhs shape= [9,128]
         [[{{node save/Assign_12}}]]

Original stack trace for 'save/Assign_12':
  File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/granite/python/run_granite_model.py", line 109, in <module>
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/absl/app.py", line 308, in run
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/absl/app.py", line 254, in _run_main
  File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/granite/python/run_granite_model.py", line 48, in main
  File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/model/python/main_function.py", line 893, in run_gematria_model_from_command_line_flags
  File "/home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/model/python/main_function.py", line 565, in _monitored_training_session_from_flags
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/saver.py", line 934, in __init__
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/saver.py", line 946, in build
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/saver.py", line 974, in _build
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/saver.py", line 543, in _build_internal
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/saver.py", line 383, in _AddRestoreOps
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/saving/saveable_object_util.py", line 86, in restore
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/ops/state_ops.py", line 353, in assign
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/framework/op_def_library.py", line 796, in _apply_op_helper
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/framework/ops.py", line 2652, in _create_op_internal
  File "/home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/framework/ops.py", line 1160, in from_node_def

The input tfrecord is converted from gematria/testing/testdata/basic_blocks_with_throughput.pbtxt.
Any suggestion?
Or do you have any working example for me to try instead? I just want to see how to train the model at this stage.
Thanks,
Hongbo

Update: this issue looks like related with checkpointing. If I remove the --gematria_checkpoint_dir=/tmp/test_model/, the command seems to work, despite some info of unexpected tokens:
(env)~/gematria$ env USE_BAZEL_VERSION=6.4.0  ../bazelisk-linux-amd64   run //gematria/granite/python:run_granite_model -- --gematria_action=train  --gematria_training_num_epochs=10  --gematria_input_file=/tmp/basic_blocks_with_throughput.tfrecord  --gematria_tokens_file=/tmp/tokens.txt
INFO: Analyzed target //gematria/granite/python:run_granite_model (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //gematria/granite/python:run_granite_model up-to-date:
  bazel-bin/gematria/granite/python/run_granite_model
INFO: Elapsed time: 0.104s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/gematria/granite/python/run_granite_model '--gematria_action=train' '--gematria_training_num_epochs=10' '--gematria_input_file=/tmp/basic_blocks_with_throughput.tfrecord' '--gematria_tokens_file=/tmp/tokens.txt'
2024-06-07 12:12:07.770863: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-06-07 12:12:07.772526: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2024-06-07 12:12:07.795890: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-06-07 12:12:07.795951: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-06-07 12:12:07.796874: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-06-07 12:12:07.800792: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2024-06-07 12:12:07.800997: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-06-07 12:12:08.327666: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
WARNING:tensorflow:From /home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/compat/v2_compat.py:108: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
W0607 12:12:09.972967<PHONE_NUMBER>96640 model_base.py:683] ModelBase._output_tensor has invalid name. Expected ModelBase.output_tensor, found concat/concat:0.
W0607 12:12:10.352898<PHONE_NUMBER>96640 model_base.py:902] ModelBase._synchronous_training is True with a single worker.
I0607 12:12:11.604943<PHONE_NUMBER>96640 timer.py:61] Creating model: TokenGraphBuilderModel: 2.383304s
I0607 12:12:11.605142<PHONE_NUMBER>96640 timer.py:61] Loading basic blocks: 0.000071s
WARNING:tensorflow:From /home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/model/python/main_function.py:545: StopAtStepHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
W0607 12:12:11.605249<PHONE_NUMBER>96640 deprecation.py:50] From /home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/model/python/main_function.py:545: StopAtStepHook.__init__ (from tensorflow.python.training.basic_session_run_hooks) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
INFO:tensorflow:Graph was finalized.
I0607 12:12:11.872429<PHONE_NUMBER>96640 monitored_session.py:240] Graph was finalized.
2024-06-07 12:12:11.911880: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:388] MLIR V1 optimization pass is not enabled
INFO:tensorflow:Running local_init_op.
I0607 12:12:12.142276<PHONE_NUMBER>96640 session_manager.py:545] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0607 12:12:12.167031<PHONE_NUMBER>96640 session_manager.py:548] Done running local_init_op.
I0607 12:12:12.242119<PHONE_NUMBER>96640 timer.py:61] Create training session: 0.636888s
WARNING:tensorflow:From /home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/io/python/tfrecord.py:52: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
`tf.data.TFRecordDataset(path)`
W0607 12:12:12.242321<PHONE_NUMBER>96640 deprecation.py:50] From /home/hrong1/.cache/bazel/_bazel_hrong1/32246067180bfaeac7e17e4449bcdc84/execroot/com_google_gematria/bazel-out/k8-fastbuild/bin/gematria/granite/python/run_granite_model.runfiles/com_google_gematria/gematria/io/python/tfrecord.py:52: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
`tf.data.TFRecordDataset(path)`
I0607 12:12:12.243422<PHONE_NUMBER>96640 utils.py:76] Processed 0 items, kept 0
Unexpected node token: RAX
Unexpected node token: RAX
Unexpected node token: RAX
Unexpected node token: RAX
Unexpected node token: RAX
Unexpected node token: RAX
Unexpected node token: RAX
Unexpected node token: RAX
Unexpected node token: RAX
Unexpected node token: RAX
Unexpected node token: LEA
Unexpected node token: LEA
Unexpected node token: RAX
Unexpected node token: RAX
Unexpected node token: LEA
Unexpected node token: RAX
Unexpected node token: LEA
Unexpected node token: LEA
Unexpected node token: RAX
Unexpected node token: RAX
Unexpected node token: RAX
Unexpected node token: LEA
Unexpected node token: LEA
Unexpected node token: RAX
Unexpected node token: LEA
Unexpected node token: RAX
Unexpected node token: LEA
Unexpected node token: LEA
Unexpected node token: LEA
Unexpected node token: LEA
Unexpected node token: LEA
Unexpected node token: LEA
Unexpected node token: RAX
Unexpected node token: RAX
Unexpected node token: RAX
I0607 12:12:12.255485<PHONE_NUMBER>96640 model_base.py:1290] ModelBase.schedule_batch: 34 blocks, 119 instructions
I0607 12:12:12.256108<PHONE_NUMBER>96640 timer.py:61] ModelBase.schedule_batch: 0.001832s
WARNING:tensorflow:From /home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py:1455: SessionRunArgs.__new__ (from tensorflow.python.training.session_run_hook) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
W0607 12:12:13.484528<PHONE_NUMBER>96640 deprecation.py:50] From /home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py:1455: SessionRunArgs.__new__ (from tensorflow.python.training.session_run_hook) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
WARNING:tensorflow:From /home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py:1454: SessionRunContext.__init__ (from tensorflow.python.training.session_run_hook) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
W0607 12:12:13.484722<PHONE_NUMBER>96640 deprecation.py:50] From /home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py:1454: SessionRunContext.__init__ (from tensorflow.python.training.session_run_hook) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
WARNING:tensorflow:From /home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py:1474: SessionRunValues.__new__ (from tensorflow.python.training.session_run_hook) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
W0607 12:12:14.555357<PHONE_NUMBER>96640 deprecation.py:50] From /home/hrong1/gematria/env/lib/python3.11/site-packages/tensorflow/python/training/monitored_session.py:1474: SessionRunValues.__new__ (from tensorflow.python.training.session_run_hook) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras instead.
I0607 12:12:14.556063<PHONE_NUMBER>96640 timer.py:61] ModelBase.train_batch: 2.299840s
I0607 12:12:14.556242<PHONE_NUMBER>96640 model_base.py:1540] Training: epoch: 1, loss: 14.202722549438477
absolute mse: [14.202723], 50%: [2.1514356], 90%: [7.161519], 95%: [7.712915], 99%: [9.870672]

Confirmed: the issue was probably that /tmp/test_model/ contains some checkpoints from previous runs of some other model. Solution to remove all files in that directory. Also add RAX and LEA to the tokens file. Then the training seems to run well.

You should probably be generating a tokens file using a script rather than trying to do it manually.
There's a script in https://github.com/google/gematria/pull/102, but it needs some work before landing, but does work currently.

Agree. That is the way to go. Thanks, Aiden!
From: Aiden Grossman @.>
Sent: Friday, June 7, 2024 4:11 PM
To: google/gematria @.>
Cc: Rong, Hongbo @.>; State change @.>
Subject: Re: [google/gematria] Assign requires shapes of both tensors to match. (Issue #120)
You should probably be generating a tokens file using a script rather than trying to do it manually.
There's a script in #102https://github.com/google/gematria/pull/102, but it needs some work before landing, but does work currently.
—
Reply to this email directly, view it on GitHubhttps://github.com/google/gematria/issues/120#issuecomment-2155679317, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACIMXLOFCVRGLLT6SKDEHB3ZGI4YHAVCNFSM6AAAAABI7E6WYOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJVGY3TSMZRG4.
You are receiving this because you modified the open/close state.Message ID<EMAIL_ADDRESS>

BTW: since an ISA does not change, it might be better if token files for ISAs of X86, etc. are checked in, so people do not have to generate the token files themselves. If they define new tokens, they can generate files for the new tokens and merge with the predefined token files.

I'm glad this got resolved.
As for the token list that is not there: this is intentional, based on several years of experience working on these models. The token list is a property of a dataset (resp. trained model). We used to have the list checked in, and it was a regular source of maintenance pain and sorrow, so we decided to make it part of the data set.
There are a few reasons to not have a canonical token list:

even though ISAs seem very stable, new tokens are added every now and then (whenever some new ISA extension is introduced). Adding new tokens would break older models, because the token counts/embedding table sizes would not match. It could also lead to changing token indices, which again breaks older models. With the token list extracted from the data set/baked into a model, old trained models remain valid as long as the format of inputs and outputs is preserved.
if a certain token never appears in a data set, this token still has an embedding vector. What's worse, this embedding vector is never updated during training and contains the initial random value (so, when the token appears during inference, the results would be garbage). Since we don't have a way to track which tokens were used during the training, there's no way to tell during the inference that we're computing from random data.

It might look like extracting the list of tokens requires more work/takes more time, but it did save me a lot of gray hair :) And as an added bonus, this provides yet another layer of separation between architectural details and training code (i.e. you can train and run inference for any architecture, as long as you can generate the protos).

I see. That is reasonable. Then it is definitely better to keep the tokens part of the model/dataset than part of ISA.
Hongbo


[Fixed in GH-6052] Safari Firefox: Missing Visual Focus on My Account: Add New/Edit Address

Chrome: ✔ (note: submit button not active when form empty - done in a11y epic)
Firefox: ✔ (note: focus border is huge)
Safari: ❓

Chrome: OK
Missing for:

edit & delete buttons on Firefox
Add new address on safari
Back to address list & Update Address & Add Address buttons on Safari
Set as default checkbox on Safari

Tested on dev3.

Chrome ✅
Firefox ✅
Safari ✅

Only issue is pressing ENTER on the Edit button does not redirect user to edit address form page.


Fix new member added while offline doesn't show in approval flow list

Explanation of Change

Fixed Issues

$ https://github.com/Expensify/App/issues/53931
PROPOSAL: https://github.com/Expensify/App/issues/53931#issuecomment-2536068813
Tests

Same as QA Steps

[x] Verify that no errors appear in the JS console

Offline tests

Same as QA Steps
QA Steps

Prerequisite: workflow is enabled and upgraded to Collect

Go offline
Open the workspace setting
Add a new member
Open Workflows page
Press Add approval workflow
Verify the member added in step 3 is shown on the list
Proceed to the next step
Verify the member added in step 3 is shown on the list

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I used JaimeGPT to get English > Spanish translation. I then posted it in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.ts or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] I added unit tests for any new feature or bug fix in this PR to help automatically prevent regressions in this user flow.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native

https://github.com/user-attachments/assets/8b48bba0-1adb-4a3f-aefe-2e3cf09898b7

Android: mWeb Chrome

https://github.com/user-attachments/assets/99c28cc1-68e8-415b-8209-696a45e6629b

iOS: Native

https://github.com/user-attachments/assets/855f9c82-9287-48e4-b143-2279936bc21d

iOS: mWeb Safari

https://github.com/user-attachments/assets/a16dbe08-4963-464d-b583-3f164cc7b22a

MacOS: Chrome / Safari

https://github.com/user-attachments/assets/bdbface2-bfec-4e15-81c5-f5822b5d54c6

MacOS: Desktop

https://github.com/user-attachments/assets/e7334713-e944-42fc-9daa-e3938f07e758

BugZero Checklist:

[ ] [Contributor] Classify the bug:

Bug classification

Source of bug:

[ ] 1a. Result of the original design (eg. a case wasn't considered)
[ ] 1b. Mistake during implementation
[ ] 1c. Backend bug
[ ] 1z. Other:

Where bug was reported:

[ ] 2a. Reported on production (eg. bug slipped through the normal regression and PR testing process on staging)
[ ] 2b. Reported on staging (eg. found during regression or PR testing)
[ ] 2d. Reported on a PR
[ ] 2z. Other:

Who reported the bug:

[ ] 3a. Expensify user
[ ] 3b. Expensify employee
[ ] 3c. Contributor
[ ] 3d. QA
[ ] 3z. Other:

[ ] [Contributor] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake.
Link to comment:

[ ] [Contributor] If the regression was CRITICAL (e.g. interrupts a core flow) A discussion in #expensify-open-source has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner.
Link to discussion:

[ ] [Contributor] If it was decided to create a regression test for the bug, please propose the regression test steps using the template below to ensure the same bug will not reach production again.

Regression Test Proposal Template

[ ] [BugZero Assignee] Create a GH issue for creating/updating the regression test once above steps have been agreed upon.
Link to issue:

Regression Test Proposal
Precondition:

Test:

Do we agree 👍 or 👎

Reviewer Checklist

[ ] I have verified the author checklist is complete (all boxes are checked off).
[ ] I verified the correct issue is linked in the ### Fixed Issues section above
[ ] I verified testing steps are clear and they cover the changes made in this PR

[ ] I verified the steps for local testing are in the Tests section
[ ] I verified the steps for Staging and/or Production testing are in the QA steps section
[ ] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[ ] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[ ] I checked that screenshots or videos are included for tests on all platforms
[ ] I included screenshots or videos for tests on all platforms
[ ] I verified tests pass on all platforms & I tested again on:

[ ] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

[ ] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[ ] I verified proper code patterns were followed (see Reviewing the code)

[ ] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[ ] I verified that comments were added to code that is not self explanatory
[ ] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[ ] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
[ ] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[ ] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[ ] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[ ] I verified the JSDocs style guidelines (in STYLE.md) were followed

[ ] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[ ] I verified that this PR follows the guidelines as stated in the Review Guidelines
[ ] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[ ] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[ ] I verified any variables that can be defined as constants (ie. in CONST.ts or at the top of the file that uses the constant) are defined as such
[ ] If a new component is created I verified that:

[ ] A similar component doesn't exist in the codebase
[ ] All props are defined accurately and each prop has a /** comment above it */
[ ] The file is named correctly
[ ] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[ ] The only data being stored in the state is data necessary for rendering and nothing else
[ ] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[ ] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[ ] All JSX used for rendering exists in the render method
[ ] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[ ] If any new file was added I verified that:

[ ] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[ ] If a new CSS style is added I verified that:

[ ] A similar style doesn't already exist
[ ] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG)

[ ] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[ ] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[ ] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[ ] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[ ] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[ ] I verified that all the inputs inside a form are aligned with each other.
[ ] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[ ] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[ ] For any bug fix or new feature in this PR, I verified that sufficient unit tests are included to prevent regressions in this flow.
[ ] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[ ] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari

MacOS: Desktop

I'm getting this at step 7 if I add a completely new user:

I guess it's unrelated to your PR though.

Yes, it's unrelated since both email and submitsTo are a string.

@bernhardoj Hmm it happens consistently for me on both the Chrome and Desktop app. I agree it's not within scope of this issue though since it's a bug that was just hidden because a member added offline wouldn't be showing up.

@bernhardoj Yeah it doesn't sound like it fits straightforwardly into this issue.
@AndrewGable Do you have any thoughts on how to proceed? I think ideally we fix the issue first before merging this PR. Maybe if @bernhardoj has a fix in mind we can double the price of this issue and fix it one go?

Friendly bump @AndrewGable :bow:
I think adding a new member whilst offline and adding a completely new user you've never chatted with before is a fairly niche combo, so I vote for fixing this issue as a follow-up.

Follow up sounds reasonable, thank you!

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.


Add gfycat sample

https://www.ampproject.org/docs/reference/components/amp-gfycat

Fixed by #463


Updated Mobile System Requirements

We need to change the system requirements for native mobile apps built with Mendix to reflect our insights from ongoing projects and advise customers to appropriate devices.

Looks good to me. These updates do the following, and were made in conjunction with Jonathan, Koert, and Mendix legal:

Change which iOS/Android versions we support
Establish that there are certain mobile OS versions we can take support tickets for
Establish recommended mobile hardware


Fixed premultiplied alpha

Modified the shader and the GL setting to deal with premultiplied alpha.

@doppioslash thanks! It would've taken ages for me to figure this

My pleasure :)


[LookupTables] Adds new JSON file import CLI command

to: @ryandeivert @chunyong-lin @blakemotl @fusionrace
cc: @airbnb/streamalert-maintainers
Background
Per request from @fusionrace, LookupTables can now import data from a JSON file.
Changes

Adds a new lookup tables CLI command: lookup-tables set-from-json-file
Adds two options to list-add: --sort and --unique
Refactors some magic code into its own class with disclaimer

How to use
There is now a subcommand under the lookup-tables command, called set-from-json-file
$ ./manage.py lookup-tables --help
usage: ./manage.py lookup-tables [-h]
                                 {describe-tables,get,set,list-add,set-from-json-file}
                                 ...

Describe and manage your LookupTables

positional arguments:
  {describe-tables,get,set,list-add,set-from-json-file}

optional arguments:
  -h, --help            show this help message and exit

Available Sub-Commands:

	describe-tables               Show information about all currently configured LookupTables
	get                           Retrieve a key from an existing LookupTable
	set                           Update the key of an existing LookupTable
	list-add                      Add a value to a list-typed LookupTables key
	set-from-json-file            NotImplemented

Examples:

    manage.py lookup-tables [describe-tables|get|set]

set-from-json-file takes 3 arguments: --table, --key, and --file.
usage: ./manage.py lookup-tables set-from-json-file [-h] -t TABLE -k KEY -f
                                                    FILE

Pushes the contents of a given json file into the LookupTable key

optional arguments:
  -h, --help            show this help message and exit
  -t TABLE, --table TABLE
                        Name of the LookupTable
  -k KEY, --key KEY     Key to modify on the LookupTable
  -f FILE, --file FILE  Path to the json file, relative to the current working
                        directory

Examples:

    manage.py lookup-tables set-from-json-file -t [table] -k [key] -f [path/to/file.json]

It takes the entire JSON blob in the given .json file and puts it into the LookupTable key as a dict/list.

Coverage decreased (-0.2%) to 96.201% when pulling ea6eeec223dcb351cb7c963c59b4001ccb7c1721 on dw--lookuptables-cli into 4e69f0f0427b577103d29fd0e122c8d8f0403ef3 on release-3-0-0.

@Ryxias we try to abide by the Google Python Style guide. example of the usage here: http://google.github.io/styleguide/pyguide.html#doc-function-raises


Transparent mode on Windows Server 2022 (AWS)

Problem Description
It seems like transparent mode works well on most Windows machines, but I have encountered an issue where it does not work at all with Windows Server 2022.
The tool just shows that it's listening but the log stays empty, and in that state I get time outs when trying to browse any URL that should be intercepted (even for mitm.it)

In addition, using Wireshark I can see there isn't any traffic going over the local API port 8085 after it's initial writing of the PID (at https://github.com/mitmproxy/mitmproxy/blob/8cf0cca70240b232b15c8b972c2d8fa32d73f964/mitmproxy/platform/windows.py#L69)
After some debugging it looks like the packets are modified and re-injected with pydivert, but then the proxy does not receive them for some reason.
Currently I'm trying to debug the issue further and get some more information about it, I'd love if anyone can shed more light on this.
Steps to reproduce the behavior:

Set up a Windows Server 2022 Datacenter on AWS
Install mitmproxy
Set up transparent mode according to the instructions here
Start mitmproxy --mode transparent

System Information
"mitmproxy --version" output:
Mitmproxy: 10.3.0 binary
Python:    3.12.3
OpenSSL:   OpenSSL 3.2.1 30 Jan 2024
Platform:  Windows-2022Server-10.0.20348-SP0
Some info about the machine:
OS Name:                   Microsoft Windows Server 2022 Datacenter
OS Version:                10.0.20348 N/A Build 20348
It's also worth noting that this machine is set up as a Domain Controller and has the server roles: "AD DS" and "DNS" (although it does not seem like a DNS issue)

Have you tried running local redirect mode?
https://mitmproxy.org/posts/local-redirect/windows/

I was not aware of that mode, but it seems to be working well for my case! It also seems to be capturing all traffic, as opposed to transparent mode which only captured TLS and web traffic.
Would be nice to add it to the "Modes of operation" page in the docs.
Is this mode planned to replace the current transparent mode?
Also thanks for the heads up!


The JS/TS language service immediately crashed 5 times. The service will not be restarted.

Type: Bug
I have restarted the TS Server, deleted the repo and recloned, deleted VSCode and reinstalled, restarted my machine, etc.
TS version: 5.4.5 sdk
node: 20.13.1
vscode: 1.89.1
VS Code version: Code 1.89.1 (Universal) (dc96b837cf6bb4af9cd736aa3af08cf8279f7685, 2024-05-07T05:14:24.611Z)
OS version: Darwin arm64 23.5.0
Modes:

System Info

Item
Value

CPUs
Apple M1 Max (10 x 24)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: enabled_ondirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_graphite: disabled_offvideo_decode: enabledvideo_encode: enabledwebgl: enabledwebgl2: enabledwebgpu: enabled

Load (avg)
4, 10, 25

Memory (System)
32.00GB (5.39GB free)

Process Argv
--crash-reporter-id ca72c7b2-ea2a-4e6a-b6e7-a3cadca3bde9

Screen Reader
no

VM
0%

Extensions (1)

Extension
Author (truncated)
Version

vscode-jest
Ort
6.2.5

A/B Experiments
vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscod805cf:30301675
binariesv615:30325510
vsaa593cf:30376535
py29gd2263:31024239
c4g48928:30535728
azure-dev_surveyone:30548225
2i9eh265:30646982
962ge761:30959799
pythongtdpath:30769146
welcomedialog:30910333
pythonidxpt:30866567
pythonnoceb:30805159
asynctok:30898717
pythontestfixt:30902429
pythonregdiag2:30936856
pythonmypyd1:30879173
pythoncet0:30885854
h48ei257:31000450
pythontbext0:30879054
accentitlementsc:30995553
dsvsc016:30899300
dsvsc017:30899301
dsvsc018:30899302
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
chatpanelc:31048052
dsvsc021:30996838
g316j359:31013175
pythoncenvptcf:31049071
a69g1124:31046351
pythonprt:31047983
dwnewjupyter:31046869
26j00206:31048877

5.4.5 sdk

This is a custom TS version. I believe it is patched by yarn. Please start by filing an issue against them

@matt-snow-deepsee I have the same issue!!! Gone through the same steps.

If anyone else is experiencing something similar, we finally found the solution that worked for us.  it's an experimental vscode setting called "typescript.tsserver.experimental.useVsCodeWatcher" that somehow got turned on by default, but didn't show up in the settings.json file.  set it to false and reload your window. Hope this helps, and good luck.


Update VMs for June Patch Tuesday

/azp run

Full rebuild running: https://dev.azure.com/vcpkg/public/_build/results?buildId=73749&view=results

Reran another full rebuild due to the opengl crash on arm64: https://dev.azure.com/vcpkg/public/_build/results?buildId=73949&view=results

/azp run

Seems like our pipeline stucked?

Seems like our pipeline stucked?

Looks like it did the correct thing to me?


Correct met level, wrong actual level

I just transferred a few pokemon from Red to LeafGreen and I noticed that they had the correct met level, but the wrong actual level. The only pokemon that had its correct level was the first one (a lv 25 Kangaskhan). Every other one was either level 15 or 18, not including a stabilized Mew I also had. I'm typing this on my phone so give me a minute to send the save file. The nicknames in the pictures sent are supposed to be backwards so ignore them.

On an unrelated note, is sending glitch pokemon or pokemon above lv 100 supported because I have some in a few boxes.

The save should be on the correct box (361-390 [Box 13]). Ignore the other boxes. I looked for a moment on pkhex but I wouldn't even begin to know what I'm looking for to figure out why this happened.
POKEMON LEAF [BPGE01].zip
I have no way to back up regular game boy games so I wouldn't be able to show you how they originally were.

That's... really interesting. The EXP data is right in the middle of the data structure, so there shouldn't be any issue of dropped data. I'll take a look into what could've gone wrong- it's also really weird that it's specifically level 15 or 18, and that Kangaskhan was correct.
I'll keep you updated- so sorry that you had issues! If you'd like me to, I can send you back a fixed version of the save file with the correct levels!

If it means anything, I was using a 3rd party cord and attempted to use the transfer 3 times on an original game boy before I switched to sp and it worked.
It shouldn't, considering every time it failed nothing was corrupted.
As for correcting the levels, I think it's funny having them at that level. If I really wanted to I could just boost their levels in pkhex, unless that messes with transferring.

Not gonna lie I rushed the transfer. I wasn't paying attention because I was happy that I actually got the transfer to work in general and also my sp was on red (I had a second one charging). I can make another transfer to see if it happens again, one with a mew and one without, just to see if that may be a thing.

 I ran my test save file through a transfer and all the Pokémon were level 15 or 18- clearly something got messed up on my end with the code... whoops! I'm really glad you caught that haha

Adding onto this issue. When I tried transferring from my copy of Crystal to Emerald (both English copies), all but the first slot of my party was either level 15 and 18 with their met levels being as they should've. However I've also observed in PKHeX when trying to correct them that they did not even have games of origin listed, somehow. This is strange, because recently I've done another transfer with my copy of Blue and the end result was perfectly normal, and it even had it's "game of origin"? (Third image, the Mankey)

OH! I just noticed something else that I'm unsure of. Somehow, most of the other Pokemon I've transferred were listed as being "from" Heartgold in PKHeX. Is this supposed to be correct?

Thanks for the extra info! It definitely seems like the conversion data is getting messed up somewhere- I'll take a look at it later today and hopefully have a solution soon!

It seems that the incorrect level bug is still present, somehow?? I just did four test transfers (same games, English Emerald and English Crystal) after updating my cartridge to the latest version of Poke Transporter GB and every single time it's ended up with the same result, first Pokemon in the PC being at a normal level and everything else at level 15/18, along with the other issue of no game of origin when checking in PKHeX (Was that addressed in the latest version? I don't know). Strange how the bug continues to persist even after fixing it, I'm almost certain I've updated my hardware correctly to the latest version. Is this just a me thing on the update, or is this still happening for others?

https://github.com/user-attachments/assets/1994b999-7935-46ae-99a3-ca02d01e97c7
https://github.com/user-attachments/assets/80d17749-8889-4413-acd8-466dbc974a04

I'm so sorry that this is still an issue- I definitely ran some tests on my end, but clearly that wasn't enough 😅. Do you happen to still have your save file that the issue was happening with? I'd love to be able to use exactly what you have in order to -for sure this time- fix this bug

Yes of course! Would you prefer before or after the transfer?

Yes, having the save file before the transfer as well would be perfect!!

Alright! Let me get that for you real quick. This one's an auto-backup from a few days ago when I last touched the file.
Before Transfer.zip

Awesome- Do you happen to have the GameBoy save file before the transfer as well? Sorry for all the requests haha 😅

Yup! Don't worry about the requests by the way, It's just part of the process of figuring this out.
Crystal Transfer.zip

Perfect!! This is exactly what I need to truly squash this bug- thank you so much!!

Alrighty- good news, good news, and bad news. Good news is that I figured out what the bug is and what's causing it. Bad news is that it's something that's pretty baked in, and I'm going to need to rewrite the conversion process in order to fix it. Good news about that though is that it'll end up being a lot more optimized and is really something I've been meaning to do for a while... it'll just take a bit to fully fix.

Good news! That was a bit easier to fix than expected! The most recent release should work now.
Also! The missing met game is just because PKHeX doesn't like having a Gen 3 Pokemon with a Gen 4 game as the met location- it'll be fine once it gets sent to Gen 4! Figured that out while debugging today 🙃


Manage attribute length.

It will not be used, in particular after dropping support of data types char and varchar


Merge Conflict incorrectly removing conflict markers

Does this issue occur when all extensions are disabled?: Yes/No

Version: 1.62.1 (user setup)
Commit: f4af3cbf5a99787542e2a30fe1fd37cd644cc31f
Date: 2021-11-05T10:57:55.946Z
Electron: 13.5.2
Chrome: 91.0.4472.164
Node.js: 14.16.0
V8: 9.1.269.39-electron.0
OS: Windows_NT x64 10.0.18363
I believe this is likely the same issue described by #131067, but I can reliably trigger it. I can't use my own code (proprietary) to show this, but I hacked together an example using the vscode repo. (Note, I butchered the code to make this, ignore the actual contents).
If someone knew what caused the delay between presses of "Accept", they could probably reproduce this in a much cleaner way. I did it by increasing the file length via gratuitous copy & paste, but it still was fairly short compared to my own repo.

I don't know all the inner workings of VSCode, so my terminology might be a little off, but from what I can see:

Pressing Accept on one of the conflict resolution lines puts the file/module into a state that begins to refresh the page to update the conflict resolution lines.
During whatever process is running after the previous Accept is triggered, there is a window in which you can press Accept on a different conflict marker, and it will try to process it. This causes it to incorrectly delete conflict markers. Once triggered, the conflict markers are now in an invalid state and don't process correctly. I'm guessing there is a tracking mechanism that gets desync somewhere internally.
Something about the background process occurring after hitting Accept can take longer than usual - Not sure if it's related to the file size, project size, or the cleanliness of the git project at the time.  The longer the process takes, however, the larger the window to trigger this bug. I can very easily trigger it in my own code, which takes ~1 second or more to finish the conflict updates.
To 'avoid' the failure to edit cleanly, you have to wait until the original conflict resolution line (Accept Current, Accept Incoming, etc) vanishes and the editor removes the line. Once that line vanishes, the next conflict resolution will work just fine.

I see the same this. it's not new, though, this issue has been present for at least a year.
when clicking one of the conflict resolution lens links ("Accept Current Change", etc...), while the change to the text document takes place immediately, it can take at least a few seconds for the conflict resolution lens to disappear. this seems to indicate that there's some background processing going on to re-locate the other conflict resolution lenses in the same document.
If one of those other conflict resolution lens links is clicked before the first lens has been removed, the diffs applied to document will be applied at the lines numbers originally indicated by the second lens. however, if the first lens was above the second in the document then those line numbers are likely no longer valid and the second diff is applied at the wrong location in the document.

I see the same thing. it's not new, though, this issue has been present for at least a year.
when clicking one of the conflict resolution lens links ("Accept Current Change", etc...), while the change to the text document takes place immediately, it can take at least a few seconds for the conflict resolution lens to disappear. this seems to indicate that there's some background processing going on to re-locate the other conflict resolution lenses in the same document.
If one of those other conflict resolution lens links is clicked before the first lens has been removed, the diffs applied to document will be applied at the lines numbers originally indicated by the second lens. however, if the first lens was above the second in the document then those line numbers are likely no longer valid and the second diff is applied at the wrong location in the document.
Another symptom is that you can (if you're quick enough) repeatedly click the same "Accept Current Change" (link before it disappears) and basically swallow the whole of the rest of your document below that point.
not sure if it's relevant, but in my case i'm using vscode on windows connecting to a WSL guest. my .git directory is ~3GB with 60K commits.

I also have this issue and it makes large merges an absolute pain. Seems to occur for both rebasing and squash/merge. Similar to Spongman I am also running in a WSL2 environment.

During whatever process is running after the previous Accept is triggered, there is a window in which you can press Accept on a different conflict marker, and it will try to process it. This causes it to incorrectly delete conflict markers.

In my experience it's not just quickly clicking two "Accept" links that triggers it. Editing the buffer by hand and then clicking "Accept" causes the same issue. This seems to be async gone wrong: when clicking "Accept" just after the file changed, it is using an outdated model of the file to determine which lines to remove.


Create Contributing.md File for Clear Contributor Guidelines

The project currently lacks a Contributing.md file, which makes it difficult for new contributors to understand the contribution process. Adding this file with clear guidelines will make onboarding smoother and encourage more contributions. can you please assign it to me under gssoc and hacktoberfest

we're no longer accepting documentation changes. If you still want to contribute, you are good to go, but we'll not add any level to it.


Support for easier installation of certain VMODs

Hey folks,
Thanks for the nice container. I was wondering if it would be possible to provide an easier way to install certain VMODs, mainly those that are official (https://github.com/varnish/varnish-modules).
I think it would make sense to provide something like RUN install-vmod.sh xkey as a simplification. The container would not really become a lot bigger by just this file but it would allow to make installation of the most commonly used VMODs in custom containers way easier as well as centralize efforts in keeping the instructions needed in a central place rather than having different people maintaining their own commands 😄
For example I just tried to install xkey for a few local tests but I'm completely new to the whole ecosystem so I have no clue what dependencies are needed etc.

Hmm... Since there's now an official varnish Docker image. I think we should deprecate this one? It's been a long journey, but it's coming to an end.