TPU v4 install guide

This PR validates that it is possible to use optimum-tpu on older GCP TPUs, especially on the TPU v4 generation.
Originally, this repository is targeting TPU v5e. However, there is still both an existing large installed based for older TPU generations and, importantly, the TPU Research Council grants only encompass the older generations v2, v3, and v4 but no v5 and newer. Therefore, I wanted to ascertain that also on TPU v4 optimum-tpu can be used to accelerate research via huggingface.
Essentially, I have provided a validated and tested install plus the deactivation of pallas and Jetstream, which only target TPU v5.
Note that the install script is still intended to be run by hand although full automation should be straightforward.
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?

Travelling today. Will have a look and test tomorrow.
From: Alvaro Moran @.>
Date: Monday, 21. October 2024 at 10:19
To: huggingface/optimum-tpu @.>
Cc: Artus Krohn-Grimberghe @.>, Author @.>
Subject: Re: [huggingface/optimum-tpu] TPU v4 install guide (PR #108)
@tengomucho requested changes on this pull request.
Thanks for the contributions! It's true that optimum-tpu is focused mostly on v5 and future platforms, but if this helps you I would be happy to have that.
I just moved most of the install script to the cli.py script, do you think you can do the same?

In install-on-TPU-v4.shhttps://github.com/huggingface/optimum-tpu/pull/108#discussion_r1808297192:

@@ -0,0 +1,24 @@
+sudo apt remove unattended-upgrades
+sudo apt update
+export PJRT_DEVICE=TPU
+export PATH="/home/artuskg/.local/bin:$PATH"

I think you can generalize this, e.g.:
⬇ Suggested change
-export PATH="/home/artuskg/.local/bin:$PATH"
+export PATH="$HOME/.local/bin:$PATH"

In install-on-TPU-v4.shhttps://github.com/huggingface/optimum-tpu/pull/108#discussion_r1808298190:

@@ -0,0 +1,24 @@
+sudo apt remove unattended-upgrades
+sudo apt update
+export PJRT_DEVICE=TPU

you do not need this variable to install the packages

In install-on-TPU-v4.shhttps://github.com/huggingface/optimum-tpu/pull/108#discussion_r1808298588:

@@ -0,0 +1,24 @@
+sudo apt remove unattended-upgrades

why do you remove unattended-upgrades?

In install-on-TPU-v4.shhttps://github.com/huggingface/optimum-tpu/pull/108#discussion_r1808305685:

+python -m venv optimum_tpu_env
+source optimum_tpu_env/bin/activate

why do you need a virtual environment?

In install-on-TPU-v4.shhttps://github.com/huggingface/optimum-tpu/pull/108#discussion_r1808306585:

+pip install build
+pip install --upgrade setuptools
+sudo apt install python3.10-venv

+git clone https://github.com/huggingface/optimum-tpu.git
+
+cd optimum-tpu
+make
+make build_dist_install_tools
+make build_dist
+
+python -m venv optimum_tpu_env
+source optimum_tpu_env/bin/activate
+
+pip install torch==2.4.0 torch_xla[tpu]==2.4.0 torchvision -f https://storage.googleapis.com/libtpu-releases/index.html
+pip uninstall torchvision # it might insist von 2.4.1
I think it is simply better to avoid installing torchvision

In install-on-TPU-v4.shhttps://github.com/huggingface/optimum-tpu/pull/108#discussion_r1808306774:

+git clone https://github.com/huggingface/optimum-tpu.git

+cd optimum-tpu
+make
+make build_dist_install_tools
+make build_dist
+
+python -m venv optimum_tpu_env
+source optimum_tpu_env/bin/activate
+
+pip install torch==2.4.0 torch_xla[tpu]==2.4.0 torchvision -f https://storage.googleapis.com/libtpu-releases/index.html
+pip uninstall torchvision # it might insist von 2.4.1
+pip install -e .
+
+huggingface-cli login
+gsutil cp -r gs://entropix/huggingface_hub ~/.cache/huggingface/hub
what is this for?

In pyproject.tomlhttps://github.com/huggingface/optimum-tpu/pull/108#discussion_r1808308886:

@@ -61,10 +61,11 @@ tests = ["pytest", "safetensors"]
quality = ["black", "ruff", "isort"]

Jetstream/Pytorch support is experimental for now, it needs to be installed manually.
Pallas is pulled because it will install a compatible version of jax[tpu].
-jetstream-pt = [
you do not need to comment this: you will only install it if you do pip install optimum-tpu[pallas], otherwise it should not pull the dependency
—
Reply to this email directly, view it on GitHubhttps://github.com/huggingface/optimum-tpu/pull/108#pullrequestreview-2381253921, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMYYD63YPAGO4VWVGETFJ73Z4S2H7AVCNFSM6AAAAABQIO7SXSVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDGOBRGI2TGOJSGE.
You are receiving this because you authored the thread.Message ID: @.***>

Hola,
when using pip install optimum-tpu to continue to use it as a cli install tool, I have to realize that your cli changes are not yet implemented as of the latest pypi release. When I then go back to installing via pip -e and I am not allowed to venv, I still end up in this spot:
pip install --upgrade setuptools
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: setuptools in /home/artuskg/.local/lib/python3.10/site-packages (75.2.0)
artuskg@t1v-n-be3833de-w-0:~/optimum-tpu$ pip install -e .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/artuskg/optimum-tpu
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Installing collected packages: UNKNOWN
Running setup.py develop for UNKNOWN
error: subprocess-exited-with-error
× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [44 lines of output]
    /tmp/pip-build-env-odocea89/overlay/local/lib/python3.10/dist-packages/setuptools_scm/_integration/setuptools.py:31: RuntimeWarning:
    ERROR: setuptools==59.6.0 is used in combination with setuptools_scm>=8.x

    Your build configuration is incomplete and previously worked by accident!
    setuptools_scm requires setuptools>=61

    Suggested workaround if applicable:
     - migrating from the deprecated setup_requires mechanism to pep517/518
       and using a pyproject.toml to declare build dependencies
       which are reliably pre-installed before running the build tools

      warnings.warn(
    running develop
    /usr/lib/python3/dist-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    WARNING: The user site-packages directory is disabled.
    /usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    error: can't create or remove files in install directory

    The following error occurred while trying to add or remove files in the
    installation directory:

        [Errno 13] Permission denied: '/usr/local/lib/python3.10/dist-packages/test-easy-install-11654.write-test'

    The installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:

        /usr/local/lib/python3.10/dist-packages/

    Perhaps your account does not have write access to this directory?  If the
    installation directory is a system-owned directory, you may need to sign in
    as the administrator or "root" account.  If you do not have administrative
    access to this machine, you may wish to choose a different installation
    directory, preferably one that is listed in your PYTHONPATH environment
    variable.

    For information on other options, you may wish to consult the
    documentation at:

      https://setuptools.pypa.io/en/latest/deprecated/easy_install.html

    Please make the appropriate changes for your system and try again.

    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

error: subprocess-exited-with-error
× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [44 lines of output]
/tmp/pip-build-env-odocea89/overlay/local/lib/python3.10/dist-packages/setuptools_scm/_integration/setuptools.py:31: RuntimeWarning:
ERROR: setuptools==59.6.0 is used in combination with setuptools_scm>=8.x
Your build configuration is incomplete and previously worked by accident!
setuptools_scm requires setuptools>=61

Suggested workaround if applicable:
 - migrating from the deprecated setup_requires mechanism to pep517/518
   and using a pyproject.toml to declare build dependencies
   which are reliably pre-installed before running the build tools

  warnings.warn(
running develop
/usr/lib/python3/dist-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
WARNING: The user site-packages directory is disabled.
/usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 13] Permission denied: '/usr/local/lib/python3.10/dist-packages/test-easy-install-11654.write-test'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /usr/local/lib/python3.10/dist-packages/

Perhaps your account does not have write access to this directory?  If the
installation directory is a system-owned directory, you may need to sign in
as the administrator or "root" account.  If you do not have administrative
access to this machine, you may wish to choose a different installation
directory, preferably one that is listed in your PYTHONPATH environment
variable.

For information on other options, you may wish to consult the
documentation at:

  https://setuptools.pypa.io/en/latest/deprecated/easy_install.html

Please make the appropriate changes for your system and try again.

[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
artuskg@t1v-n-be3833de-w-0:~/optimum-tpu$ pip install --user --upgrade 'setuptools>=61'
Requirement already satisfied: setuptools>=61 in /home/artuskg/.local/lib/python3.10/site-packages (75.2.0)
artuskg@t1v-n-be3833de-w-0:~/optimum-tpu$ pip install --user -e .
Obtaining file:///home/artuskg/optimum-tpu
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Installing collected packages: UNKNOWN
Running setup.py develop for UNKNOWN
error: subprocess-exited-with-error
× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [44 lines of output]
    /tmp/pip-build-env-_j6dh9z3/overlay/local/lib/python3.10/dist-packages/setuptools_scm/_integration/setuptools.py:31: RuntimeWarning:
    ERROR: setuptools==59.6.0 is used in combination with setuptools_scm>=8.x

    Your build configuration is incomplete and previously worked by accident!
    setuptools_scm requires setuptools>=61

    Suggested workaround if applicable:
     - migrating from the deprecated setup_requires mechanism to pep517/518
       and using a pyproject.toml to declare build dependencies
       which are reliably pre-installed before running the build tools

      warnings.warn(
    running develop
    /usr/lib/python3/dist-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    WARNING: The user site-packages directory is disabled.
    /usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    error: can't create or remove files in install directory

    The following error occurred while trying to add or remove files in the
    installation directory:

        [Errno 13] Permission denied: '/usr/local/lib/python3.10/dist-packages/test-easy-install-11798.write-test'

    The installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:

        /usr/local/lib/python3.10/dist-packages/

    Perhaps your account does not have write access to this directory?  If the
    installation directory is a system-owned directory, you may need to sign in
    as the administrator or "root" account.  If you do not have administrative
    access to this machine, you may wish to choose a different installation
    directory, preferably one that is listed in your PYTHONPATH environment
    variable.

    For information on other options, you may wish to consult the
    documentation at:

      https://setuptools.pypa.io/en/latest/deprecated/easy_install.html

    Please make the appropriate changes for your system and try again.

    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

error: subprocess-exited-with-error
× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [44 lines of output]
/tmp/pip-build-env-_j6dh9z3/overlay/local/lib/python3.10/dist-packages/setuptools_scm/_integration/setuptools.py:31: RuntimeWarning:
ERROR: setuptools==59.6.0 is used in combination with setuptools_scm>=8.x
Your build configuration is incomplete and previously worked by accident!
setuptools_scm requires setuptools>=61

Suggested workaround if applicable:
 - migrating from the deprecated setup_requires mechanism to pep517/518
   and using a pyproject.toml to declare build dependencies
   which are reliably pre-installed before running the build tools

  warnings.warn(
running develop
/usr/lib/python3/dist-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
WARNING: The user site-packages directory is disabled.
/usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 13] Permission denied: '/usr/local/lib/python3.10/dist-packages/test-easy-install-11798.write-test'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /usr/local/lib/python3.10/dist-packages/

Perhaps your account does not have write access to this directory?  If the
installation directory is a system-owned directory, you may need to sign in
as the administrator or "root" account.  If you do not have administrative
access to this machine, you may wish to choose a different installation
directory, preferably one that is listed in your PYTHONPATH environment
variable.

For information on other options, you may wish to consult the
documentation at:

  https://setuptools.pypa.io/en/latest/deprecated/easy_install.html

Please make the appropriate changes for your system and try again.

[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
This was on a newly created TPU v4. Please advise on how I should best test the cli.py installation feature because in general I am sure this should work as you indicated.

I think creating a virtual environment (venv, virtualenv, docker or conda) is a good way to to prepare your VM for optimum-tpu installation. Once you activate/enter that, it should install setuptools in the environment and stop complaining trying to install that system-wide and you should be able to avoid other issues. I will make a release soon (probably by the end of the week), so hopefully some things will be clearer by then.


EventHub Trigger does not run if Azure Functions blade is closed

Hi there,
Should an EventHub-triggered Azure Function continue to run if I close my Azure Function blade / logging environment?
So far, if I browse away from my Azure Function while it's running, the data stops flowing.
I tried adding a cron-based trigger to my function, but that didn't work / help.
Thanks for any insight...
TTFN
Gregg

Definitely it should keep running. Is your function running in a Dynamic plan or Classic plan? If classic, make sure it is Basic or Standard, and has Always On turned on. When running under the Dynamic plan, the infrastructure will handle that for you.
Also, can you share your function app name, either directly or indirectly? This will help us investigate.

May be related to https://github.com/Azure/azure-webjobs-sdk-script/issues/336.

Hi there,
Yes, it looks exactly like issue #336. I'm using Event Hubs as a trigger, but that's the only difference.
I can't click "Always On" because the only service plan listing I can find is "Free" way over in the "Scale Up" blade. I had assumed that Azure Functions were billed differently.
Azure Functions says it is billed "per invocation" rather than being a regular Azure Web Site. So, do you get double-billed (Azure Function invocations + Azure Web Site tier) just to make your Azure Function work?
I mean, I intend for it to be "Always On" (or "On every few minutes" - it doesn't really need to be triggered at EVERY EventHub write - but you can't currently cron an Event Hub)... but I had thought the point of Azure Functions was to not maintain more infrastructure than you need just to shovel some data from one place to another...
Thanks for explaining the issue!
TTFN
Gregg

OK, I see what the issue was. I didn't create a "New" plan - I used one I already had going to reduce the clutter in my subscription. So I never saw the Dynamic vs. Classic selection.
It sounds like that will fix everything - including the billing!
Thanks!

Hi Folks
I am facing the same issue for last 2 weeks. I have a Event Hub Trigger function app with consumption plan. It was working, when I open the portal in UI & continue to run for 3-4 hours , after that it automatically sleeps, when no request coming to event hub.
Next day when request come again , it is not triggering, until I open portal.
I have to swap forcefully from Consumption Plan to App Service Plan to run this.
When we can expect fix for Consumption Plan????
Thanks.

Hi folks
I am with the same problem related by @navilwayne.
There is a expectation to solve it?
Thanks.

Hello
count me in. I got the same problem.
Thanks
Kai

Hi Kai
Please refer below link & solved ur problem:
Azure Function Not Triggering by Event Hub #122
U have to remove entity path from connection string in last. Then it will work.


Polish validation message translation

Current coverage is 89.88%

Merging #12 into master will not affect coverage as of 9f8a9b7

@@            master     #12   diff @@
======================================
  Files            5       5
  Stmts          168     168
  Branches         0       0
  Methods         32      32
======================================
  Hit            151     151
  Partial          0       0
  Missed          17      17

Review entire Coverage Diff as of 9f8a9b7

Powered by Codecov. Updated on successful CI builds.

Thanks :)

Thanks for great "product" :)


Upgrade next - just upgrade packages

Who is this for?
anyone that is using the codebase, updated tools make the developer experience better
What is it doing for them?
react version updated to 18.2.0 in:

cardigan
catalogue
common
content
identity

react-dom version updated to 18.2.0 in:

common

next version updated to 13.1.6 in:

common
catalogue
content
identity

I made use of yarn list [package-name] to find where I needed to update, because multiple versions of react will cause context issues
transpiling modules is supported natively so next-transpile-modules is now deprecated, and will be removed

removed from common next.config.json
removed from identity

next/mdx updated to latest version
ran the new link codemod
npx @next/codemod new-link .

ran the new Image initial codemod
npx @next/codemod next-image-to-legacy-image .

I seem to be having some issues with links when running locally:

Run yarn content
Go to http://localhost:3000/whats-on
Click to go to another page, e.g. click on "In Plain Sight" ~> I get dumped back to the /whats-on/ page
Click to go to that page a second time ~> I actually go to the page

Any idea what might be causing this?

I'm also getting a bunch of warnings:
warn  - Invalid next.config.js options detected:
  - The root value has an unexpected property, webpackDevMiddleware, which is not in the list of allowed properties (amp, analyticsId, assetPrefix, basePath, cleanDistDir, compiler, compress, crossOrigin, devIndicators, distDir, env, eslint, excludeDefaultMomentLocales, experimental, exportPathMap, generateBuildId, generateEtags, headers, httpAgentOptions, i18n, images, modularizeImports, onDemandEntries, optimizeFonts, output, outputFileTracing, pageExtensions, poweredByHeader, productionBrowserSourceMaps, publicRuntimeConfig, reactStrictMode, redirects, rewrites, sassOptions, serverRuntimeConfig, skipMiddlewareUrlNormalize, skipTrailingSlashRedirect, staticPageGenerationTimeout, swcMinify, trailingSlash, transpilePackages, typescript, useFileSystemPublicRoutes, webpack).
  - The root value has an unexpected property, configOrigin, which is not in the list of allowed properties (amp, analyticsId, assetPrefix, basePath, cleanDistDir, compiler, compress, crossOrigin, devIndicators, distDir, env, eslint, excludeDefaultMomentLocales, experimental, exportPathMap, generateBuildId, generateEtags, headers, httpAgentOptions, i18n, images, modularizeImports, onDemandEntries, optimizeFonts, output, outputFileTracing, pageExtensions, poweredByHeader, productionBrowserSourceMaps, publicRuntimeConfig, reactStrictMode, redirects, rewrites, sassOptions, serverRuntimeConfig, skipMiddlewareUrlNormalize, skipTrailingSlashRedirect, staticPageGenerationTimeout, swcMinify, trailingSlash, transpilePackages, typescript, useFileSystemPublicRoutes, webpack).
  - The root value has an unexpected property, target, which is not in the list of allowed properties (amp, analyticsId, assetPrefix, basePath, cleanDistDir, compiler, compress, crossOrigin, devIndicators, distDir, env, eslint, excludeDefaultMomentLocales, experimental, exportPathMap, generateBuildId, generateEtags, headers, httpAgentOptions, i18n, images, modularizeImports, onDemandEntries, optimizeFonts, output, outputFileTracing, pageExtensions, poweredByHeader, productionBrowserSourceMaps, publicRuntimeConfig, reactStrictMode, redirects, rewrites, sassOptions, serverRuntimeConfig, skipMiddlewareUrlNormalize, skipTrailingSlashRedirect, staticPageGenerationTimeout, swcMinify, trailingSlash, transpilePackages, typescript, useFileSystemPublicRoutes, webpack).
  - The value at .amp.canonicalBase must be 1 character or more but it was 0 characters.
  - The value at .assetPrefix must be 1 character or more but it was 0 characters.
  - The value at .experimental.outputFileTracingRoot must be 1 character or more but it was 0 characters.
  - The value at .i18n must be an object but it was null.
  - The value at .reactStrictMode must be a boolean but it was null.

See more info here: https://nextjs.org/docs/messages/invalid-next-config
We detected TypeScript in your project and reconfigured your tsconfig.json file for you. Strict-mode is set to false by default.

The following suggested values were added to your tsconfig.json. These values can be changed to fit your project's needs:

	- incremental was set to true

but I guess we might fix these in a subsequent PR?

This is odd – I'm only getting it intermittently.
A possible clue: I'm seeing this in my console whenever the "loop back to What’s On" occurs:

warn  - Fast Refresh had to perform a full reload. Read more: https://nextjs.org/docs/messages/fast-refresh-reload

But there's nothing I'm doing that would trigger that, I think? I’m not changing any files locally.
Also I think this might be a WebKit issue (yay!), I can see it in Safari but I haven't seen it in Chrome yet.


startOnBoot not working (android)

StartOnBoot is not working with a barebones android app with React 0.40. Is there a setting elsewhere I need to enable?

Notice: this issue has been closed because it has been inactive for 218 days. You may reopen this issue if it has been closed in error.


Outdated rustc version

api_server_1  |  Downloading itoa v0.4.1
api_server_1  |    Compiling rocket_codegen v0.3.10
api_server_1  |    Compiling diesel_infer_schema v0.16.0
api_server_1  |    Compiling dotenv v0.10.1
api_server_1  |    Compiling diesel v1.2.2
api_server_1  | error: failed to run custom build command for `rocket_codegen v0.3.10`
api_server_1  | process didn't exit successfully: `/usr/src/app/target/debug/build/rocket_codegen-f122471f5345efe5/build-script-build` (exit code: 101)
api_server_1  | --- stderr
api_server_1  | Error: Rocket codegen requires a more recent version of rustc.
api_server_1  | Use `rustup update` or your preferred method to update Rust.
api_server_1  | Installed version is: 1.27.0-nightly (2018-05-02). Minimum required: 1.27.0-nightly (2018-05-04).
api_server_1  | thread 'main' panicked at 'Aborting compilation due to incompatible compiler.', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rocket_codegen-0.3.10/build.rs:50:17
api_server_1  | note: Run with `RUST_BACKTRACE=1` for a backtrace.
api_server_1  |
api_server_1  | warning: build failed, waiting for other jobs to finish...
web_1         | Compiled successfully!
web_1         |
web_1         | You can now view web in the browser.
web_1         |
web_1         |   Local:            http://localhost:3000/
web_1         |   On Your Network:  http://<IP_ADDRESS>:3000/
web_1         |
web_1         | Note that the development build is not optimized.
web_1         | To create a production build, use yarn build.
web_1         |
api_server_1  | error: build failed

Does this mean that the latest published rustlang/rust:nightly is not up-to-date with what rocket_codegen requires?

Do the steps in the Troubleshooting section https://github.com/ghotiphud/rust-web-starter#troubleshooting clear it up?

Cleaning up this older issue after recent updates.


Some PDFs cause docling-parse to fail with assertion errors

[x] PMC9356128.pdf
the above doc produces this error and python terminates with SIGABRT
Assertion failed: (_range.first<=c and c<=_range.second), function set_mapping, file cmap.h, line 203

This is handled since docling 1.8.5. Closing until further evidence appears.


Multi Modal RAG for avoiding the hallucinations to some extenet .....

The main problem with the traditional RAG was hallucination if our data source contains different types of data rather than text. Especially if the data sources cotains tables as images , the RAG system needs to work very well. I have a solution to solve this to some extent using unstructured.io methods.
@Neilblaze @UppuluriKalyani @SaiNivedh26

what does the following question mean?
Attach before & after screenshots in your PR for clarity.

@UppuluriKalyani @SaiNivedh26  @Neilblaze

what does the following question mean?
Attach before & after screenshots in your PR for clarity.

@UppuluriKalyani @SaiNivedh26 @Neilblaze
Results

what does the following question mean?
Attach before & after screenshots in your PR for clarity.

@UppuluriKalyani @SaiNivedh26 @Neilblaze

what does the following question mean?
Attach before & after screenshots in your PR for clarity.

@UppuluriKalyani @SaiNivedh26 @Neilblaze

Results

@chakka-guna-sekhar-venkata-chennaiah Screenshots of your implementation results.
before and after mean if you find any bug or something you should show it like before it looks like this after update it looks like this.

@UppuluriKalyani Can we add the scratch project for the raised issue instead of solving the bug? Because i want to contribute that project to this repo.

@UppuluriKalyani Can we add the scratch project for the raised issue instead of solving the bug? Because i want to contribute that project to this repo.

Yes you can I'm just saying if it is bug then you should do like that if it is feature just show me the results.

@UppuluriKalyani  I thought that open source contribution means creating new projects that's why I raised the pr. But later you informed that I need to change or add the functionality of any existing project right. That's why I'm deciding to delete my pr's


Will there be an option to add a public IP?

It would be nice to be able to add a public IP to the VM.

Yes, we can add an option for a public IP. Adding to pipeline.


SW 5040 Created VoteDecisions data tables

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#1757  👈
#1756
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @tommylau523 and the rest of your teammates on  Graphite

Merge activity

Mar 14, 1:07 PM EDT: @tommylau523 started a stack merge that includes this pull request via Graphite.


Needs testing: Biber support

This fixes #507 .
Additions

Added biber compiler as option in a bibtex run configuration.
Added \printbibliography and \addbibresource to autocomplete
A bibtex run configuration with biber compiler set as default is created if \printbibliography is used in the fileset.

I tested with and without the use of the auxil/ directory, but I didn't test other folder configurations.
Note that somethings the bibtex run configuration is not created automatically, but that also happens on master for me so I think that's a different problem.
to do:

[ ] test on linux

In any case you can now change the compiler to biber easily after having created a bibtex run config.
Test files:
\documentclass{article}

\usepackage[autostyle]{csquotes}
\usepackage{biblatex}

\addbibresource{books.bib}

\begin{document}
    This sentence cites Knuth~\cite{knuth1990}.
    \printbibliography
\end{document}

@Book{knuth1990,
    author    = {Knuth, Donald E.},
    title     = {The {\TeX}book},
    year      = {1990},
    isbn      = {0-201-13447-0},
    publisher = {Addison\,\textendash\,Wesley},
}

If anybody needs instructions how to create such a bibtex run configuration I can add them to the readme.
PS It was easy to find where in the code I had to add this but I'd appreciate it if you document your functions next time ;)

Please test so we can merge the PR :)

I tested on linux but bibtex didn't work anymore: the biber run config got removed from the latex run config when I clicked 'ok'. Same for a bibtex run config, although it was still run somehow. Sometimes. Need to test more. Also, some run configs were greyed out for some reason.
Everything which this pull request is about, the biber run configuration itself, worked though, when I run the right configs in the right order the right number of times manually then the result was fine.

What is the status on this?

I first had to fix some things on my linux system before I could try again, I hope I did that yesterday so I will try again soon. Merging master hopefully helps to fix the run config confusion - theoretically it should work so I don't expect big changes to the code before this works. Thanks for reviewing!

Tested on Windows and Linux that it works now, though I still have to create the bibtex run configuration manually.

Resolved conflicts :)

So, merge or push to b0.6.3?

So, merge or push to b0.6.3?

Merge? As I said I tested on Windows and Linux now that it works, what else would you want to be fixed?

Nvm. :)


feat(lambda): Added FunctionURLConfig resource

Description of your changes
Added Lambda FunctionURLConfig resource
I have:

[x] Read and followed Crossplane's contribution process.
[x] Run make reviewable test to ensure this PR is ready for review.

How has this code been tested

make run in a test cluster
tested create / update / delete
tested using an existing Function resource as ref

Thanks for your contribution @pintonunes. Could you add an example to resource as well?

@MisterMX, added example.

@MisterMX after merging the #1259 the check-diff started to fail.
18:24:01 [ .. ] checking that branch is clean
M apis/batch/v1alpha1/zz_compute_environment.go
M apis/batch/v1alpha1/zz_enums.go
M apis/batch/v1alpha1/zz_generated.deepcopy.go
M apis/batch/v1alpha1/zz_job_queue.go
M apis/batch/v1alpha1/zz_types.go
M package/crds/batch.aws.crossplane.io_computeenvironments.yaml
M package/crds/batch.aws.crossplane.io_jobqueues.yaml
M pkg/controller/batch/computeenvironment/zz_conversions.go
M pkg/controller/batch/jobqueue/zz_conversions.go
Go sdk was updated to 1.44 and that will generate more stuff that was not included on #1259

The check-diff error has been fixed in https://github.com/crossplane-contrib/provider-aws/pull/1561. @pintonunes can you rebase on the latest master?

done @MisterMX


Add Travis to the project

It would be a good idea to add a travis script for continuous integration to ensure that every new PR follows quality.

@eeshadutta Is this bug up for solving?

Hi @akshitac8 Travis has already been added to the project


Stop querying status of infrastructure in IMC once configured

There is probably no reason to continue checking the status of configured infrastructure. If it changes from configured to something else, this information is not used anyway (I think).

Change made.


move FB SDK dependency to 4.20.2

move FB SDK dependency to 4.20.2, debugged, it works well!

@dbotha @samkaufman please help review this PR. Thanks!

@dbotha @samkaufman can you take a minute to review this?


Update vector.rst

Couldn't find get method

@github-actions crossbow submit preview-docs

@kou @lidavidm shall we make the title name as Minor: Docs Update vector.rst to use lookup instead of get or something meaningful?

@kou @lidavidm shall we make the title name as Minor: Docs Update vector.rst to use lookup instead of get or something meaningful?

@duhanmin appreciate if you could make this change 🙂

http://crossbow.voltrondata.com/pr_docs/40940/java/vector.html#dictionary-encoding

er, yes, the PR title needs to change as a basic requirement - that goes without saying

Agreed!
Unfortunately I don't have the edit title access.
Thanks @kou 👍