WebGL support

I'm attempting to use this binary to create PDFs from pages that include WebGL canvas elements created by Mapbox. Basically I'm wondering if this should this work out of the box with the default flags, or different set of flags and/or a custom built binary will be necessary(to enable something like swiftshader).
With version 1.14.0 and the default args exposed by this package, and visiting somewhere like:
https://docs.mapbox.com/mapbox-gl-js/example/simple-map/
or
http://mapbox.github.io/mapbox-gl-supported/diagnostics.html
and attempting to generate a PDF results in blank spots in the PDF.
If this is something you think should be supported, then I can provide a more fleshed out reproducer.

@michaeltmiller I don't see why that shouldn't be supported, so I'd be happy to look into it if you could provide some steps to replicate. Also, is this something that affects puppeteer as well or just this package? If the former, it might be better to open an issue with puppeteer repo instead.

This issue can probably be closed as I've found a workaround.
Copying the swiftshader binaries (libEGL.so & libGLESv2.so) into /tmp/swiftshader as described here , removing the --disable-gpu flag, and adding --enable-webgl to the args array exposed by this package results in webgl canvas rendering as desired.

Thanks for the feedback @michaeltmiller, I compressed these SOs with Brotli and they amount to 981K:
383K libEGL.so
99K  libEGL.so.br
3.4M libGLESv2.so
882K libGLESv2.so.br

Do you think it would be worthwhile shipping these and adding a helper method to enable WebGL support?

Related: https://github.com/GoogleChrome/puppeteer/issues/1260

Absolutely! That would simply our use case moving forward. Looking around the web it seems like it's a problem others are running into as well, so adding support with minimal configuration sounds like a pretty big win to me. Especially if the added overhead is that small for users that don't require it.
Thanks @alixaxel

It would be nice to have support for WebGL !

@michaeltmiller @TimotheeJeannin Out of the box support for WebGL been released on 1.15.0.
Let me know if you face any issues.

Working on my side. :) Thank you!

I'm trying to create a PDF with v10.1.0 on lambda with preserveDrawingBuffer in the map, though still, the map is empty, not sure why here are my chrome args
[
  '--autoplay-policy=user-gesture-required',
  '--disable-background-networking',
  '--disable-background-timer-throttling',
  '--disable-backgrounding-occluded-windows',
  '--disable-breakpad',
  '--disable-client-side-phishing-detection',
  '--disable-component-update',
  '--disable-default-apps',
  '--disable-dev-shm-usage',
  '--disable-domain-reliability',
  '--disable-extensions',
  '--disable-features=AudioServiceOutOfProcess,IsolateOrigins,site-per-process',
  '--disable-hang-monitor',
  '--disable-ipc-flooding-protection',
  '--disable-offer-store-unmasked-wallet-cards',
  '--disable-popup-blocking',
  '--disable-print-preview',
  '--disable-prompt-on-repost',
  '--disable-renderer-backgrounding',
  '--disable-setuid-sandbox',
  '--disable-speech-api',
  '--disable-sync',
  '--disable-web-security',
  '--disk-cache-size=33554432',
  '--hide-scrollbars',
  '--ignore-gpu-blocklist',
  '--metrics-recording-only',
  '--mute-audio',
  '--no-default-browser-check',
  '--no-first-run',
  '--no-pings',
  '--no-sandbox',
  '--no-zygote',
  '--password-store=basic',
  '--use-gl=swiftshader',
  '--use-mock-keychain',
  '--window-size=1920,1080',
  '--start-maximized'
]

Any steps that might be useful for debugging what's going wrong?


I want to have a name feature

Umm hey I want to have a custom name

Can you clarify please?

Well I want somthing else than “GUEST” as the username of the phonebook

GUEST is the default name when a person hasn't signed in. To change it, export a phonebook. It will ask for the username. You can also import a ready made phonebook.


Question about wordvec_array.pickle

Hello, I am interested in your word vectors. Could you elaborate on how to generate this file?

Hello,Have you found the file 'val_17_48.json', 'val_unseen.json' and ‘train_17_48.json’ mentioned in the code.Can you share it?Thank you so much.


No font's Loading

When running astro in development mode, everything works as expected. However, when I build with node as my adapter and build it. It never loads the fonts.
This is in my <head>
<AstroFont
	config={[
		{
			name: "BankPrinter",
			src: [
				{
					style: "inherit",
					weight: "inherit",
					path: "/fonts/BankPrinter.ttf",
				},
			],
			preload: true,
			fallback: "sans-serif",
			display: "swap",
			selector: ".BankPrinter",
		},
		{
			name: "StretchPro",
			src: [
				{
					style: "inherit",
					weight: "inherit",
					path: "/fonts/StretchPro.otf",
				},
			],
			preload: true,
			fallback: "sans-serif",
			display: "swap",
			selector: ".StretchPro",
		},
	]}
/>

In the dist folder
<link href=/../../../../../../../..//fonts/BankPrinter.ttf rel=preload type=font/ttf as=font crossorigin>
<link href=/../../../../../../../..//fonts/StretchPro.otf rel=preload type=font/otf as=font crossorigin>

And if I change the paths to load paths like "./fonts/"..., it does this and still never loads the font.
<link href=/../fonts/BankPrinter.ttf rel=preload type=font/ttf as=font crossorigin>
<link href=/../fonts/StretchPro.otf rel=preload type=font/otf as=font crossorigin>

Any fixes?

Hey @Dan-J-D,
In which directory in the project's directory do the font reside?

The fonts reside in astro's public directory with relative path of "./font/"...
Aswell, I have set preload to false and the fonts never get fetched. They have an inline css style but no request to fetch for the font

got it.
so you'd want to use this config instead:
---
import {join} from 'node:path';
import {AstroFont} from 'astro-font';
---

<AstroFont
	config={[
		{
			name: "BankPrinter",
			src: [
				{
					style: "inherit",
					weight: "inherit",
					path: join(process.cwd(), 'public', 'fonts', 'BankPrinter.ttf')
				},
			],
			preload: true,
			fallback: "sans-serif",
			display: "swap",
			selector: ".BankPrinter",
		},
		{
			name: "StretchPro",
			src: [
				{
					style: "inherit",
					weight: "inherit",
					path: join(process.cwd(), 'public', 'fonts', 'StretchPro.otf')
				},
			],
			preload: true,
			fallback: "sans-serif",
			display: "swap",
			selector: ".StretchPro",
		},
	]}
/>

path of "./font/"...

or "./fonts/" instead? because I see that in your paths you are using /fonts and not /font

I have tried the above and it doesn't work. I believe it is because I am compiling with output mode as "hybrid" so the relative path there would be "../client/". Since, astro produces a client & server folder. But I have tried to fix this by setting the base path to "../client/" but then it bugs and produces this.
<link href=/../web/fonts/BankPrinter.ttf rel=preload type=font/ttf as=font crossorigin>
<link href=/../../web/fonts/StretchPro.otf rel=preload type=font/otf as=font crossorigin>

oh nope, the paths won't changes as /client/ never comes into the play.

Can you share the project in a git repo or a zip?
I am happy to PR.

Yes, please give me a couple minutes to make a git repo.

Here you go: https://github.com/Dan-J-D/hello-world-font-test

Hey @Dan-J-D,
It works!
Here's how you'd wanna test it:
npm run build
npm run preview

I've created a PR.

https://github.com/rishi-raj-jain/astro-font/assets/46300090/401a5cdf-e944-45ab-842b-1cba5e35a009

@rishi-raj-jain Please try running in dev mode. The font should look like this

Sure, checking out the dev mode.

Hey @Dan-J-D,
I just checked astro-font works just as fine.
The font files are messed up by astro-critters. Try removing the critters integration in my PR changes and you'd see it works as expected.

Oh, perfect. Thank you so much!

No worries!


Local setup failing when using Ollama

Hello! I have been trying out the project, and I like the idea very much. Congrats!
Now to the issue:
I have been able to setup the ecosystem using OpenAI, now I'm trying to run it using the local setup (with Ollama), and I'm facing some issues when trying to interact with the chat. I have been able to use other functionalities, like login, saving webpage snapshots to the DB, and so on. Here's the traceback from the backend:

Traceback
Traceback (most recent call last):
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_transports/default.py", line 72, in map_httpcore_exceptions
    yield
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_transports/default.py", line 236, in handle_request
    resp = self._pool.handle_request(req)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
    raise exc from None
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 122, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "/Users/my.user/.pyenv/versions/3.10.13/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 61] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/middleware/cors.py", line 93, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/middleware/cors.py", line 144, in simple_response
    await self.app(scope, receive, send)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2405, in run_sync_in_worker_thread
    return await future
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 914, in run
    result = context.run(func, *args)
  File "/Users/my.user/Github/SurfSense/backend/server.py", line 108, in get_user_query_response
    localreturn = index.local_search(query=sub_query, search_space=search_space)
  File "/Users/my.user/Github/SurfSense/backend/HIndices.py", line 246, in local_search
    top_summaries_compressed_docs = top_summaries_retreiver.invoke(query)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_core/retrievers.py", line 254, in invoke
    raise e
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_core/retrievers.py", line 247, in invoke
    result = self._get_relevant_documents(
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain/retrievers/contextual_compression.py", line 44, in _get_relevant_documents
    docs = self.base_retriever.invoke(
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_core/retrievers.py", line 254, in invoke
    raise e
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_core/retrievers.py", line 247, in invoke
    result = self._get_relevant_documents(
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_core/vectorstores/base.py", line 1039, in _get_relevant_documents
    docs = self.vectorstore.similarity_search(query, **self.search_kwargs)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_chroma/vectorstores.py", line 582, in similarity_search
    docs_and_scores = self.similarity_search_with_score(
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_chroma/vectorstores.py", line 679, in similarity_search_with_score
    query_embedding = self._embedding_function.embed_query(query)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_ollama/embeddings.py", line 164, in embed_query
    return self.embed_documents([text])[0]
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/langchain_ollama/embeddings.py", line 159, in embed_documents
    embedded_docs = self._client.embed(self.model, texts)["embeddings"]
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/ollama/_client.py", line 262, in embed
    return self._request(
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/ollama/_client.py", line 70, in _request
    response = self._client.request(method, url, **kwargs)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_client.py", line 837, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_client.py", line 926, in send
    response = self._send_handling_auth(
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_client.py", line 954, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_client.py", line 991, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_client.py", line 1027, in _send_single_request
    response = transport.handle_request(request)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_transports/default.py", line 235, in handle_request
    with map_httpcore_exceptions():
  File "/Users/my.user/.pyenv/versions/3.10.13/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/my.user/.pyenv/versions/3.10.13-surfsense/lib/python3.10/site-packages/httpx/_transports/default.py", line 89, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 61] Connection refused

What can be the issue? I believe the issue is closely tide to the Ollama implementation, the only difference between my working example with OpenAI is that I'm setting IS_LOCAL_SETUP= 'true'. Let me know if you'd like more details. Running from a Mac M2 (ARM64) chip.

I think I understand my issue then: I wasn't running Ollama locally, I was under the impression this would be managed by the backend code.
Then, to use this setup I would need to run Ollama (probably following those instructions here: https://github.com/ollama/ollama?tab=readme-ov-file#ollama), and use the corresponding endpoint as a base_url. Is that it?

I think I understand my issue then: I wasn't running Ollama locally, I was under the impression this would be managed by the backend code.
Then, to use this setup I would need to run Ollama (probably following those instructions here: https://github.com/ollama/ollama?tab=readme-ov-file#ollama), and use the corresponding endpoint as a base_url. Is that it?

You can skip base_url if Ollama is running at their default port :)


GRPC - queryRoutes empty routes array

Problem.
lncli queryroutes ... ... return  routes full array  but rpc queryRoutes at the same moment  response with no error and routes [] length - 0

The queryroutes command for lncli, actually just hits the QueryRoutes RPC. I'd check your manual arguments to the RPC.

Resolved on IRC.

My mistake.


feat: Implement apikeys/ endpoint from swagger spec.

@ngoduykhanh
Hello,
The swagger specs defines a pdnsadmin/apikeys/<key_id> endpoint that was not implemented (it falls back on the /apikeys which return a list of keys instead of a specific key).
Added here

Thanks for your contribution. Merging the PR.

Thanks for your contribution. Merging the PR.


PublicSuffix.valid? returns true for any value containing two periods

Since the new PSL algorithm PublicSuffix.valid? returns true for any value containing two+ periods. This was previously not the case, and I'm not sure if it's expected behavior (didn't find any tests to indicate this).
irb(main):006:0> PublicSuffix.valid?("this is not valid...")
=> true

irb(main):010:0> PublicSuffix.valid?("this is not valid.")
=> false

If I understand correctly I should be able to pass in a different default_rule to fix the issue in our app for now, so I'll go ahead and do that.
If this is indeed an issue and there's anything I can do to help, please let me know.

Validating input is something I'm always torn about. As ideally, the input you provide to the library should be already filtered and validated.
The purpose of this library is not to validate the format of a domain, rather to determine how to decompose the domain name passed as argument.
I'm saying that because there are a large number of possible invalid inputs you can pass. The string you provided, for instance, contain spaces.
I already do some kind of limited validations:
https://github.com/weppos/publicsuffix-ruby/blob/master/lib/public_suffix.rb#L165-L167
but frankly I'm not inclined to add more complexity for a validation that is beyond the scope of this library.
I'll take some time to consider if adding multiple dots. However, adding it will solve one issue, but then what if

you pass dot at the end?
you pass a string with a space?

In this case, passing a different default_rule would not affect the algorithm.

I also have one more question for you. To better determine the extent of this issue, can I ask you how you currently use the library? What is the use case?

We currently use PublicSuffix.valid? to determine if user supplied input is a url, ip or other string. From your previous reply I understand that this is probably not the intended usage of the library, and I should probably consider a different approach.

Yes, correct. The purpose of this library is really not to handle URL validation.

FYI https://rubygems.org/gems/addressable is probably a better solution.

Yes, Addressable makes sense. I'll look into switching to that, we already have a dependency on it elsewhere.
Should I close this issue?
Thanks for your help!


Add all module variables

The problem
Currently, when you execute from manim import *, you are importing a whooping 553 names into your local namespace. For reference, numpy adds 621 names, while matplotlib adds 250. I would like to keep this down as much as possible. Further, a lot of the names imported by manim are unnecessary, and just the product of very messy practices when using imports within the codebase. This PR starts to address the latter problem by adding module __all__ variables. These determine what gets imported when one does from module import *. Now only the stuff define in each module is exported (as opposed to other imports as well). For example, after this PR, manim will no longer export references to odd libraries like re, os, sys, minidom, inspect, etc.
After this PR, the number of names exported by manim is reduced to 501. Look at the bottom to see a list of names that are no longer being exported.
List of Changes

Added __all__ variables to almost all modules.
Had to fix a couple of things in __init__ as well

Explanation for Changes
PEP8 specifies that the all variable should be put right after the module docstring and before any imports.
Testing Status
Tests pass locally
Further Comments
There is much much more that can be done in order to clean our imports, but I think this is a good first step. For example, do we really need to expose SceneFileWriter to the user? I don't think so... but let's leave that for another PR maybe.
Acknowledgement

[x] I have read the Contributing Guidelines

Names that are no longer being exported
There are 52 names that manim will no longer export. Here is a complete list of all of them, as well as the file were they are defined (where applicable).
animation.py
'DEFAULT_ANIMATION_RUN_TIME'
'DEFAULT_ANIMATION_LAG_RATIO'
composition.py
'DEFAULT_LAGGED_START_LAG_RATIO'
fading.py
'DEFAULT_FADE_LAG_RATIO'
geometry.py
'DEFAULT_DOT_RADIUS'
'DEFAULT_SMALL_DOT_RADIUS'
'DEFAULT_DASH_LENGTH'
'DEFAULT_ARROW_TIP_LENGTH'
matrix.py
'VECTOR_LABEL_SCALE_FACTOR'
probability.py
'EPSILON'
tex_mobject.py
'TEX_MOB_SCALE_FACTOR'
text_mobject.py
'TEXT_MOB_SCALE_FACTOR'
vector_field.py
'DEFAULT_SCALAR_FIELD_COLORS'
vector_space_scene.py
'X_COLOR'
'Y_COLOR'
'Z_COLOR'
bezier.py
'CLOSED_THRESHOLD'
paths.py
'STRAIGHT_PATH_THRESHOLD'
simple_functions.py
'CHOOSE_CACHE'
tex_mobject.py
'tex_to_svg_file'
libraries
os
re
sys
random
math
linalg
it
inspect
minidom
op
sp
string
time
warnings
HtmlFormatter
hashlib
highlight
numbers
copy
deepcopy
cairo
html
functions/classes imported from libraries
reduce
pdist
platform
Image
Color
ProgressDisplay
get_lexer_by_name

What is it with that "format" test failing?

What is it with that "format" test failing?

Read the discussion here #362


(Local) Agent autoregistering with specified environment

Hi,
When I use the LWRP the creation of local Go agents, the environments parameter doesn't seem to do anything.

Here's an example of my LWRP implementation:
for i in 0..(node['wrapper_gocd']['agents'] -1)
  name = "go-agent-#{i}"
  gocd_agent name do
    go_server_host 'localhost'
    go_server_port 8153
    daemon true
    vnc    true
    autoregister_key
    autoregister_hostname name
    environments "Alpha"
  end
end

I've tried a few different inputs besides a string, such as an array, or referencing an attribute like #{node['wrapper_gocd']['go_env']}.
I haven't tried with remote agents, but I don't see any issues raised for that, so could this be an issue exclusive to remote agent creation or am I missing something (most likely)?
Thanks,
Raid

Hi, it may be a bug.
You can pass a string or array of strings, that should not be an issue.

the environments parameter doesn't seem to do anything.

Could you check the contents of autoregister.properties file in your agent's directory? It should have something like
agent.auto.register.environments = Alpha

Also do you have environment created in Go server?

Hey @tomzo
There's no autoregister.properties actually created in my agent's directory. The agent registers to the server just fine, however.
I ran into this when testing the Go backup/restore functionality, so the environments are created as part of the restore. The agents are created after the restore.


This file is unreadable error (Sabaki)

I downloaded a random game from GoKifu (test b.sgf) and KGS (test a and test c.sgf).
Test b-analyzed and test c-analyzed opens successfully in Sabaki and q5go!
but
Test a-analyzed cannot be opended in Sabaki (This file is unreadable error) and q5go (Erros found in SGF file).
I think because Test a is a unfinished game?
Attachments:
test.zip
And a extra question. How do i update analyze.sgf? And how do I check if there is an update?
Thank you in advance!

I will check the files soon.
And try npm outdate, you can check if analyze-sgf is outdate, and if so, simply type npm install -g analyze-sgf.

Fixed (63f5c8e). That was a bug.
Please type again npm install -g analyze-sgf.
Thank you!

The bug is fixed!
With the new version the analyzed version of test c (not test a) opens now in Sabaki and q5go.


bug: cannot install on RHEL 8.6

Summary
The script complains that it requires libopenblas to install the GGML plugin on RHEL 8.6.
Reproduction steps

Run the run-llm.sh script on a CPU-only RHEL 8.6 machine.
See the error message.

Screenshots
No response
Any logs you want to share for showing the specific issue
The command and error
# bash <(curl -sSfL 'https://code.flows.network/webhook/iwYN1SdN3AmPgR5ao5Gt/run-llm.sh')
Checking the operating system (macOS and Linux supported) ...

Checking prerequisites ...
'libopenblas' is required for wasi-nn plugin to run.

The system info
# cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.6 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.6"

Model Information
N/A
Operating system information
RHEL 8.6
ARCH
amd64
CPU Information
Intel on HP machines
Memory Size
N/A
GPU Information
N/A
VRAM Size
N/A

@juntao
Could you try to remove the following section and rerun it again?
https://github.com/second-state/LlamaEdge/blob/main/run-llm.sh#L49-L55
We no longer ship any built-in openblas feature in the default assets.

This is fixed by https://github.com/second-state/LlamaEdge/pull/77


Poetry

Would it be an option to use poetry for version control of all the used packages ?
The requierents.txt isn't exhaustive as of now. We could make a script to have it update automatically, but poetry makes the package management and addition of new packages very easy.
With 3 terminal commands we can

add new packages.
update all packages
synchronize your local venv with the list of requirements

Yes, the requierment.txt is not used currently, it should be removed from the project.
The project is supposed to use Hatch, with is somewhat similar but different from Poetry.
But recently, I thought of using only UV instead of hatch.
https://docs.astral.sh/uv/
The last release of UV is beast. It does all we need.
For more discussion on this topic, I recommend this article : https://alpopkes.com/posts/python/packaging_tools/


add e2e test: alluxioruntime resource setting

Signed-off-by: wangshulin<EMAIL_ADDRESS>
Ⅰ. Describe what this PR does
add e2e test 8-设置AlluxioRuntime各组件Pod的资源需求 in https://github.com/fluid-cloudnative/community/blob/master/testcase/fluid-testcase.md
Ⅱ. Does this pull request fix one issue?

fixes #XXXX
Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

/test fluid-e2e

/lgtm
/approve


do_package_write_deb non-existent task

Hi, I am trying to build meta-scikit-learn with the following config:
BB_VERSION           = "1.44.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "ubuntu-18.04"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "imx8mqevk"
DISTRO               = "fsl-imx-wayland"
DISTRO_VERSION       = "5.4-zeus"
TUNE_FEATURES        = "aarch64 cortexa53 crc crypto"
TARGET_FPU           = ""
meta
meta-poky            = "HEAD:d88d62c20d7d8da85f02edb170dae0280624ad7e"
meta-oe
meta-multimedia
meta-python          = "HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b"
meta-freescale       = "HEAD:14f1a630a47375432f93c556927b879b51d84c4e"
meta-freescale-3rdparty = "HEAD:dbcc686f52c3c84db8cb86aa8973a4e373651b98"
meta-freescale-distro = "HEAD:ca27d12e4964d1336e662bcc60184bbff526c857"
meta-aws             = "HEAD:4dcf78148d951339699b7650319fafda3ce0c3fc"
meta-virtualization  = "zeus:48ec1b49871fef7d331ac1d0680a48d362c7482b"
meta-scikit-learn    = "WIP:39031e8e706bdb57e35d1b85a9fb6f7da3ae61b1"
meta-bsp
meta-sdk
meta-ml              = "HEAD:c4620c1b711d180a3f93d69ec6ca416d886f9277"
meta-nxp-demo-experience = "HEAD:c7263d9f3cc7bbf44e7164ffeda494cf283d3dec"
meta-browser         = "HEAD:ee3be3b5986a4aa0e73df2204a625ae1fe5df37e"
meta-rust            = "HEAD:53bfa324891966a2daf5d36dc13d4a43725aebed"
meta-clang           = "HEAD:711e593d5984aad3bf35c51b7ac4482982bc16c7"
meta-gnome
meta-networking
meta-filesystems     = "HEAD:2b5dd1eb81cd08bc065bc76125f2856e9383e98b"
meta-qt5             = "HEAD:21ce4c124d9a972d9122f87c64ac2773bf04c284"
meta-python2         = "HEAD:4400f9155ec193d028208cf0c66aeed2ba2b00ab"

I know this package is supposed to be supported on gatesgarth but I tried zeus just for my purposes. Ran into the following error during compilation:
ERROR: Task do_populate_sdk in /home/ubuntu/environment/imx-yocto-bsp/sources/meta-imx/meta-sdk/dynamic-layers/qt5-layer/recipes-fsl/images/imx-image-full.bb rdepends upon non-existent task do_package_write_deb in /home/ubuntu/environment/imx-yocto-bsp/sources/meta-scikit-learn/recipes-devtools/python/python3-scipy-native_1.5.2.bb
ERROR: Command execution failed: 1

Any hints on what to check?

Maybe try drop native-sdk from recipe scikit-learn-native?

Maybe try drop native-sdk from recipe scikit-learn-native?

Hi @dhwalters423
I've just pushed a new -native recipe from @nandra, this will hopefully go upstream to meta-scipy and we can switch to using the meta-scipy layer as a dependency. Until then, see if this new recipe works better for you.

Hi @dhwalters423
I've just pushed a new -native recipe from @nandra, this will hopefully go upstream to meta-scipy and we can switch to using the meta-scipy layer as a dependency. Until then, see if this new recipe works better for you.

Hi @nandra and @jackmitch
Thanks for the quick response. I am still running into issues compiling after using the new scipy-native recipe.
ERROR: python3-scikit-learn-0.24.1-r0 do_compile: Execution of '/home/ubuntu/environment/imx-yocto-bsp/build-dir/tmp/work/aarch64-poky-linux/python3-scikit-learn/0.24.1-r0/temp/run.do_compile.23472' failed with exit code 1:
cp: cannot stat '/home/ubuntu/environment/imx-yocto-bsp/build-dir/tmp/work/aarch64-poky-linux/python3-scikit-learn/0.24.1-r0/recipe-sysroot/usr/lib/python3.9/site-packages/numpy/core/lib/npy-pkg-config/*': No such file or directory
WARNING: exit code 1 from a shell command.

ERROR: Logfile of failure stored in: /home/ubuntu/environment/imx-yocto-bsp/build-dir/tmp/work/aarch64-poky-linux/python3-scikit-learn/0.24.1-r0/temp/log.do_compile.23472
Log data follows:
| DEBUG: Executing shell function do_compile
| cp: cannot stat '/home/ubuntu/environment/imx-yocto-bsp/build-dir/tmp/work/aarch64-poky-linux/python3-scikit-learn/0.24.1-r0/recipe-sysroot/usr/lib/python3.9/site-packages/numpy/core/lib/npy-pkg-config/*': No such file or directory
| WARNING: exit code 1 from a shell command.
| ERROR: Execution of '/home/ubuntu/environment/imx-yocto-bsp/build-dir/tmp/work/aarch64-poky-linux/python3-scikit-learn/0.24.1-r0/temp/run.do_compile.23472' failed with exit code 1:
| cp: cannot stat '/home/ubuntu/environment/imx-yocto-bsp/build-dir/tmp/work/aarch64-poky-linux/python3-scikit-learn/0.24.1-r0/recipe-sysroot/usr/lib/python3.9/site-packages/numpy/core/lib/npy-pkg-config/*': No such file or directory
| WARNING: exit code 1 from a shell command.
|
ERROR: Task (/home/ubuntu/environment/imx-yocto-bsp/sources/meta-scikit-learn/recipes-devtools/python/python3-scikit-learn_0.24.1.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 8566 tasks of which 802 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /home/ubuntu/environment/imx-yocto-bsp/sources/meta-scikit-learn/recipes-devtools/python/python3-scikit-learn_0.24.1.bb:do_compile
Summary: There were 8 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

I also tried compiling without scipy-native to the same result.

Hi @nandra and @jackmitch
Thanks for the quick response. I am still running into issues compiling after using the new scipy-native recipe.
ERROR: python3-scikit-learn-0.24.1-r0 do_compile: Execution of '/home/ubuntu/environment/imx-yocto-bsp/build-dir/tmp/work/aarch64-poky-linux/python3-scikit-learn/0.24.1-r0/temp/run.do_compile.23472' failed with exit code 1:
cp: cannot stat '/home/ubuntu/environment/imx-yocto-bsp/build-dir/tmp/work/aarch64-poky-linux/python3-scikit-learn/0.24.1-r0/recipe-sysroot/usr/lib/python3.9/site-packages/numpy/core/lib/npy-pkg-config/*': No such file or directory
WARNING: exit code 1 from a shell command.

ERROR: Logfile of failure stored in: /home/ubuntu/environment/imx-yocto-bsp/build-dir/tmp/work/aarch64-poky-linux/python3-scikit-learn/0.24.1-r0/temp/log.do_compile.23472
Log data follows:
| DEBUG: Executing shell function do_compile
| cp: cannot stat '/home/ubuntu/environment/imx-yocto-bsp/build-dir/tmp/work/aarch64-poky-linux/python3-scikit-learn/0.24.1-r0/recipe-sysroot/usr/lib/python3.9/site-packages/numpy/core/lib/npy-pkg-config/*': No such file or directory
| WARNING: exit code 1 from a shell command.
| ERROR: Execution of '/home/ubuntu/environment/imx-yocto-bsp/build-dir/tmp/work/aarch64-poky-linux/python3-scikit-learn/0.24.1-r0/temp/run.do_compile.23472' failed with exit code 1:
| cp: cannot stat '/home/ubuntu/environment/imx-yocto-bsp/build-dir/tmp/work/aarch64-poky-linux/python3-scikit-learn/0.24.1-r0/recipe-sysroot/usr/lib/python3.9/site-packages/numpy/core/lib/npy-pkg-config/*': No such file or directory
| WARNING: exit code 1 from a shell command.
|
ERROR: Task (/home/ubuntu/environment/imx-yocto-bsp/sources/meta-scikit-learn/recipes-devtools/python/python3-scikit-learn_0.24.1.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 8566 tasks of which 802 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /home/ubuntu/environment/imx-yocto-bsp/sources/meta-scikit-learn/recipes-devtools/python/python3-scikit-learn_0.24.1.bb:do_compile
Summary: There were 8 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

I also tried compiling without scipy-native to the same result.

Hi David, looks like you need to take a poke about in the sysroot and see why these files don't exist.
/home/ubuntu/environment/imx-yocto-bsp/build-dir/tmp/work/aarch64-poky-linux/python3-scikit-learn/0.24.1-r0/recipe-sysroot/usr/lib/python3.9/site-packages/numpy/

as missing those files is what's causing this error:
cp: cannot stat '/home/ubuntu/environment/imx-yocto-bsp/build-dir/tmp/work/aarch64-poky-linux/python3-scikit-learn/0.24.1-r0/recipe-sysroot/usr/lib/python3.9/site-packages/numpy/core/lib/npy-pkg-config/*': No such file or directory

I would ensure that your numpy is new enough to ship these files as they were only added in a recentish version. You could manually update the numpy version in your build. If that works then I would be happy to branch off and carry an updated numpy for Zeus.

Hi David, looks like you need to take a poke about in the sysroot and see why these files don't exist.
/home/ubuntu/environment/imx-yocto-bsp/build-dir/tmp/work/aarch64-poky-linux/python3-scikit-learn/0.24.1-r0/recipe-sysroot/usr/lib/python3.9/site-packages/numpy/

as missing those files is what's causing this error:
cp: cannot stat '/home/ubuntu/environment/imx-yocto-bsp/build-dir/tmp/work/aarch64-poky-linux/python3-scikit-learn/0.24.1-r0/recipe-sysroot/usr/lib/python3.9/site-packages/numpy/core/lib/npy-pkg-config/*': No such file or directory

I would ensure that your numpy is new enough to ship these files as they were only added in a recentish version. You could manually update the numpy version in your build. If that works then I would be happy to branch off and carry an updated numpy for Zeus.


API Module - Path

Is your feature request related to a problem? Please describe.
As a user, I want to signup to a path (like a language in Duolingo). This path should progress me from entry level lessons to advanced lessons of increasing difficulty as I progress.
Describe the solution you'd like

A Path entity in the API that has a many to many relationship to User
A Path should have:

name Name of the path (EG: Javascript, HTML, etc)
icon Link to icon of the path
description Short paragraph explaining the Path
modules All Module (Not completed) entities that belong to the path

@raysercast1 Pull this branch and have a look https://github.com/code-mentoring/learn/tree/feature/path, we can talk about it later today. I encourage you to add update and delete path.


Table not identified

https://www.sec.gov/Archives/edgar/data/1675149/000119312518236766/d828236d10q.htm
https://discord.com/channels/1164249739836018698/1164622878147346552/1173250016668823605
https://parser.alphanome.app/?p=view_parsed&view_parsed=semantic_elements&q=1675149%2F000119312518236766

Fixed as part of other bug fixing. Will include this document to end-to-end testing as it contains more a unique structure


Updates to Logging

[x] Exposes the log channel via settings.
It defaults to the shared instance, but framework consumers will be able to implement their own LogChannel and set it using:
Log.channel = mySpecialChannel

[x] Removes system log (which used a private log writer)
This will allow framework consumers to re-direct all logging through a central log channel/writer/formatter as they wish. This means, that framework consumers can use the default Log.Channel instance, but could set a special log writer, for example:
Log.writer = mySpecialLogWriter

which is equivalent of:
Log.channel.writer = mySpecialLogWriter

[x] For platforms which support OSLog
Added an API to use the default LogWriter (Log.Writers.OSLogWriter) but with a given instance of OSLog - this will allow framework consumers to setup the log using their own subsystem and category. Unfortunately, at this time, we can only use a single OSLog instance. It is recommended that app developers create a OSLog instance dedicated for ProcedureKit and use their bundle ID as the subsystem, and "ProcedureKit" as the category.

Created following discussions in #899
So, for example, it's now possible to do this:
Log.writer = Log.Writers.Redirecting(writers: [
    Log.Writers.OSLogWriter(log: myOSLog),
    remoteCloudLogger])

@danthorpe are devs able or supposed to use Log facilitates outside of Operations?

Well, when I initially refactored the logger, I wanted ProcedureKit contributors to be able to get logs on how the framework works, which would be separate from the logger that framework consumers might want to use for their own purposes.
But to be honest, following discussions in #899 - it's kinda easier to put everything into a single logger really.


GH-7: Add some data processing Groovy deps

Fixes spring-cloud-stream-app-starters/scriptable-transform#7

Reformat POMs for better readability and fix types
Some of Groovy deps are commented out because no presence
in Spring Boot yet

We can consider to include an explicit Groovy version property into our POM to uncomment those extra deps since there is no them in Spring Boot BOM.
@Haybu,
would be great if you could build this and test your script without any @Grab.

PR Works well without @Grab. Thanks @artembilan

Good, but I think we need to hold this PR until Spring Boot resolution in the https://github.com/spring-projects/spring-boot/issues/15468. So, we will be able to have more useful Groovy deps and still don't manage its version ourselves.
Then I'll back-port the fix too other Groovy-based projects here.
Nevertheless it is fully not related to your work with an ivy dependency and test with @Grab.

So, Spring Boot issue for Groovy dependencies has been fixed.
Therefore this PR is ready for review and merge.
Thanks

rebased and merged at: a761a403137312cd8a4d0a88601cd6354e5ded06


Crawling fails if for new java language lavels

For projects that use new Jaca features like try-with-resources, crawling fails with:
Variable resource not allowed here for source level below 9 at /home/jakob/exasol/project-keeper-maven-plugin/src/main/java/com/exasol/projectkeeper/validators/files/ProjectFilesValidator.java:69

dublicate

dublicate


add Forms and Counter

Please have a look :) I added the new form functionality and last the textcounter

Sieht richtig gut aus!


Add Gloss

Adds Gloss to SwiftPM Library
Great project! (And newsletters) 😄

Thanks!

i couldve sworn i put that in alphabetically - whoops!


Use dynamic path for plugin loading

.flowconfig edited to support dynamic path loading without flow errors.

Is this creating the path to the plugin index file in the PlanConfig ?

 It's added here : lib/ExecutionControlEpic/DevtoolLoadingFeature/Model/DevtoolLoadingHandler.js:22


Release Mutation Observer version?

Hey,
When do you think you will be able to release the last version of the script (using the Mutation Observer API)?
It would make audit tools happy and stop warning in console, so it would be really nice.

Hey, sorry for such a long overdue. You can check 2.3.0 - it contains the update :)

+1
Here a great example with aos:
https://codepen.io/artursopelnik/pen/qYwxeQ


WIP - Property collectors

So this is a bit rough and ready due to #40.
As you can see from the commit history, I did 7bb7c31 a week ago. I hadn't realised this wasn't part of v0.3.0 so this is a bigger change than anticipated.
The current code does a _vmware_get_obj on one thread and then for each result it spawns another thread. Behind the pyvmomi magic, each attribute access on a new ManagedObject triggers an API call. So with 100 vms, thats 101 API calls.
The PropertyCollector API is a lot more sane - you can tell it what kind of object to retrieve, where to retrieve it from and what properties of that object to retrieve. It retrieves it all in one go. This is fast - in my environment its as fast as the current version but doesn't need any child threads and only quries a single API call.
I did this for the _vmware_get_vm_metrics code path already, and for #40 i've expanded it for the datastore and host code paths.

Can you add 3e04e22 to this PR and install precommit? precommit does neat stuffs :)?
pre-commit install -f && pre-commit install --install-hooks -t commit-msg -f

This incorporates the changes requested.
precommit does not work for me.

I'm out christmas shopping and would like to do a thorough review of this when I get back. Sorry I've been disconnected a bit lately.

Tests on my end look really good, the performance improvement is stellar! down from ~20s to ~5s per scrape on a ~400 VMs environment.
@Jc2k really impressive!
My only issue with this change is the buffered stdout output, make logs kind of useless...

2.5s per scrape! Nice! I think I might be able to get a bit more out of it - if I can make prop collectors work for the inventory stage. But one thing at a time..

Went to merge this and it still looks like we are missing some metrics. I need to do a bit more testing

It looks like just these now. I had a cached build which had more stuff missing. ``` # HELP vmware_vm_disk_usage_average vmware_vm_disk_usage_average
TYPE vmware_vm_disk_usage_average gauge
HELP vmware_vm_disk_read_average vmware_vm_disk_read_average
TYPE vmware_vm_disk_read_average gauge
HELP vmware_vm_disk_write_average vmware_vm_disk_write_average
TYPE vmware_vm_disk_write_average gauge ```

Curious, I checked and I have these metrics for all my VMs.


[BUG] Removing _index field by remove ingest processor causes OpenSearch to crash

Describe the bug
When removing the _index field by remove ingest processor, the OpenSearch cluster crashes with a failed assertion.
To Reproduce
Steps to reproduce the behavior:

Create a ingest pipeline

PUT _ingest/pipeline/test1
{
  "processors": [
      {
        "remove": {
          "field": "_index"
        }
      }
    ]
}

Add a doc

POST test2/_doc/1?pipeline=test1
{
   "foo1": "bar"
}

you can see that OpenSearch cluster crashed with the following exception:
[2023-10-19T11:11:14,602][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] fatal error in thread [opensearch[node-1][write][T#2]], exiting
java.lang.AssertionError: null
	at org.opensearch.ingest.IngestService.executePipelines(IngestService.java:671) ~[classes/:?]
	at org.opensearch.ingest.IngestService$3.doRun(IngestService.java:560) ~[classes/:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) ~[classes/:?]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[classes/:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
fatal error in thread [opensearch[node-1][write][T#2]], exiting

Expected behavior
OpenSearch can handle this case correctly, the metadata fields such as _index, _id and others could not be removed by remove ingest processor.
Host/Environment (please complete the following information):

OS: [macOS]
Version [2.9]

Removing _version field causes null_pointer_exception:
Request:
PUT _ingest/pipeline/test1
{
  "processors": [
      {
        "remove": {
          "field": "_version"
        }
      }
    ]
}

Response:
{
  "error": {
    "root_cause": [
      {
        "type": "null_pointer_exception",
        "reason": "Cannot invoke \"java.lang.Number.longValue()\" because the return value of \"java.util.Map.get(Object)\" is null"
      }
    ],
    "type": "null_pointer_exception",
    "reason": "Cannot invoke \"java.lang.Number.longValue()\" because the return value of \"java.util.Map.get(Object)\" is null"
  },
  "status": 500
}

In most time remove _id by remove ingest processor works well, but if we pass version and version_type parameters when calling index API, removing _id will also cause OpenSearch to crash.
PUT _ingest/pipeline/test
{
  "processors": [
      {
        "remove": {
          "field": "_id"
        }
      }
    ]
}
POST test_123/_doc/1?version=1&version_type=external&pipeline=test2
{
  "foo.x":1
}

, then you can see the following error:
[2023-10-24T16:01:40,373][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] fatal error in thread [opensearch[node-1][write][T#4]], exiting
java.lang.AssertionError: version: 1 type: EXTERNAL
	at org.opensearch.index.engine.InternalEngine.assertPrimaryCanOptimizeAddDocument(InternalEngine.java:801) ~[classes/:?]
	at org.opensearch.index.engine.InternalEngine.canOptimizeAddDocument(InternalEngine.java:779) ~[classes/:?]
	at org.opensearch.index.engine.InternalEngine.planIndexingAsPrimary(InternalEngine.java:1041) ~[classes/:?]
	at org.opensearch.index.engine.InternalEngine.indexingStrategyForOperation(InternalEngine.java:1029) ~[classes/:?]
	at org.opensearch.index.engine.InternalEngine.index(InternalEngine.java:883) ~[classes/:?]
	at org.opensearch.index.shard.IndexShard.index(IndexShard.java:1118) ~[classes/:?]
	at org.opensearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1063) ~[classes/:?]
	at org.opensearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:954) ~[classes/:?]
	at org.opensearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:622) ~[classes/:?]
	at org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:469) ~[classes/:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) ~[classes/:?]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[classes/:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]

Closed by https://github.com/opensearch-project/OpenSearch/pull/10895.


Fix box-sizing for shadow-dom elements

As discussed on Discord, this PR changes the way we reset box-sizing in our base/preflight styles.
Context
Previously we were setting it on <html> and letting everyone else inherit from there. Sadly, this approach doesn't take into account elements dependent on the shadow-dom.
Since we don't have control over those elements, we can only do one of this:

Explicitly set box-sizing for every element and pseudo-element
Target the shadow-<div> inside <details>

Second approach would make this change safer but require us to keep adding things to our reset as users encounter new issues with other existing or future elements. PR implements the first option as it was also the one chosen at cssremedy.
Hopefully nothing breaks because of this change 🤞

@adamwathan could you take a look at this for v1.2.0? Let me know if you need me to change/research/test anything before merging. I'm not as afk as I was last week :D

Seems good, thanks!


JSmin/JSShrink confusion?

I'm using Minify as a library as a way to provide minification of inline HTML/CSS/JS for Craft CMS via Twig.
I'm attempting to upgrade my plugin to use Minify 3.x, and I'm slightly confused by JSMin/JSShrink.
It appears that JSMin has been removed: https://github.com/mrclay/minify/blob/4626f6c1e002f7129e826448be43566296c00c98/HISTORY.md#version-300-2017-04-03
And I can see the JSShrink class referenced in the autoload_classmap.php:
    'Minify\\JS\\JShrink' => $vendorDir . '/mrclay/minify/lib/Minify/JS/JShrink.php',

But the package tedivm/jshrink is only listed under require-dev and the (I thought deprecated) mrclay/jsmin-php is listed under require... and yet nowhere in the autoload_classmap.php do I see JSMin referenced.
So... if I want to minify JavaScript directly, should I be using the bundled JSMin (which strangely isn't auto-loaded) or should I be including JSShink in my composer.json and using that?
If I do the latter, everything seems to work... it's just that something seems off here?

Run composer install to update your vendors.
The old JSMin was replaced by JSMin\JSMin, provided by the mrclay/jsmin-php package.

hmmm. I did a fresh composer install after deleting composer.lock as well as the entire vendor directory.
It did include mrclay/jsmin-php in the actual vendor folder, but it's not auto-loaded (it doesn't appear anywhere in the autoload_classmap.php
Is that expected behavior?

Yes it's autoloaded. It's in vendor/composer/autoload_namespaces.php.

ugh, sorry for that. I can't believe I overlooked autoload_namespaces.php


Multiple routes having same component

If same component is attached to multiple routes. Then if it navigates from one route to other (Both routes having same component), attached hook is not called. Same for history back.

It's expected behavior since the component is reused in-place, and not destroyed and created and attached again.
Checkout router's data, activate hooks, they should be what you need.
If you do want it to be called, use
route: {canReuse: false}


BUG- access role selector

BUG Access role selector
Issue Number(s): #160 .
What does this PR change and why?
Update access management role selector styling to realign buttons

Checklist

[ ] Requirements have been implemented
[ ] Acceptance criteria are met
[ ] Database schema docs have been updated or are not necessary
[ ] Code follows design and style guidelines
[ ] Commits follow guidelines (concise, squashed, etc)
[x] Relevant reviewers (Senior Dev/EM/Designers) have been assigned to this PR

Critical Changes

Database change/migration to run
Environment config change
Breaking API change

Related PRs

#number_of_pr

Testing
Enumerate steps to test the functionality of your ticket. This should include edge cases and testing of error handling, if applicable.

Resolved by #304


Restic fixes: pruning, process substitution

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Fits CONTRIBUTING.md.

I think it would be nice to keep the options mostly in line with the borgbackup module. Have a look at it and see if you find some inspiration.

Okay, I took another stab at this, see af4bfd178e9.

0a2c4f2 uses RuntimeDirectory instead.

Hello! I'm very interested in this. What is the state of this PR?

Sorry, I'd left a bunch of the discussion unresolved.  I just re-read everything in light of the current tip of the restic-fixes branch (00976b194fe32d3e425ff3b457eeb6e4ec7ba12c), and I think all the comments have been addressed.
Well, except "keep the options mostly in line with the borgbackup module".  Perhaps I looked at that back when I was working on this, I just don't recall.  :)
In any event, I've been running with these changes locally for a long time now, I haven't run into any issues with them.

That's good to hear! I hope @Mic92 will be able to take another look at it sometime 🙂

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/126

superseeded by https://github.com/NixOS/nixpkgs/pull/78886


Output of value from a MatchInfo type object changes the following output to terminal

Prerequisites

[X] Write a descriptive title.
[X] Make sure you are able to repro it on the latest released version
[X] Search the existing issues.
[X] Refer to the FAQ.
[X] Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce
The base_select var is of type MatchInfo. Running the below code will output the values of the vars, and ALL type properties. Comment out line 4 (#$base_select) and run, and the type info output is the common properties, not all.
$base = cmd.exe /c set
$base_select = cmd.exe /c set | Select-String -SimpleMatch "LOGONSERVER"
$base
$base_select
$base.GetType()
$base_select.GetType()
Expected behavior
IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     Object[]                                 System.Array
True     False    MatchInfo                                System.Object

Actual behavior
Exploded type info for lines 5 and 6 when only common properties expected.

Error details
No response
Environment data
Tested with ver 5 and 7. VSCode and ISE.

Visuals
No response

Expected behaviour. If one command outputs two kinds of object the PowerShell formatter doesn't know how to format the second object
try
 $base_select.GetType(); $base_select

$base_select;  $base_select.GetType();

Thank you @jhoneill. I seem to remember that caveat from years ago now.


KEYCLOAK-10864 Pass URI path upstream verbatim

Fixes KEYCLOAK-10864 which will also fix KEYCLOAK-11276 and KEYCLOAK-13315
Closes #528
Closes #483 by superseding it as a bug fix, not an option

No documentation needed, as this is a bug fix.
See discussion on Keycloak Dev but please add comments here.

@abstractj Please consider this PR as an improvement over #483 (details on the dev list)

If this PR is using the original path when invoking upstream, while using the normalized path when applying permission checks, then I'm in favour of this approach.

If this PR is using the original path when invoking upstream, while using the normalized path when applying permission checks, then I'm in favour of this approach.

@stianst That is it exactly.

+1 On this approach from me - haven't looked at the code though

@Nuru thank you so much for submitting this and apologies if it took longer than expected. Your PR was superseded by https://github.com/louketo/louketo-proxy/pull/626, which is pretty much a rebase of your changes.
If you have any questions, please let me know.


Add placeholders for audio chapters

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


Added new dungeon

Added a new dungeon, a castle that the player can access from 3-1.
It has three rooms initially along with multiple floors. Second floor is still in progress.
Let me know if everything is okay before I continue working on this.
Fixes #103

I like it. You have a special skill of storytelling which I lack.
Add some emojis to the story! 😃 👍

One more important thing I noticed. Your PR has some merge conflicts. The reason is because there is already a floor named 7 in the normal-dungeon, and that was added by me in the latest PR #108. Maybe the reason you didn't notice it is because you did not make your fork up to date.
Maybe renaming the folder name might fix the issue, I think. @ming-tsai, any other ways to fix this?

One more important thing I noticed. Your PR has some merge conflicts. The reason is because there is already a floor named 7 in the normal-dungeon, and that was added by me in the latest PR #108. Maybe the reason you didn't notice it is because you did not make your fork up to date.
Maybe renaming the folder name might fix the issue, I think. @ming-tsai, any other ways to fix this?

So THAT's what it is! I was wondering what was causing that!
That's weird because I made sure to make a pull before I started working; I must've accidentally pulled from my fork instead of the main branch.
I'll try renaming the files and seeing if that works!

Yes @paulsonjpaul

One more important thing I noticed. Your PR has some merge conflicts. The reason is because there is already a floor named 7 in the normal-dungeon, and that was added by me in the latest PR #108. Maybe the reason you didn't notice it is because you did not make your fork up to date.
Maybe renaming the folder name might fix the issue, I think. @ming-tsai, any other ways to fix this?

So THAT's what it is! I was wondering what was causing that!
That's weird because I made sure to make a pull before I started working; I must've accidentally pulled from my fork instead of the main branch.
I'll try renaming the files and seeing if that works!

If you want to make your fork up to date, this video will help:
https://www.youtube.com/watch?v=deEYHVpE1c8&t=26s

One more important thing I noticed. Your PR has some merge conflicts. The reason is because there is already a floor named 7 in the normal-dungeon, and that was added by me in the latest PR #108. Maybe the reason you didn't notice it is because you did not make your fork up to date.
Maybe renaming the folder name might fix the issue, I think. @ming-tsai, any other ways to fix this?

So THAT's what it is! I was wondering what was causing that!
That's weird because I made sure to make a pull before I started working; I must've accidentally pulled from my fork instead of the main branch.
I'll try renaming the files and seeing if that works!

If you want to make your fork up to date, this video will help:
https://www.youtube.com/watch?v=deEYHVpE1c8&t=26s

Thanks for the link; think I have everything properly synced now

Hi @FukurouMakoto, are you merging correctly? Because some of your file has this
<======== head
...
=============

This normal show when you have merging conflict on the file and don't solve it, maybe you could move your part to another floor

Hi @FukurouMakoto, are you merging correctly? Because some of your file has this
<======== head
...
=============

This normally is showing when you have merging conflict on the file and doesn't solve it, maybe you could move your part to another floor

To another floor as in rename the folders? I can do that if that's what you mean.

Yes

Hi @FukurouMakoto, are you merging correctly? Because some of your file has this
<======== head
...
=============

This normally is showing when you have merging conflict on the file and doesn't solve it, maybe you could move your part to another floor

To another floor as in rename the folders? I can do that if that's what you mean.

Yes

Hi @FukurouMakoto, are you merging correctly? Because some of your file has this
<======== head
...
=============

This normally is showing when you have merging conflict on the file and doesn't solve it, maybe you could move your part to another floor

To another floor as in rename the folders? I can do that if that's what you mean.

Yes

Okay done. Let me know if that worked.

@FukurouMakoto the next floor is 8 please rename the folder 9 to 8

@FukurouMakoto the next floor is 8 please rename the folder 9 to 8

To make sure, 8 was what the folder was named originally, and was causing the errors before as seen below:

Hi @FukurouMakoto, are you merging correctly? Because some of your file has this
<======== head
...
=============

This normally is showing when you have merging conflict on the file and doesn't solve it, maybe you could move your part to another floor

To another floor as in rename the folders? I can do that if that's what you mean.

Yes

What should I rename it to then?

Ok, keep it on 9, but please delete folder .vscode

I think the .vscode file is created by VS Code Live Server Extension. It had happened to me also.

Ok


Seeing if the README files will work on readthedoc

Please merge #2206 before looking at this change.

This change doesn't work, seems you can't use .. like this.....
/home/docs/checkouts/readthedocs.org/user_builds/verible/checkouts/2207/doc/index.rst:4: WARNING: toctree contains reference to nonexisting document 'README'
/home/docs/checkouts/readthedocs.org/user_builds/verible/checkouts/2207/doc/index.rst:4: WARNING: toctree contains reference to nonexisting document 'common/README'
/home/docs/checkouts/readthedocs.org/user_builds/verible/checkouts/2207/doc/index.rst:4: WARNING: toctree contains reference to nonexisting document 'verilog/tools/lint/README'
/home/docs/checkouts/readthedocs.org/user_builds/verible/checkouts/2207/doc/indexing.md:1: WARNING: Document headings start at H2, not H1 [myst.header]


Compared to the previous version

I noticed that compared to the previous version, the latest code has removed 'multi_grid' from the backbone (resnet) and changed the settings of parameters such as dilations.
May I ask if the adjusted version can improve the performance?

To avoid confusion, we only make these changes on Cityscapes.
We change the dilation from [False, True, True] to [False, False, True]. Actually, this will lower the performance (around 1%), but will significantly speed up the training (around 2x) and require much less GPU memory. As for the multi_grid, we do not think it makes much difference to performance. Our improvements on Cityscapes from axXiv v1 to arXiv v2 (camera-ready version) mainly come from the confidence threshold changed from 0.95 to 0.

Thanks.

And will the latest weights on Cityscapes be made public?

Which weight do you need?

If possible, could you provide latest trained models on the Cityscapes dataset?
I think it will help other researchers quickly reproduce latest results, thanks a lot!

We have provided the checkpoints of our best models for each split and each backbone on Cityscapes and COCO.
Please refer to https://github.com/LiheYoung/UniMatch#cityscapes and https://github.com/LiheYoung/UniMatch#coco.

Thanks a lot!

我们为Cityscapes和COCO上的每个拆分和每个主干提供了我们最好的模型的检查点。您可以点击数字下载相应的检查点。

Unexpected key(s) in state_dict: "model", "optimizer", "epoch", "previous_best".

As you can see in the error information, "Unexpected key(s) in state_dict: "model", "optimizer", "epoch", "previous_best"".
You should load checkpoint['model'], as done in: https://github.com/LiheYoung/UniMatch/blob/a5d2a52223d000d14959b2100d4e0324a4f871e9/unimatch.py#L99

As you can see in the error information, "Unexpected key(s) in state_dict: "model", "optimizer", "epoch", "previous_best"".
You should load checkpoint['model'], as done in:
https://github.com/LiheYoung/UniMatch/blob/a5d2a52223d000d14959b2100d4e0324a4f871e9/unimatch.py#L99

I have done it as you said but then it show this error. How to avoid this key errors?:

As you can see in the error information, "Unexpected key(s) in state_dict: "model", "optimizer", "epoch", "previous_best"".
You should load checkpoint['model'], as done in:
https://github.com/LiheYoung/UniMatch/blob/a5d2a52223d000d14959b2100d4e0324a4f871e9/unimatch.py#L99

I have done it as you said but then it show this error. How to avoid this key errors?:

Hi, you should
model = torch.nn.parallel.DistributedDataParallel(model)


Fixes #46

I tried to keep as much of the original structure as possible while replacing any interfacing with the bucket with API calls. This allowed me to keep the existing in-memory implementation. If this seems janky (i.e. using Stowage and the Amazon SDK - although Stowage is not used for anything to do with Amazon directly), I'm happy to remove the FileStorage layer and have a direct implementation of the SharedFileService - shouldn't be too hard. A few comments have been left in for review

This change is

src/SIL.Machine.AspNetCore/Services/S3FileStorage.cs line 39 at r1 (raw file):

        var response = await _client.ListObjectsV2Async(request, cancellationToken);

        return response.S3Objects.Any();

This will also return objects that are in a "subdirectory" - even though that concept does not exist in S3 buckets.  This is fine.

src/SIL.Machine.AspNetCore/Services/S3FileStorage.cs line 44 at r1 (raw file):

    public async Task<IReadOnlyCollection<IOEntry>> Ls(
        IOPath? path = null,
        bool recurse = false,

recurse now longer does anything.  There is an ability to do something with "delimiter as per here: https://stackoverflow.com/questions/73295517/get-list-of-files-from-an-s3-bucket-but-not-recursively.  Currently, it should work recursively.  The only place this is called is in DeleteAsync - which calls it recursively.  Either rename as LsRecursive and remove the parameter - or restore the delimiter logic (preferred).

src/SIL.Machine.AspNetCore/Services/S3WriteStream.cs line 72 at r1 (raw file):

        catch (Exception)
        {
            Abort();

dotnet 7 is giving a warning - Because this call is not awaited, execution of the current method continues before the call is completed. Consider applying the 'await' operator to the result of the call.  This should be fixed in some way...

src/SIL.Machine.AspNetCore/Services/S3FileStorage.cs line 76 at r1 (raw file):

    public async Task<Stream?> OpenRead(IOPath path, CancellationToken cancellationToken = default)
    {
        var objectId = IOPath.IsRoot(path)

Why are we no longer checking the path for null?

src/SIL.Machine.AspNetCore/Services/S3FileStorage.cs line 86 at r1 (raw file):

    public async Task<Stream> OpenWrite(IOPath path, CancellationToken cancellationToken = default)
    {
        Console.WriteLine("PATH ||| " + path.ToString());

Why no null checking?

src/SIL.Machine.AspNetCore/Services/S3FileStorage.cs line 89 at r1 (raw file):

        var objectId = IOPath.IsRoot(path)
            ? string.Empty
            : IOPath.Normalize(path, removeLeadingSlash: true, appendTrailingSlash: true);

This normalization logic occurs 4 times - can we make it into a function?  Can NormalizeWithBasePath be reintroduced?

src/SIL.Machine.AspNetCore/Services/S3FileStorage.cs line 89 at r1 (raw file):
Previously, johnml1135 (John Lambert) wrote…
This normalization logic occurs 4 times - can we make it into a function?  Can NormalizeWithBasePath be reintroduced?

Or call it PathToObjectId or similar...

src/SIL.Machine.AspNetCore/Services/S3WriteStream.cs line 2 at r1 (raw file):

using Amazon.S3;
using Amazon.S3.Model;

global using...

src/SIL.Machine.AspNetCore/Services/S3WriteStream.cs line 68 at r1 (raw file):

            //     }
            // );

What is this commented code for?  Is it needed or should it be removed?

src/SIL.Machine.AspNetCore/Services/S3WriteStream.cs line 96 at r1 (raw file):

            //     }
            // );

Remove commented code (or otherwise fix).

src/SIL.Machine.AspNetCore/Services/S3WriteStream.cs line 72 at r1 (raw file):
Previously, johnml1135 (John Lambert) wrote…
dotnet 7 is giving a warning - Because this call is not awaited, execution of the current method continues before the call is completed. Consider applying the 'await' operator to the result of the call.  This should be fixed in some way...

add a .wait() with a timeout?  I see that you are trying to not have it be async.

Still working on this: I'll ping you when I'm ready for review.

One quick question and then I can push a final draft @ddaspit @johnml1135 : So I've made in-memory and local storage implementations alongside the S3 Bucket implementation (since Stowage provided both and we had both potentially utilized in the current logic). I wrote some unit tests for both (which are all passing). My question is this: Assuming it's valuable to check in the tests, what directory should I give in the unit tests as a base directory for the local file storage? I.e. a directory I'll have access to and is safe to use. I suppose I can generate a unique one and then get permissions, but I figured there was a right way to do this. Sorry for the delay.

src/SIL.Machine.AspNetCore/Services/S3WriteStream.cs line 68 at r1 (raw file):
Previously, johnml1135 (John Lambert) wrote…
What is this commented code for?  Is it needed or should it be removed?

I was just wondering if you'd like to log (maybe at a low log level: trace, let's say) the progress of file uploads to s3.

src/SIL.Machine.AspNetCore/Services/S3FileStorage.cs line 44 at r1 (raw file):
Previously, johnml1135 (John Lambert) wrote…
recurse now longer does anything.  There is an ability to do something with "delimiter as per here: https://stackoverflow.com/questions/73295517/get-list-of-files-from-an-s3-bucket-but-not-recursively.  Currently, it should work recursively.  The only place this is called is in DeleteAsync - which calls it recursively.  Either rename as LsRecursive and remove the parameter - or restore the delimiter logic (preferred).

You added recuse logic - I am assuming it works.

src/SIL.Machine.AspNetCore/Services/S3FileStorage.cs line 49 at r3 (raw file):

    {
        if (path != null && !path.EndsWith("/"))
            throw new ArgumentException("Path must be a folder", nameof(path));

"a folder and end with a '/'."

src/SIL.Machine.AspNetCore/Services/S3FileStorage.cs line 76 at r1 (raw file):
Previously, johnml1135 (John Lambert) wrote…
Why are we no longer checking the path for null?

Standard normalization

src/SIL.Machine.AspNetCore/Services/S3FileStorage.cs line 86 at r1 (raw file):
Previously, johnml1135 (John Lambert) wrote…
Why no null checking?

Standard normalization - great!

src/SIL.Machine.AspNetCore/Services/S3WriteStream.cs line 72 at r1 (raw file):
Previously, johnml1135 (John Lambert) wrote…
add a .wait() with a timeout?  I see that you are trying to not have it be async.

@ddaspit - is this an appropriate recommendation?  It appears sound to me...

src/SIL.Machine.AspNetCore/Services/S3WriteStream.cs line 68 at r1 (raw file):
Previously, Enkidu93 (Eli C. Lowry) wrote…
I was just wondering if you'd like to log (maybe at a low log level: trace, let's say) the progress of file uploads to s3.

Let's say no for now.  The files should be fairly small - and not take too long.  If they fail, they will be retried a few times.  If they really fail, there will be error logs.
If we start uploading 100MB or larger files, then progress may become important - but I don't see that happening soon.  Serval is not build for big corpora, and the fine tuned models will stay in python land (for now).

src/SIL.Machine.AspNetCore/Services/S3WriteStream.cs line 68 at r1 (raw file):
Previously, johnml1135 (John Lambert) wrote…
Let's say no for now.  The files should be fairly small - and not take too long.  If they fail, they will be retried a few times.  If they really fail, there will be error logs.
If we start uploading 100MB or larger files, then progress may become important - but I don't see that happening soon.  Serval is not build for big corpora, and the fine tuned models will stay in python land (for now).

If we need to pull fine tuned models from S3 buckets to run inferencing and we choose Dotnet rather than python (very plausible), even then, I don't see needing user feedback nor having more logging than we have right now.  so my vote is for removing the comment.  @ddaspit, any thoughts?

My best guess is to just make a tmp directory - https://learn.microsoft.com/en-us/dotnet/api/system.io.directory.createtempsubdirectory?view=net-7.0.  We would have to upgrade to dotnet 7.0 to use it.  @ddaspit  is there any reason we should not?

src/SIL.Machine.AspNetCore/Services/S3FileStorage.cs line 49 at r3 (raw file):
Previously, johnml1135 (John Lambert) wrote…
"a folder and end with a '/'."

Following what there was before, I was conceptualizing a folder as a path that ends with "/"  by definition. The null check with short-circuit evaluation just prevents the second check from failing.

src/SIL.Machine.AspNetCore/Services/S3FileStorage.cs line 49 at r3 (raw file):
Previously, Enkidu93 (Eli C. Lowry) wrote…
Following what there was before, I was conceptualizing a folder as a path that ends with "/"  by definition. The null check with short-circuit evaluation just prevents the second check from failing.

I was thinking about the message in the exception - The path could very well be a folder, but giving the user insight that the trailing slash is needed.

tests/SIL.Machine.AspNetCore.Tests/Services/FileStorageTests.cs line 90 at r3 (raw file):
Previously, johnml1135 (John Lambert) wrote…
Is there some other way to do this?  A few ideas:

Make the test category "do not run" and have it not be commented - and then update the standard launchers and CI builds and don't run them
Either way, there should be some comment explaining why these are commented out/not normally run.

  Will they all pass now on CI?

tests/SIL.Machine.AspNetCore.Tests/Services/FileStorageTests.cs line 90 at r3 (raw file):
Previously, johnml1135 (John Lambert) wrote…
  Will they all pass now on CI?

Ah - temp directory.  It should be all ok now.

Damien's comments are resolved through:

https://github.com/sillsdev/machine/issues/63
https://github.com/sillsdev/machine/issues/64


Atomically delete a message and send another?

The title is self-explanatory, I think. I have the following pseudo-code:
rsmq.deleteMessage({qname:"myqueue", id:message.id}, function(err, resp){
    // the program could crash here and the job would never be finished
    rsmq.sendMessage({qname:"otherqueue", message:"finished-my-job"})
})

How to deal with these issues? I know this might not be directly related to RSMQ, I first thought "BRPOPLPUSH " from redis could help me somehow, but not sure how exactly.

There currently is no way to do this case atomically. You would either have to come up with something yourself, or change the pattern you are using. Like use a helper queue for these cases.

I understand.
Could you elaborate on the "helper queue" pattern? I would appreciate it.

Create a message in a helper queue that you use to finish the above problem and only when it is finished delete the message.

Thanks! I thought about that, but there is still a risk of the message being sent twice or more to the queue if it happens to crash. I'll figure it out.


GCP docs

Hi @mirii1994 , I've resolved the comments, I believe

@nico-shishkin I'm closing this PR because there's an issue with the Netlify preview, i'll open a new one instead.


alloated_ip_range should be optional

TL;DR
allocated_ip_range is required here, but optional in the provider (and the Cloud SQL Admin API)
Expected behavior
The field should be optional.
Observed behavior
The field is required. See https://github.com/terraform-google-modules/terraform-google-sql-db/blob/71b04fc59bce61bf108d012a83fe0ecf3b0922e9/modules/safer_mysql/variables.tf#L52.
Terraform Configuration
n/a

Terraform Version
n/a

Additional information
No response

This issue has been fixed in https://github.com/terraform-google-modules/terraform-google-sql-db/pull/395

Tahnks @ibhaskar2!

Thanks a lot taking care of this @ibhaskar2. I'm going to close this for now.

@g-awmalik has this been released as part of a recent tag?

@enocom - this should be available with v14.0.0. We should be releasing that early next week.

Thank you!


Build Transphoner in Python

Paper's implementation uses Scala: https://github.com/msavva/transphoner
Maybe we can use this Python code for generating the phonemes: https://github.com/xinjli/transphone

Outline of the process:

Their data is available at: https://github.com/msavva/transphoner/tree/master/data

Use ALINE for phonetic similarity

Could also consider scraping wiktionary for all IPA pronunciations: https://en.wiktionary.org/wiki/Category:Indonesian_terms_with_IPA_pronunciation

To measure the similarity between two IPA strings, you could use a string distance metric such as the Levenshtein distance, which is simple to understand and implement. It counts the minimum number of edits (insertions, deletions, substitutions) needed to transform one string into the other.
However, if you want to account for the phonetic similarity between different phonemes, you could consider using a phonetic alignment algorithm like Needleman-Wunsch or Smith-Waterman, with a substitution matrix that reflects the phonetic similarity between different phonemes.
Keep in mind that this is a relatively simple approach and might not account for all the nuances of phonetic similarity. Also, because IPA is a broad phonetic transcription system, the same sound can be transcribed differently depending on the level of detail you want to include, and different languages might use different symbols to represent similar sounds. Therefore, you might need to preprocess your IPA strings to ensure they use a consistent set of symbols.
Here is a simplified example of how you might use the Levenshtein distance to compare two IPA strings in Python, using the python-Levenshtein package:
import Levenshtein as lev

ipa1 = "hɛ.ˈloʊ"  # English "hello"
ipa2 = "hə.ˈlo"  # Hypothetical Indonesian word

distance = lev.distance(ipa1, ipa2)

print(distance)

Remember to install the package with pip install python-Levenshtein if you haven't already.
The output is the minimum number of edits needed to transform ipa1 into ipa2. The lower the number, the more similar the two strings are.

Sure, the Needleman-Wunsch algorithm is a standard algorithm for sequence alignment in bioinformatics, typically used for DNA or protein sequence alignment. The idea is to find the best alignment (including gaps) between two sequences, given a substitution matrix that scores the similarity between each pair of elements.
In the case of IPA strings, you could define a substitution matrix that scores the phonetic similarity between different phonemes. This would be a complex task in itself, as you would need to define a similarity score for each pair of IPA symbols, but let's assume you have such a matrix.
Here's a simple implementation of the Needleman-Wunsch algorithm in Python, adapted to work with strings of IPA symbols:
Copy code
import numpy as np

def needleman_wunsch(seq1, seq2, sub_matrix, gap_penalty=-1):
    """Performs the Needleman-Wunsch sequence alignment."""
    n, m = len(seq1), len(seq2)

    # Initialize score matrix with zeros
    score_matrix = np.zeros((n+1, m+1))

    # Initialize first row and column with gap penalty
    score_matrix[0, :] = np.arange(m+1) * gap_penalty
    score_matrix[:, 0] = np.arange(n+1) * gap_penalty

    # Fill in the rest of the matrix
    for i in range(1, n+1):
        for j in range(1, m+1):
            match = score_matrix[i-1, j-1] + sub_matrix[seq1[i-1], seq2[j-1]]
            delete = score_matrix[i-1, j] + gap_penalty
            insert = score_matrix[i, j-1] + gap_penalty
            score_matrix[i, j] = max(match, delete, insert)

    return score_matrix[n, m]

# Example usage
ipa1 = "hɛ.ˈloʊ"  # English "hello"
ipa2 = "hə.ˈlo"  # Hypothetical Indonesian word

# Define a simple substitution matrix where identical symbols score 1
# and different symbols score -1
sub_matrix = {
    a: {b: 1 if a == b else -1 for b in set(ipa1 + ipa2)}
    for a in set(ipa1 + ipa2)
}

score = needleman_wunsch(ipa1, ipa2, sub_matrix)

print(score)

This code performs a simple global sequence alignment of the two IPA strings, scoring 1 for identical symbols and -1 for different symbols. It also includes a gap penalty of -1. The output is the maximum score of any alignment of the two strings.
Please note that this is a basic example and there are many ways to improve and refine this approach, such as using a more complex substitution matrix, adjusting the gap penalty, or implementing local alignment (Smith-Waterman) instead of global alignment (Needleman-Wunsch). Also, this simple implementation doesn't actually construct the alignment itself, just the score of the optimal alignment. Constructing the alignment would require additional code to backtrack through the score matrix.

nltk's ALINE seems to run into a lot of issues, because some characters do not exist in its feature matrix

Could try https://abydos.readthedocs.io/en/latest/abydos.distance.html#abydos.distance.ALINE and add the newer / missing IPA characters as a dict to it

Could copy source-code of https://abydos.readthedocs.io/en/latest/_modules/abydos/distance/_aline.html#ALINE and add missing characters to it. It also includes c_skip and c_sub which was not found in nltk's implementation

See https://github.com/msavva/transphoner/tree/553c0d5923af1d5055bbccb34d8cbbf0fdd1d8ef/libtransphone/src/main/scala/org/babysherlock/transphoner/phonetics of what to do with c_skip etc.

Rewrite NLTK's or abydos' ALINE code and use scala implementation as guidance. Also make sure all (new) IPA characters are supported, including breaks.

Also include special characters found in https://github.com/msavva/transphoner/blob/master/libtransphone/src/main/scala/org/babysherlock/transphoner/phonetics/Phone.scala#L218

https://www.reddit.com/r/linguistics/comments/ovewb3/is_there_a_tool_that_can_convert_a_language_to/

https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/tts/g2p.html#

https://github.com/lingjzhu/CharsiuG2P
https://huggingface.co/charsiu/g2p_multilingual_byT5_small_100

import panphon.distance
dst = panphon.distance.Distance().feature_edit_distance
# add all English words here
WORDS = ["closing", "cucci", "gnocchi", "kissing"]
closest_word = min(WORDS, key=lambda x: dst("kucing", x))
# cucci
print(closest_word)


LOCALE_ID is always en-US during server side renderring, and inconsistent with browser build

LOCALE_ID is always en-US during server side renderring. And if we run the same prodution build in browser, LOCALE_ID is correct as expect.

Bug Report or Feature Request (mark with an x)
- [x] bug report -> please search issues before submitting
- [ ] feature request

Command (mark with an x)
- [ ] new
- [x] build
- [ ] serve
- [ ] test
- [ ] e2e
- [ ] generate
- [ ] add
- [ ] update
- [ ] lint
- [ ] xi18n
- [ ] run
- [ ] config
- [ ] help
- [ ] version
- [ ] doc

Versions
Angular CLI: 6.1.5
Node: 8.11.2
OS: darwin x64
Angular: 6.1.6
... animations, common, compiler, compiler-cli, core, forms
... http, language-service, platform-browser
... platform-browser-dynamic, platform-server, router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.7.5
@angular-devkit/build-angular     0.7.5
@angular-devkit/build-optimizer   0.7.5
@angular-devkit/build-webpack     0.7.5
@angular-devkit/core              0.7.5
@angular-devkit/schematics        0.7.5
@angular/cli                      6.1.5
@ngtools/webpack                  6.1.5
@schematics/angular               0.7.5
@schematics/update                0.7.5
rxjs                              6.3.1
typescript                        2.9.2
webpack                           4.9.2

Repro steps
Checkout https://github.com/clarkorz/angular6-ssr-issue-locale-fix-to-en
$ git clone git@github.com:clarkorz/angular6-ssr-issue-locale-fix-to-en.git
$ cd angular6-ssr-issue-locale-fix-to-en
$ yarn
$ yarn build:i18n-ssr
$ yarn serve:ssr

Access es endpoint, expect LOCALE_ID to be es.
$ curl http://localhost:4000/es/

...
<app-root ng-version="6.1.6">
Hello

locale is en-US
</app-root>
...

If I run the same prodution build in browser, I got es for LOCALE_ID:
$ yarn serve:browser

Open browser to access http://localhost:5000/es/, I got es in browser and LOCALE_ID is es in console.
The log given by the failure
LOCALE_ID is always en-US during server side renderring.
// in another terminal
$ curl http://localhost:4000/es/

// in serve terminal
Node Express server listening on http://localhost:4000
LOCALE_ID is en-US

Desired functionality
LOCALE_ID  should be as same as the --i18n-locale parameter of ng build or configuration parameter i18nLocale in angular.json.
LOCALE_ID  should be consistent on both server side and browser side.
Mention any other details that might be useful

No.

Hi, from the looks of it you are only generating a single server application https://github.com/clarkorz/angular6-ssr-issue-locale-fix-to-en/blob/master/angular.json#L143, which can only load a single locale.
In your case you have 3 browsers configurations production-es, production-en, production, and a single build for the server being production which rightly so is taking the default locale that being en-US.
I am going to leave this open to add docs related to this.

I sent a PR a few months ago to allow having multiple locales with the same server. It was closed because this use case was supposed to be better supported natively. Is it still planned?

Yeah, with the new renderer the translations will be applied at runtime.

Any update on this?

Here us a solution:
Use your server.ts file to send the locale and inject your own custom LOCALE_ID :)

Do you have a suggestion how to use that on express ?

Looks like this is the intended behavior. We can add a guide on i18n with SSR, but if we go in this direction we should probably track that separately.

@mgechev how do you suggest we do that? You can direct me and I will be glad to write down the missing part to the docs except that I don't know how to fix it. Thank you

You can find some workarounds here.
We're working on a more dynamic localization implementation that will enable simpler behavior.


refactor(tree-wide): misc improvements

[x] @accent-color to @accent, @lookup to @flavor.
[x] Simplify duplicate media queries.
[ ] Introduce newly documented @catppuccin-filters method for image theming (PARTLY COMPLETE).

Sorry for pings! Doing the last tree-wide PR for a while, some cleanup and QOL improvements for all userstyles.


internal/copyright: TestCopyright doesn't report errors in builders

Proposal Details
This is what when I run gotip test internal/copyright locally:
--- FAIL: TestCopyright (11.43s)
    copyright_test.go:67: C:\Users\qmuntaldiaz\code\go\go\src\cmd\go\internal\gofips\gofips.go: missing copyright notice
    copyright_test.go:67: C:\Users\qmuntaldiaz\code\go\go\src\hash\boring_test.go: missing copyright notice
    copyright_test.go:67: C:\Users\qmuntaldiaz\code\go\go\src\hash\notboring_test.go: missing copyright notice

The failures are correct, those fails don't have a copyright header when they should.
Strangely, these errors are not reported in the [Build Dashboard](https://build.golang.org/.
Note that I can also reproduce this on Linux.
@rsc @ian

I had a stale GOROOT, closing.


Translate country vocabulary

The dynamic rs.gbif.org vocabulary needs to be translated into Spanish:

Online: http://rs.gbif.org/vocabulary/iso/3166-1_alpha2.xml
SVN: http://code.google.com/p/gbif-registry/source/browse/trunk/rs.gbif.org/vocabulary/iso/3166-1_alpha2.xml

Original issue reported on code.google.com by wixner on 2011-09-27 10:04:37

Vocabulary translation is an ongoing activity depending on the availability of volunteer translators. Closing issue.


Update kaldi_io.py

If the file is empty the reading is not proceeded starting from the next bit, the whole file is skipped starting from the failing bit. Exception catching is added in a bit dirty way, so that when there is an empty file, it is skipped and the read proceeds to the next file.

Merged as part of #58. I apologize for the loooong delay and thank you for the contribution.
Karel


[BUG] System.FormatException: Input string was not in a correct format.

Acknowledgement of preliminary instructions

[X] I have read the preliminary instructions, and I am certain that my problem has not already been addressed.
[X] I have thoroughly looked through the available Wiki articles and could not find a solution to my problem.
[X] I am using the latest version of Bloxstrap.
[ ] I did not answer truthfully to all the above checkboxes.

What problem did you encounter?
meu bloxtrap não está funcionando.
Bloxstrap Log
2024-10-16T21:50:40Z [App::OnStartup] Starting Bloxstrap v2.8.0
2024-10-16T21:50:40Z [App::OnStartup] Compiled Wednesday, 16 October 2024 at 6:44:37 AM from commit 42ffec26cf2e080fe22210305a08fcb286aeb577 (tag/v2.8.0)
2024-10-16T21:50:40Z [App::OnStartup] Loaded from %UserProfile%\AppData\Local\Temp\Bloxstrap-v2.8.0.exe
2024-10-16T21:50:40Z [App::OnStartup] Temp path is %UserProfile%\AppData\Local\Temp\Bloxstrap
2024-10-16T21:50:40Z [App::OnStartup] WindowsStartMenu path is %UserProfile%\AppData\Roaming\Microsoft\Windows\Start Menu\Programs
2024-10-16T21:50:40Z [Logger::Initialize] Initializing at %UserProfile%\AppData\Local\Bloxstrap\Logs\Bloxstrap_20241016T215040Z.log
2024-10-16T21:50:40Z [Logger::Initialize] Finished initializing!
2024-10-16T21:50:40Z [JsonManager::Load] Loading from %UserProfile%\AppData\Local\Bloxstrap\Settings.json...
2024-10-16T21:50:40Z [JsonManager::Load] Loaded successfully!
2024-10-16T21:50:40Z [JsonManager::Load] Loading from %UserProfile%\AppData\Local\Bloxstrap\State.json...
2024-10-16T21:50:40Z [JsonManager::Load] Loaded successfully!
2024-10-16T21:50:40Z [FastFlagManager::Load] Loading from %UserProfile%\AppData\Local\Bloxstrap\Modifications\ClientSettings\ClientAppSettings.json...
2024-10-16T21:50:40Z [FastFlagManager::Load] Loaded successfully!
2024-10-16T21:50:41Z [App::GlobalExceptionHandler] An exception occurred
2024-10-16T21:50:41Z [App::FinalizeExceptionHandling] System.FormatException: Input string was not in a correct format.
at System.Number.ThrowOverflowOrFormatException(ParsingStatus status, TypeCode type)
at System.Version.TryParseComponent(ReadOnlySpan1 component, String componentName, Boolean throwOnFailure, Int32& parsedComponent)    at System.Version.ParseVersion(ReadOnlySpan1 input, Boolean throwOnFailure)
at System.Version.Parse(String input)
at System.Version..ctor(String version)
at Bloxstrap.Utilities.CompareVersions(String versionStr1, String versionStr2)
at Bloxstrap.Installer.HandleUpgrade()
at Bloxstrap.App.OnStartup(StartupEventArgs e)
at System.Windows.Application.<.ctor>b__1_0(Object unused)
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)

Duplicate of #3193