Crash on train-segment

Here are the logs:
2023-11-28 02:57:35
machine-engine: 2023-11-28T07:57:31.956844993Z info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app/engine_server/
2023-11-28 02:57:32
machine-engine: 2023-11-28T07:57:31.956781112Z info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
2023-11-28 02:57:32
machine-engine: 2023-11-28T07:57:31.956756852Z info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
2023-11-28 02:57:32
machine-engine: 2023-11-28T07:57:31.956059779Z info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://[::]:80
2023-11-28 02:57:32
machine-engine: 2023-11-28T07:57:31.939813561Z info: SIL.Machine.AspNetCore.Services.SmtTransferEngineCommitService[0]
      SMT transfer engine commit service started.
2023-11-28 02:57:31
machine-engine: 2023-11-28T07:57:31.950085029Z warn: Microsoft.AspNetCore.Server.Kestrel[0]
      Overriding address(es) 'http://+:80'. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead.
2023-11-28 02:57:31
serval-api: 2023-11-28T07:57:30.389080935Z fail: Serval.Shared.Controllers.ServiceUnavailableExceptionFilter[0]
      A user tried to access an unavailable service. See health-check logs for more details.
2023-11-28 02:57:31
serval-api: 2023-11-28T07:57:30.387655249Z info: Grpc.Net.Client.Internal.GrpcCall[3]
      Call failed with gRPC error status. Status code: 'Unavailable', Message: 'Error connecting to subchannel.'.
2023-11-28 02:57:31
serval-api: 2023-11-28T07:57:30.387533477Z fail: Grpc.Net.Client.Internal.GrpcCall[6]
      Error starting gRPC call.
      Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error connecting to subchannel.", DebugException="System.Net.Sockets.SocketException: Connection refused")
       ---> System.Net.Sockets.SocketException (111): Connection refused
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
         at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
         at Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport.TryConnectAsync(ConnectContext context)
         --- End of inner exception stack trace ---

2023-11-28 02:57:31
serval-api: 2023-11-28T07:57:28.354145404Z info: Serval.Shared.Controllers.ErrorResultFilter[0]
      Client ZVFeO8livsYcjG8P7zX7eDParKlaNRHN@clients made request:
       {
        "action": "TrainSegment",
        "controller": "TranslationEngines",
        "id": "6539938fe73fc85d889b0ebf",
        "version": "1"
      }.
       Serval responded with code 503. Trace: 00-3ce99279eaa728e79d93030d6f49feb2-356da3090cddc602-00
2023-11-28 02:57:29
serval-api: 2023-11-28T07:57:28.350804443Z fail: Serval.Shared.Controllers.ServiceUnavailableExceptionFilter[0]
      A user tried to access an unavailable service. See health-check logs for more details.
2023-11-28 02:57:29
serval-api: 2023-11-28T07:57:28.348960229Z info: Grpc.Net.Client.Internal.GrpcCall[3]
      Call failed with gRPC error status. Status code: 'Unavailable', Message: 'Error starting gRPC call. HttpRequestException: An error occurred while sending the request. IOException: The request was aborted. IOException: The response ended prematurely while waiting for the next frame from the server.'.
2023-11-28 02:57:29
serval-api: 2023-11-28T07:57:28.347322419Z fail: Grpc.Net.Client.Internal.GrpcCall[6]
      Error starting gRPC call.
      System.Net.Http.HttpRequestException: An error occurred while sending the request.
       ---> System.IO.IOException: The request was aborted.
       ---> System.IO.IOException: The response ended prematurely while waiting for the next frame from the server.
         at System.Net.Http.Http2Connection.<ReadFrameAsync>g__ThrowMissingFrame|58_1()
         at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
         at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
         --- End of inner exception stack trace ---
         at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
         at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
         at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
         at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationTo
2023-11-28 02:57:29
serval-api: 2023-11-28T07:57:27.5036754Z info: System.Net.Http.HttpClient.SmtTransfer.ClientHandler[100]
      Sending HTTP request POST http://machine-engine/serval.translation.v1.TranslationEngineApi/TrainSegmentPair
2023-11-28 02:57:28
machine-engine: 2023-11-28T07:57:10.362128764Z info: Grpc.AspNetCore.Server.ServerCallHandler[7]
      Error status code 'Aborted' with detail 'The engine must be built first.' raised.
2023-11-28 02:57:28
serval-api: 2023-11-28T07:57:27.503632229Z info: System.Net.Http.HttpClient.SmtTransfer.LogicalHandler[100]
      Start processing HTTP request POST http://machine-engine/serval.translation.v1.TranslationEngineApi/TrainSegmentPair
2023-11-28 02:57:28
serval-api: 2023-11-28T07:57:10.397540812Z info: Serval.Shared.Controllers.ErrorResultFilter[0]
      Client ZVFeO8livsYcjG8P7zX7eDParKlaNRHN@clients made request:
       {
        "action": "GetWordGraph",
        "controller": "TranslationEngines",
        "id": "6539938fe73fc85d889b0ebf",
        "version": "1"
      }.
       Serval responded with code 409. Trace: 00-a407832ca91de6e3863280a95fed2a8e-4d60dd22e7603fd8-00
2023-11-28 02:57:11
serval-api: 2023-11-28T07:57:10.387553859Z info: Grpc.Net.Client.Internal.GrpcCall[3]
      Call failed with gRPC error status. Status code: 'Aborted', Message: 'The engine must be built first.'.
2023-11-28 02:57:11
serval-api: 2023-11-28T07:57:10.385825257Z info: System.Net.Http.HttpClient.SmtTransfer.LogicalHandler[101]
      End processing HTTP request after 81.0251ms - 200
2023-11-28 02:57:11
serval-api: 2023-11-28T07:57:10.385792177Z info: System.Net.Http.HttpClient.SmtTransfer.ClientHandler[101]
      Received HTTP response headers after 77.3348ms - 200
2023-11-28 02:57:11
serval-api: 2023-11-28T07:57:10.29479925Z info: System.Net.Http.HttpClient.SmtTransfer.ClientHandler[100]
      Sending HTTP request POST http://machine-engine/serval.translation.v1.TranslationEngineApi/GetWordGraph
2023-11-28 02:57:11
serval-api: 2023-11-28T07:57:10.293951424Z info: System.Net.Http.HttpClient.SmtTransfer.LogicalHandler[100]
      Start processing HTTP request POST http://machine-engine/serval.translation.v1.TranslationEngineApi/GetWordGraph

It appears that, while the engine is not built, it is in a corrupted state that is, although engine.BuildRevision == 0, when training a segment pair, it takes the poorly initilized engine and crashes the server.  This edge case should likely also be handled in Thot.

I don't see in the log where it is crashing the server.

There is no log where it crashes - I am assuming it was a segfault.  There is just a log for the missing service (and then a later log for the service restarting).

The engine is corrupted, but it doesn't cause a segmentation fault when it is loaded. I added code that still allows the engine to load. I tried training a segment pair, which is what seemed to cause the crash in this instance, and it did not cause a segmentation fault either. I cannot seem to reproduce the segmentation fault. I'm guessing that the low disk space caused the model in memory to get corrupted, which caused the segmentation fault. Did this crash occur once or over and over again as we saw in the other case of a corrupted model?
My plan right now is to make Thot more robust when loading a model with this kind of corruption. Hopefully, that will be sufficient to handle this kind of issue in the future.


Remove token

The new cassette for the test_data_links has some information in it pertaining to the user who executed the test. This PR redacts it.
BTW - at the hackathon I stated that I was going to add a VCR cassette for test_data_links which I attempted to PR today only to find the same work had been merged three days ago. We need to coordinate better.

📚 Documentation preview 📚: https://earthaccess--528.org.readthedocs.build/en/528/

Yup. I looked into using
with my_vcr.use_cassette('test.yml', filter_post_data_parameters=['api_key']):     requests.post('http://api.com/postdata', data={'api_key': 'secretstring'})
but I couldn't get it working properly.

Yup. I looked into using with my_vcr.use_cassette('test.yml', filter_post_data_parameters=['api_key']): requests.post('http://api.com/postdata', data={'api_key': 'secretstring'}) but I couldn't get it working properly.

Ok. I'll take a look. If I can't sort it out in short order, I'll let you know and we can approve this redaction in the meantime. If that's the case, then I'll open a new issue to later address this via vcrpy functionality.

Sorry, I should have been more specific. I can keep looking at it. Just not immediately.

BTW - at the hackathon I stated that I was going to add a VCR cassette for test_data_links which I attempted to PR today only to find the same work had been merged three days ago. We need to coordinate better.

One way that might help here is to put PRs into, or back into, draft status if there's planned future work but the PR appears to be in ready-for-review or ready-to-merge state.

This is a 'preliminary pull request' because I wanted to get your eyes on the redact_key_values code. Not being a python coder I'm sure it requires some love. I'm going to work on redacting uri: https://urs.earthdata.nasa.gov/api/users/dschuck?client_id=

I've made all the suggestions changes with the exception of moving redact_key_values out of TestCase which prevents it from being called. Looking into that next.

@doug-newman-nasa, I put these at the top level in tests/unit/test_results.py (preceding class TestResults), and everything works fine for me:
def redact_login_request(request):
    if "/api/users/" in request.path and "/api/users/tokens" not in request.path:
        _, user_name = os.path.split(request.path)
        request.uri = request.uri.replace(user_name, REDACTED_STRING)

    return request

def redact_key_values(keys_to_redact):
    def redact(payload):
        for key in keys_to_redact:
            if key in payload:
                payload[key] = REDACTED_STRING
        return payload

    def before_record_response(response):
        body = response["body"]["string"].decode("utf8")

        with contextlib.suppress(json.JSONDecodeError):
            payload = json.loads(body)
            redacted_payload = (
                list(map(redact, payload))
                if isinstance(payload, list)
                else redact(payload)
            )
            response["body"]["string"] = json.dumps(redacted_payload).encode()

        return response

    return before_record_response

Give it a shot and let me know what you see.
Also, I'd like to request a minor tweak to an assertion in the function test_get_all_more_than_2k. Because the number of granules is changing, if someone decide to regenerate all of the cassettes, this test will fail due to a difference in the length of the granules list.
So instead of this assertion:
        self.assertEqual(len(granules), 2520)

let's do this, which will be less strict, but still validate the results correctly:
        self.assertIn(len(granules), range(2001, 3001))

Regarding test_get_all_more_than_2k what I am trying to test here is that the correct number of granules were marshaled from the result. Using a range comparison will limit the effectiveness of that test. However, I'm not actually testing against the value in the result rather the result I fished out manually. So I'm going to counter by asserting
len(granules) against the value of 'hits' in the response
string: '{"hits":2520,"took":138,"items":[]}'
Does that work?

Regarding test_get_all_more_than_2k what I am trying to test here is that the correct number of granules were marshaled from the result. Using a range comparison will limit the effectiveness of that test. However, I'm not actually testing against the value in the result rather the result I fished out manually. So I'm going to counter by asserting equality of len(granules) against the value of 'hits' in the response string: '{"hits":2520,"took":138,"items":[]}' Does that work?

Perfect! Even better!

I've approved this, but does anybody else want to review this?
BTW, I'd like to suggest that we squash merge PRs to have a tidier, more readable change history.

Last call for someone else to review. If I don't get any takers today, I'll proceed with a squash merge near COB US Eastern.


Convert method definitions

This PR is for conversion to method definition only, because there are many differences in git.
Automatic conversion in lebab does not occur if the constructor is defined in a class declaration.
The first purpose of Revert is to reduce manual conversion.

Just curious, would https://github.com/hexojs/hexo/blob/master/lib/hexo/post.js benefit from refactor to this style?

If you reference hexo.model() or hexo.database, you will benefit from it.


Having trouble installing and using jhub-apps in TLJH

Context
I have installed the latest TLJH on a Ubuntu instance and have been trying to install jhub-apps into it following the instructions provided in the readme. And it appears to spawn the API docs on port 10202 but not the actual jhub-apps on port 8000. I don't see any particular errors in the journalctl that would indicate a start issue. Is there something I am missing?
I have jhub-apps installed for both /opt/tljh/hub and /opt/tljh/user.

Jul 23 17:17:34 TSFV-NEBARI systemd[1]: Started jupyterhub.service.
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.688 JupyterHub app:2885] Running JupyterHub version 4.1.5
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.689 JupyterHub app:2915] Using Authenticator: oauthenticator.generic.GenericOAuthenticator-16.3.1
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.689 JupyterHub app:2915] Using Spawner: jhub_apps.spawner.spawner_creation.JHubSpawner
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.689 JupyterHub app:2915] Using Proxy: jupyterhub_traefik_proxy.fileprovider.TraefikFileProviderProxy-1.1.0
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.699 JupyterHub app:1683] Loading cookie_secret from /opt/tljh/state/jupyterhub_cookie_secret
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [W 2024-07-23 17:17:35.778 JupyterHub configurable:214] Config option userdata_method not recognized by GenericOAuthenticator.  Did you mean one of: userdata_params, userdata_token_method, userdata_url?
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.784 JupyterHub app:2005] Not using allowed_users. Any authenticated user will be allowed.
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.813 JupyterHub provider:662] Updating oauth client service-japps
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.854 JupyterHub fileprovider:99] Creating the dynamic configuration file: /opt/tljh/state/rules/rules.toml
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.873 JupyterHub app:2954] Initialized 0 spawners in 0.020 seconds
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.879 JupyterHub metrics:279] Found 2 active users in the last ActiveUserPeriods.twenty_four_hours
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.880 JupyterHub metrics:279] Found 2 active users in the last ActiveUserPeriods.seven_days
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.880 JupyterHub metrics:279] Found 2 active users in the last ActiveUserPeriods.thirty_days
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.881 JupyterHub app:3168] Not starting proxy
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.884 JupyterHub app:3204] Hub API listening on http://<IP_ADDRESS>:15001/hub/
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.884 JupyterHub app:3215] Starting managed service cull-idle
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.884 JupyterHub service:386] Starting service 'cull-idle': ['/opt/tljh/hub/bin/python3', '-m', 'jupyterhub_idle_culler', '--timeout=600', '--cull-every=60', '--concurrency=5', '--max-age=0']
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.885 JupyterHub service:134] Spawning /opt/tljh/hub/bin/python3 -m jupyterhub_idle_culler --timeout=600 --cull-every=60 --concurrency=5 --max-age=0
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.891 JupyterHub app:3215] Starting managed service japps at http://<IP_ADDRESS>:10202
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.891 JupyterHub service:386] Starting service 'japps': ['python3', '-m', 'uvicorn', 'jhub_apps.service.app:app', '--port=10202', '--host=<IP_ADDRESS>', '--workers=2']
Jul 23 17:17:35 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:35.893 JupyterHub service:134] Spawning python3 -m uvicorn jhub_apps.service.app:app --port=10202 --host=<IP_ADDRESS> --workers=2
Jul 23 17:17:36 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:36.013 JupyterHub log:192] 200 GET /hub/api/<EMAIL_ADDRESS>13.10ms
Jul 23 17:17:36 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:36.022 JupyterHub log:192] 200 GET /hub/api/users?state=[secret]<EMAIL_ADDRESS>7.72ms
Jul 23 17:17:36 TSFV-NEBARI python3[22206]: INFO:     Uvicorn running on http://<IP_ADDRESS>:10202 (Press CTRL+C to quit)
Jul 23 17:17:36 TSFV-NEBARI python3[22206]: INFO:     Started parent process [22206]
Jul 23 17:17:36 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:36.792 JupyterHub log:192] 200 GET /hub/api (@<IP_ADDRESS>) 0.73ms
Jul 23 17:17:37 TSFV-NEBARI python3[22210]: INFO:     Started server process [22210]
Jul 23 17:17:37 TSFV-NEBARI python3[22210]: INFO:     Waiting for application startup.
Jul 23 17:17:37 TSFV-NEBARI python3[22210]: INFO:     Application startup complete.
Jul 23 17:17:37 TSFV-NEBARI python3[22209]: INFO:     Started server process [22209]
Jul 23 17:17:37 TSFV-NEBARI python3[22209]: INFO:     Waiting for application startup.
Jul 23 17:17:37 TSFV-NEBARI python3[22209]: INFO:     Application startup complete.
Jul 23 17:17:37 TSFV-NEBARI python3[22209]: INFO:     <IP_ADDRESS>:52602 - "GET /services/japps/ HTTP/1.1" 200 OK
Jul 23 17:17:37 TSFV-NEBARI python3[22198]: [I 2024-07-23 17:17:37.715 JupyterHub app:3273] JupyterHub is now running, internal Hub API at http://<IP_ADDRESS>:15001/hub/

Value and/or benefit
tljh integration usage
Anything else?
No response

Hey @tsi-rnguyen thanks for reporting this. I have a few questions:

What does your jupyterhub_config.py looks like
What's your environment look like?

What do you see on jupyterhub's home page?

Hey @tsi-rnguyen thanks for reporting this. I have a few questions:

What does your jupyterhub_config.py looks like
What does your environment look like?

What do you see on jupyterhub's home page?

My config file is pretty basic, all I've configured was https, oauth, and attempted to install jhub-apps.
This is the base TLJH jupyterhub_config.py
> import os
> from glob import glob
>
> from tljh import configurer
> from tljh.config import CONFIG_DIR, INSTALL_PREFIX, USER_ENV_PREFIX
> from tljh.user_creating_spawner import UserCreatingSpawner
> from tljh.utils import get_plugin_manager
>
> c = get_config()  # noqa
> c.JupyterHub.spawner_class = UserCreatingSpawner
>
> c.JupyterHub.cleanup_servers = False
>
>
> c.JupyterHub.hub_port = 15001
>
> c.TraefikProxy.should_start = False
>
> dynamic_conf_file_path = os.path.join(INSTALL_PREFIX, "state", "rules", "rules.toml")
> c.TraefikFileProviderProxy.dynamic_config_file = dynamic_conf_file_path
> c.JupyterHub.proxy_class = "traefik_file"
>
> c.SystemdSpawner.extra_paths = [os.path.join(USER_ENV_PREFIX, "bin")]
> c.SystemdSpawner.default_shell = "/bin/bash"
>
> c.SystemdSpawner.unit_name_template = "jupyter-{USERNAME}"
>
> tljh_config = configurer.load_config()
> configurer.apply_config(tljh_config, c)
>
>
> pm = get_plugin_manager()
> pm.hook.tljh_custom_jupyterhub_config(c=c)
>
> extra_configs = sorted(glob(os.path.join(CONFIG_DIR, "jupyterhub_config.d", "*.py")))
> for ec in extra_configs:
>     load_subconfig(ec)

I have extended that config with the below:
from tljh.user_creating_spawner import UserCreatingSpawner
from jhub_apps.configuration import install_jhub_apps

c.JupyterHub.bind_url = "https://<hosted url>"
c.SystemdSpawner.unit_name_template = 'jupyter-{USERNAME}{JHUBSERVERNAME}'
c.JAppsConfig.apps_auth_type = "oauth" #  oauth or none (if you don't want authentication on apps)
c.JAppsConfig.python_exec = "python3"
c.JAppsConfig.jupyterhub_config_path = "jupyterhub_config.py"
c.JAppsConfig.conda_envs = []
c = install_jhub_apps(c, UserCreatingSpawner)

c.NotebookApp.tornado_settings = {"websocket_max_message_size": 100 * 1024 * 1024}

In addition here is my config.yaml
users:
  admin:
  - admin
Jupyterhub:
  base_url: <info has been removed>
https:
  enabled: true
  tls:
    key: <info has been removed>
    cert: <info has been removed>
auth:
  type: oauthenticator.generic.GenericOAuthenticator
  GenericOAuthenticator:
    <info has been removed>
hub:
  db:
    upgrade: true

Below is the packages installed inn /opt/tljh/hub/
Package                          Version
-------------------------------- --------------
aiohttp                          3.9.5
aiosignal                        1.3.1
alembic                          1.13.2
annotated-types                  0.7.0
anyio                            4.4.0
argon2-cffi                      23.1.0
argon2-cffi-bindings             21.2.0
arrow                            1.3.0
asttokens                        2.4.1
async-generator                  1.10
async-lru                        2.0.4
async-timeout                    4.0.3
attrs                            23.2.0
Babel                            2.15.0
backcall                         0.2.0
backoff                          2.2.1
backports.tarfile                1.2.0
bcrypt                           4.2.0
beautifulsoup4                   4.12.3
bleach                           6.1.0
bokeh                            3.1.1
bokeh-root-cmd                   0.1.2
cachetools                       5.4.0
certifi                          2024.7.4
certipy                          0.1.3
cffi                             1.16.0
charset-normalizer               3.3.2
click                            8.1.7
comm                             0.2.2
contourpy                        1.1.1
cryptography                     43.0.0
debugpy                          1.8.2
decorator                        5.1.1
defusedxml                       0.7.1
distlib                          0.3.8
dnspython                        2.6.1
email_validator                  2.2.0
escapism                         1.0.1
exceptiongroup                   1.2.2
executing                        2.0.1
fastapi                          0.111.1
fastapi-cli                      0.0.4
fastjsonschema                   2.20.0
filelock                         3.15.4
fqdn                             1.5.1
frozenlist                       1.4.1
greenlet                         3.0.3
h11                              0.14.0
hatch                            1.12.0
hatchling                        1.25.0
httpcore                         1.0.5
httptools                        0.6.1
httpx                            0.27.0
hyperlink                        21.0.0
idna                             3.7
importlib_metadata               8.2.0
importlib_resources              6.4.0
ipykernel                        6.29.5
ipython                          8.12.3
ipywidgets                       8.1.3
isoduration                      20.11.0
jaraco.classes                   3.4.0
jaraco.context                   5.3.0
jaraco.functools                 4.0.1
jedi                             0.19.1
jeepney                          0.8.0
jhsingle-native-proxy            0.8.2
jhub-apps                        2024.6.1
Jinja2                           3.1.4
json5                            0.9.25
jsonpointer                      3.0.0
jsonschema                       4.23.0
jsonschema-specifications        2023.12.1
jupyter                          1.0.0
jupyter_client                   8.6.2
jupyter-console                  6.6.3
jupyter_core                     5.7.2
jupyter-events                   0.10.0
jupyter-lsp                      2.2.5
jupyter_server                   2.14.2
jupyter_server_terminals         0.5.3
jupyter-telemetry                0.1.0
jupyterhub                       4.1.5
jupyterhub-firstuseauthenticator 1.0.0
jupyterhub-idle-culler           1.3.1
jupyterhub-ldapauthenticator     1.3.2
jupyterhub-nativeauthenticator   1.2.0
jupyterhub-systemdspawner        1.0.1
jupyterhub-tmpauthenticator      1.0.0
jupyterhub-traefik-proxy         1.1.0
jupyterlab                       4.2.4
jupyterlab_pygments              0.3.0
jupyterlab_server                2.27.3
jupyterlab_widgets               3.0.11
keyring                          25.2.1
ldap3                            2.9.1
linkify-it-py                    2.0.3
Mako                             1.3.5
Markdown                         3.6
markdown-it-py                   3.0.0
MarkupSafe                       2.1.5
matplotlib-inline                0.1.7
mdit-py-plugins                  0.4.1
mdurl                            0.1.2
mistune                          3.0.2
more-itertools                   10.3.0
multidict                        6.0.5
nbclient                         0.10.0
nbconvert                        7.16.4
nbformat                         5.10.4
nest-asyncio                     1.6.0
notebook                         7.2.1
notebook_shim                    0.2.4
numpy                            1.24.4
oauthenticator                   16.3.1
oauthlib                         3.2.2
onetimepass                      1.0.1
overrides                        7.7.0
packaging                        24.1
pamela                           1.1.0
pandas                           2.0.3
pandocfilters                    1.5.1
panel                            1.2.3
param                            2.1.1
parso                            0.8.4
passlib                          1.7.4
pathspec                         0.12.1
pexpect                          4.9.0
pickleshare                      0.7.5
pillow                           10.4.0
pip                              24.2
pkg_resources                    0.0.0
pkgutil_resolve_name             1.3.10
platformdirs                     4.2.2
plotlydash-tornado-cmd           0.0.6
pluggy                           1.5.0
prometheus_client                0.20.0
prompt_toolkit                   3.0.47
psutil                           6.0.0
ptyprocess                       0.7.0
pure_eval                        0.2.3
pyasn1                           0.6.0
pycparser                        2.22
pycurl                           7.45.2
pydantic                         2.8.2
pydantic_core                    2.20.1
Pygments                         2.18.0
PyJWT                            2.8.0
pyOpenSSL                        24.2.1
python-dateutil                  2.9.0.post0
python-dotenv                    1.0.1
python-json-logger               2.0.7
python-multipart                 0.0.9
python-slugify                   8.0.4
pytz                             2024.1
pyviz_comms                      3.0.2
PyYAML                           6.0.1
pyzmq                            26.0.3
qtconsole                        5.5.2
QtPy                             2.4.1
referencing                      0.35.1
requests                         2.32.3
rfc3339-validator                0.1.4
rfc3986-validator                0.1.1
rich                             13.7.1
rpds-py                          0.19.1
ruamel.yaml                      0.17.40
ruamel.yaml.clib                 0.2.8
SecretStorage                    3.3.3
Send2Trash                       1.8.3
setuptools                       44.0.0
shellingham                      1.5.4
simpervisor                      0.4
six                              1.16.0
sniffio                          1.3.1
soupsieve                        2.5
SQLAlchemy                       2.0.31
stack-data                       0.6.3
starlette                        0.37.2
structlog                        24.4.0
terminado                        0.18.1
text-unidecode                   1.3
the-littlest-jupyterhub          1.0.0
tinycss2                         1.3.0
toml                             0.10.2
tomli                            2.0.1
tomli_w                          1.0.0
tomlkit                          0.13.0
tornado                          6.4.1
tqdm                             4.66.4
traitlets                        5.14.3
trove-classifiers                2024.7.2
typer                            0.12.3
types-python-dateutil            <IP_ADDRESS>40316
typing_extensions                4.12.2
tzdata                           2024.1
uc-micro-py                      1.0.3
uri-template                     1.3.0
urllib3                          2.2.2
userpath                         1.9.2
uv                               0.2.30
uvicorn                          0.30.3
uvloop                           0.19.0
virtualenv                       20.26.3
watchfiles                       0.22.0
wcwidth                          0.2.13
webcolors                        24.6.0
webencodings                     0.5.1
websocket-client                 1.8.0
websockets                       12.0
widgetsnbextension               4.0.11
xyzservices                      2024.6.0
yarl                             1.9.4
zipp                             3.19.2
zstandard                        0.23.0

and lastly all I see on my jupyterhub home:

@aktech, any insights from the above logs / configs ?

ping ? If there is nothing to be learned from the above data, any ideas how to debug this ?

I am currently facing the same issue with TLJH. It seems that the japps_service is running and if you go there you see the jhub apps api. I think the problem is that I cannot find the home page to set the apps.

Thanks for the config, taking a quick look at it, it seems you're missing:
from jhub_apps import theme_template_paths
c.JupyterHub.template_paths = theme_template_paths
c.JupyterHub.default_url = "/hub/home"
c.JAppsConfig.jupyterhub_config_path = "path/to/your/jupyterhub_config.py" # or something like str(Path(__file__).resolve())

Let me know if doesn't works. I'll to reproduce the config and will get back to you guys. I just came back from PTO today :)

Thanks for the response.
It is going a step further now. It now redirects the traffic to: services/japps/jhub-login. But that page cannot be reached:
INFO:     xx.xx.xxx.xxx:0 - "GET /services/japps/spawner-profiles/ HTTP/1.1" 403 Forbidden
INFO:     xx.xx.xxx.xxx:0 - "GET /services/japps/user HTTP/1.1" 403 Forbidden
INFO:     xx.xx.xxx.xxx:0 - "GET /services/japps/jhub-login HTTP/1.1" 302 Found
Aug 07 23:54:27 ip-xx-xx-xxx-xxx python3[5236]: 2024-08-07 23:54:27,435      INFO jhub_apps.service.routes:  90: event='Logging in' view=None peer=None request=<starlette.requests.Request object at 0x7cef0d101130> request_id='d45d34ad-bf9a-4507-a4d6-466b73cd48ea'

Here is my jupyterhub_config.py file content:
from tljh.user_creating_spawner import UserCreatingSpawner
from jhub_apps.configuration import install_jhub_apps
from jhub_apps import theme_template_paths
from pathlib import Path

c.JupyterHub.bind_url = "http://<hosted url>"
c.SystemdSpawner.unit_name_template = 'jupyter-{USERNAME}{JHUBSERVERNAME}'
c.JupyterHub.template_paths = theme_template_paths
c.JupyterHub.default_url = "/hub/home"
c.JAppsConfig.apps_auth_type = "oauth" #  oauth or none (if you don't want authentication on apps)
c.JAppsConfig.python_exec = "python3"
c.JAppsConfig.jupyterhub_config_path = "jupyterhub_config.py"
c.JAppsConfig.conda_envs = []
c = install_jhub_apps(c, UserCreatingSpawner)

c.JAppsConfig.jupyterhub_config_path = str(Path(__file__).resolve())

Any help is appreciated.

@aktech Would appreciate if you have any update / advice regarding this issue.

I just deployed a fresh TLJH and created a file named config.py at /opt/tljh/config/jupyterhub_config.d/config.py, with following contents.
from tljh.user_creating_spawner import UserCreatingSpawner
from jhub_apps.configuration import install_jhub_apps
from jhub_apps import theme_template_paths
from pathlib import Path

c.JupyterHub.bind_url = "http://<IP_ADDRESS>"
c.SystemdSpawner.unit_name_template = 'jupyter-{USERNAME}{JHUBSERVERNAME}'
c.JAppsConfig.apps_auth_type = "oauth" # or none (if you don't want authentication on apps)
c.JAppsConfig.python_exec = "python3"
# Pass in the path to jupyterhub config
c.JAppsConfig.jupyterhub_config_path = str(Path(__file__).resolve())
# Either a static list of conda environments to show in the
# create panel apps form or a callable to fetch conda enviornments
# dynamically, e.g. from conda-store API
c.JAppsConfig.conda_envs = []
c = install_jhub_apps(c, UserCreatingSpawner)

c.JupyterHub.template_paths = theme_template_paths
c.JupyterHub.default_url = "/hub/home"

I was able to access the jhub-apps and deploy a panel app:

The logo didn't loaded by default, which we need to fix.

Per JHub docs, there should be a &redirect_uri=/hub/home/.

I'll take a look and get that fixed.

It is going a step further now. It now redirects the traffic to: services/japps/jhub-login. But that page cannot be reached:

@BardiaKh What do you see in the UI and the network tab, can you share a screenhot?

I've installed TLJH locally using docker as described here: https://tljh.jupyter.org/en/latest/contributing/dev-setup.html
Then I created a conda env, installed all dependencies, and created a config.py file as you described
from tljh.user_creating_spawner import UserCreatingSpawner
from jhub_apps.configuration import install_jhub_apps
from jhub_apps import theme_template_paths
from pathlib import Path

c.JupyterHub.bind_url = "http://<IP_ADDRESS>:12000"
c.SystemdSpawner.unit_name_template = 'jupyter-{USERNAME}{JHUBSERVERNAME}'
c.JAppsConfig.apps_auth_type = "oauth" # or none (if you don't want authentication on apps)
c.JAppsConfig.python_exec = "python3"
# Pass in the path to jupyterhub config
c.JAppsConfig.jupyterhub_config_path = str(Path(__file__).resolve())
# Either a static list of conda environments to show in the
# create panel apps form or a callable to fetch conda enviornments
# dynamically, e.g. from conda-store API
c.JAppsConfig.conda_envs = []
c = install_jhub_apps(c, UserCreatingSpawner)

c.JupyterHub.template_paths = theme_template_paths
c.JupyterHub.default_url = "/hub/home"

I can start JupyterLab without issues, and I can see the new Hub Control panel, but when i try to create a streamlit app
it seems to time out and there is an error "Undefined has no properties".
The app is created twice and still shown in the Control Panel but it is not able to run.
When I start it the status goes back to "Pending"

Any idea what is going wrong? Thanks a lot.

Thanks for the tipps. I reinstalled everything and i can create streamlit dashboards now if I install everything in /opt/tljh/user. There are still two issues:

Conda envs

If i create a conda env the dashboard will not start: (https://discourse.jupyter.org/t/multiple-conda-environments/22530)
sudo -E /opt/tljh/user/bin/conda create -n py3.10 python=3.10 ipykernel streamlit jhub-apps altair pandas ipykernel jhsingle-native-proxy>=0.8.2 -c conda-forge -y
sudo /opt/tljh/user/envs/py3.10/bin/python3 -m ipykernel install --name "py3.10" --display-name "Python 3.10" --prefix /opt/tljh/user
How do I add/reference the new conda environments in the config?
c.JAppsConfig.conda_envs = ['py3.10'] or ['/opt/tljh/user/envs/py3.10']

Dashboards are not shared
I created a second user but the app is not shown for him. How can I share the app with all users or a group of them?

Thanks a lot!

@aktech Could you help please?


STeps are not in order

If you complete step 4A, you need to update the document to say skip steps 4b & 4c and start with step 5.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 706b2e1a-e90e-d70e-431f-5476b0ca52c1
Version Independent ID: 70721d4c-642c-61f8-6c61-84b1dd1ae96c
Content: Deploy a two-node Storage Spaces Direct SOFS for UPD storage in Azure
Content Source: WindowsServerDocs/remote/remote-desktop-services/rds-storage-spaces-direct-deployment.md
Service: unspecified
GitHub Login: @Haley-Rowland
Microsoft Alias: harowl

Step 4 states "by doing one of the following", therefore you should choose between options a, b, and c.

I'll update it to be a bulleted lists, instead of numbered. I definitely see that the numbers make it look like you're supposed to do all of the steps. Thanks! I'll come back and close this issue once I push the change.

I changed the steps to a bulleted list and tried to clarify the introductory language in the step. Please let me know if that's not enough. Thanks!


Workaround to not error on accessing value on a data element that does not exist

We are using a pre-created cert and passing in an ACM cert arn (certificate_arn), using a domain name that's not Route53 (create_route53_record = false), and the acm cert module invocation errors with:
Error: Error running plan: 1 error(s) occurred:

* module.atlantis.module.acm.var.zone_id: Resource 'data.aws_route53_zone.this' not found for variable 'data.aws_route53_zone.this.id'

This is a workaround for terraform to not error when invoking the ACM cert module when not using route53, and not creating an ACM cert..

hi @antonbabenko, would you be able to have a look at this PR at your convenience? Thanks!

Looks good. Thank you!

v1.13.0 has been released.


Use a helper to to define the cpp toolchain type

Part of https://github.com/bazelbuild/bazel/issues/14727

@phst PTAL

I've now dropped compatibility for the Bazel versions that don't support the helper function (that was necessary because µpb now also requires it), could you please rebase and force-push to re-trigger the tests? Thanks!

I rebased and pushed, but apparently I can't approve the tests.


Add k8s-auto-healer for the Kubernetes cluster

What this PR does / why we need it:
Add k8s-auto-healer for the Kubernetes cluster running on top of cloud environment, OpenStack is the default cloud provider implementation.
k8s-auto-healer ensures that the running Kubernetes nodes are healthy by monitoring the nodes’ status periodically, searching for unhealthy instances and triggering replacements when needed, maximizing your cluster’s efficiency and performance.
The cluster provider and health check mechanism are both pluggable and easy to integrate with other clouds and customized health check approaches.
For the openstack cloud provider:

Only the Magnum cluster is supported, doesn't support the Kubernetes cluster created in other ways(kops, cluster-api, kubeadm, etc.)
Auto-healing is achieved via Heat stack update, which replaces the unhealthy resources with new ones.

Which issue this PR fixes: fixes #
Special notes for your reviewer:
Release note:

cc @adisky @dims @hogepodge

One thing needs to be added, and we also need to discuss one major issue.

Major new features need documentation. The patch can't be considered complete until there is a corresponding document that describes how to use it.

It appears that the patch depends upon Magnum, which makes it not widely usable. We need to think about how we organize provider code that presumes the existence of clusters managed by Magnum, to make it clear that the solution is not a general one. Ideally we would want to have a more general solution, but that may not be possible given the last of desired cluster state without a management layer like Magnum. If we can think of general structures that would allow us to abstract the ideal cluster state, it would be preferable.

Docs is a solid requirement we need meet before merging. The Magnum requirement just needs to be documented, and we might want to think about a different location in the repository for now.
/hold
/assign @chrigl

One thing needs to be added, and we also need to discuss one major issue.

Major new features need documentation. The patch can't be considered complete until there is a corresponding document that describes how to use it.
It appears that the patch depends upon Magnum, which makes it not widely usable. We need to think about how we organize provider code that presumes the existence of clusters managed by Magnum, to make it clear that the solution is not a general one. Ideally we would want to have a more general solution, but that may not be possible given the last of desired cluster state without a management layer like Magnum. If we can think of general structures that would allow us to abstract the ideal cluster state, it would be preferable.

Docs is a solid requirement we need meet before merging. The Magnum requirement just needs to be documented, and we might want to think about a different location in the repository for now.
/hold
/assign @chrigl

+1

One thing needs to be added, and we also need to discuss one major issue.

Major new features need documentation. The patch can't be considered complete until there is a corresponding document that describes how to use it.
It appears that the patch depends upon Magnum, which makes it not widely usable. We need to think about how we organize provider code that presumes the existence of clusters managed by Magnum, to make it clear that the solution is not a general one. Ideally we would want to have a more general solution, but that may not be possible given the last of desired cluster state without a management layer like Magnum. If we can think of general structures that would allow us to abstract the ideal cluster state, it would be preferable.

Docs is a solid requirement we need meet before merging. The Magnum requirement just needs to be documented, and we might want to think about a different location in the repository for now.

Thanks for the suggestion, Chris. I'm working on the doc and a demo at the moment.
I agree we should have a general solution, however, at the moment, I can't think of an existing solution that is production ready in openstack for kubernetes-as-a-service, except Magnum(please correct me if i missed something). A similar discussion happened in cluster-autoscaler0 as well. Additionally, the cloud provider layer provides the abstract interface which makes things pluggable.

yes, please add some document and @lingxiankong when you say plugable, does this mean later on if we have other solution we may potentially write some glue code in order to make k8s-auto-healer to use it ?
and looks like there are some existing considerations on the healthy of node
https://github.com/openshift/machine-api-operator/#machine-healthcheck-controller , FYI

yes, please add some document and @lingxiankong when you say pluggable, does this mean later on if we have other solution we may potentially write some glue code in order to make k8s-auto-healer to use it ?

Don't understand what you mean by the glue code. By pluggable, I mean different cloud providers could have their own implementation.

and looks like there are some existing considerations on the healthy of node
https://github.com/openshift/machine-api-operator/#machine-healthcheck-controller , FYI

Thanks for the link, @jichenjc. We are living in OpenStack world, I'd prefer to use Machine API Operator if the k8s cluster is created using Cluster API rather than Magnum.

Hi.
This has the same purpose as work already done in magnum: https://review.opendev.org/#/c/631378/

I'm very glad you mentioned this patch, I was involved in the design and discussion of that patch, mostly with the author because we are in the same team, and Feilong is also the co-author of this PR actually. The current solution(NPD+Draino+autoscaler) in Magnum has some problems and cannot satisfy our production requirement, e.g. it doesn't take master nodes into consideration, it cannot guarantee the original node count, etc. What's more, the whole solution involves too many components which brings complexity and maintenance overhead, and cluster-autoscaler is not designed to deal with autohealing scenario.

That's not to say that a solution on the kubernetes side can't also exist, but since this proposal is limited to magnum clusters only there isn't really much benefit.

It's not limited to Magnum, but with Magnum as the default implementation.

Something that would definitely have to be changed is that the cluster label auto_healing_enabled is already used in the magnum implementation, so can't be used here.

Once we contributed the auto-healer in CPO, Magnum will be changed to use it when auto_healing_enabled=True

One final comment, in magnum this was achieved just by a combination of pre-existing components - draino, autoscaler, node problem detector - I would say that that makes it more configurable and easier to maintain than adding a new controller here.

IMHO, as I mentioned above, they are not doing good job in autohealing but brings complexity.
I'm not sure if you are already using that combo in production, we were, and we found some problems which motivated this PR.

@lingxiankong
Thanks for explaining, this makes more sense to me now.
So you would still be using node problem detector and draino for their roles in detecting/cordoning bad nodes, but this controller instead of the autoscaler? We aren't using the new auto healing in magnum so I can't comment on how well it works.

it doesn't take master nodes into consideration

This would maybe be possible after node groups are introduced in train. The master nodes will be their own node group and it would be possible to scale them.
I just tried the mechanism you're using (shut off VM, mark stack resource unhealthy, update stack), and I encounter one thing which is CERN specific but makes this mechanism unusable for us. All our VMs get a hostname on our network and our DNS only updates every 15 minutes, so it is not possible to delete and immediately remake a VM with the same name. When I send the update to the stack, it tries to replace the marked minion, and fails:
Resource UPDATE failed: resources.kube_minions: Resource CREATE failed: BadRequest: resources[0].resources.kube-minion: Hostname already in use (HTTP 400)
This can be avoided by putting the stack index of the unhealthy minion into the removed_rsrc_list of the kube_minions stack, so that the new minion takes a new name, but I don't know if that list should (or even can) be modified manually.

So you would still be using node problem detector and draino for their roles in detecting/cordoning bad nodes, but this controller instead of the autoscaler?

No, auto-healer will replace the whole combo and designed as the autohealing purpose specifically. The NPD and Draino are both great projects and professional in their specific areas, I believe they could be integrated in other solutions that more suitable for them.  In the end, we only need autoscaler for autoscaling and autohealer for autohealing.

All our VMs get a hostname on our network and our DNS only updates every 15 minutes, so it is not possible to delete and immediately remake a VM with the same name.

I think that'd be problematic whether or not you are using auto-healer. Suppose you are using cluster-autoscaler, what if the CA scales down a node but immediately scales up a new one within the DNS refresh period? I guess the new node name will be same with the old one, no?
Because that's a specific issue in CERN, one possible solution I can think of is to maintain your own cloud provider (maybe the only difference with the openstack one is one extra function to remove the DNS record after removing the VM somehow.)

/test cloud-provider-openstack-acceptance-test-e2e-conformance

/test cloud-provider-openstack-acceptance-test-e2e-conformance-stable-branch-v1.13

@lingxiankong when a node is deleted via heat, it's stack index is put into removed_rsrc_list property on the stack, and not used again.
For example if you have nodes [0, 1, 2] and the autoscaler removes minion-1 then adds one more node, the new node will be minion-3.

@lingxiankong when a node is deleted via heat, its stack index is put into removed_rsrc_list property on the stack, and not used again.
For example if you have nodes [0, 1, 2] and the autoscaler removes minion-1 then adds one more node, the new node will be minion-3.

Thanks for the answer, then it's ok for autoscaler.

@hogepodge @adisky @tghartland documentation is added.

All our VMs get a hostname on our network and our DNS only updates every 15 minutes, so it is not possible to delete and immediately remake a VM with the same name.

I think that'd be problematic whether or not you are using auto-healer. Suppose you are using cluster-autoscaler, what if the CA scales down a node but immediately scales up a new one within the DNS refresh period? I guess the new node name will be same with the old one, no?
Because that's a specific issue in CERN, one possible solution I can think of is to maintain your own cloud provider (maybe the only difference with the openstack one is one extra function to remove the DNS record after removing the VM somehow.)

Well, if we start maintaining our own, we can't give much feedback since it is unusable by us.

Well, if we start maintaining our own, we can't give much feedback since it is unusable by us.
Scaling down and then up fast works, since it does not create vms with the same names.
Deleted vms are appended in the list of deleted resources by heat (this is the default behavior
in heat ResourceGroup).

The current Magnum implementation for node repair relies heavily on the Heat "Mark Unhealthy Resources" feature, which replaces the unhealthy resource with a new one but using the same name. Although DNS only updates every 15 minutes sounds a bit weird to me, I will try to find if there is a better option. Your suggestions are also appreciated.

/test cloud-provider-openstack-acceptance-test-csi-cinder

Well, if we start maintaining our own, we can't give much feedback since it is unusable by us.
Scaling down and then up fast works, since it does not create vms with the same names.
Deleted vms are appended in the list of deleted resources by heat (this is the default behavior
in heat ResourceGroup).

The current Magnum implementation for node repair relies heavily on the Heat "Mark Unhealthy Resources" feature, which replaces the unhealthy resource with a new one but using the same name. Although DNS only updates every 15 minutes sounds a bit weird to me, I will try to find if there is a better option. Your suggestions are also appreciated.

DNS only updates every 15 minutes The reasons for this are not relevant to this discussion.
You can delete and create a node using magnum's cluster resize api.

My suggestion: delete and create a node using magnum's cluster resize api.

Resize API doesn't support master for now.

My suggestion: delete and create a node using magnum's cluster resize api.

Resize API doesn't support master for now.

So we can make it support it instead of implementing this new heat based thing. Which option seems more reasonable to your needs and why?

This component may be useful and may have better design than counterparts, but i am skeptical about having it in cloud-provider-openstack repo, as here we generally have implementation of plugins that integrates kubernetes with openstack services.

plugins that integrates kubernetes with openstack services.

The auto-healer is actually a service running on Kubernetes but need the capability of the clouds (with OpenStack services as the first citizen)

So we can make it support it instead of implementing this new heat based thing. Which option seems more reasonable based on your needs and why?

I agree using resize API is a better option at the time when it's fully implemented in Magnum. Before that, I didn't see there is an easier way to do the autohealing than using the Heat API in OpenStack. We are not developing a service for the future, but for the Kubernetes cluster created (via Magnum) in the current OpenStack cloud.

In any case, this is up to the cloud-provider-openstack maintainers to decide. Sorry for the noise, from our point of view, we can not help on it.

Appreciate your suggestion.

/test cloud-provider-openstack-acceptance-test-e2e-conformance-stable-branch-v1.13

As the original author of Magnum /resize API and the auto-healing feature in Magnum, I'm working with Lingxian very closely on this auto healer. And as Lingxian mentioned above, the main requirement to have a dedicated auto healer is because the combination of (NPD + Draino + Cluster Autoscaler) can't 100% meet the requirement of a small public cloud or user don't want a size-unpredicted cluster when doing auto-healing.
As for the /resize API, that's on our roadmap. As long as we get the /resize API working with master nodes in Magnum, we will switch current (heat mark unhealthy) way to Magnum /resize.  This is a very useful service for any cloud provider they want the auto-healing, and I'm pretty sure it will be adopted by more cloud providers. So it would be nice if the #sig-openstack can accept this until the community can figure out a better place. Thank you.

/test cloud-provider-openstack-acceptance-test-e2e-conformance

/test cloud-provider-openstack-acceptance-test-e2e-conformance

@hogepodge Can we move the PR forward? Is there anything else we could do?

In thinking about how to support the community on these efforts, I propose that we accept the code if the name is changed. The name k8s-auto-healer has a large potential to confuse end users, and to make it clear that this requires magnum I propose that we change it to "magnum-auto-healer" or magnum-k8s-auto-healer"

/test cloud-provider-openstack-acceptance-test-keystone-authentication-authorization

@hogepodge the binary has been renamed to magnum-auto-healer (and all other places related)

/test cloud-provider-openstack-acceptance-test-e2e-conformance

/approve

/hold cancel

/lgtm


coqPackages.CertiGraph: init at 8.19

Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Successfully built on coq-8.19 on "live" branch.
"CPS-Coq8.16" branch still failed to build with following error:
error: builder for '/nix/store/443shclw14zasrpjy4dkgh9wra49vkzy-coq8.15-Certigraph-dev.drv' failed with exit code 2;
       last 10 log lines:
       > [cannot-open-path,filesystem]
       > Warning:
       > Cannot open /nix/store/43z69yv2nv8yrjmkrbds7pjdfp29x3im-coq-8.15.2/lib/coq/user-contrib/compcert
       > [cannot-open-path,filesystem]
       > File ".//msl_ext/iter_sepcon.v", line 253, characters 54-74:
       > Error: The reference NPeano.Nat.lt_0_succ was not found in the current
       > environment.
       >
       > make: *** [Makefile:216: msl_ext/iter_sepcon.vo] Error 1
       > make: *** Waiting for unfinished jobs....
       For full logs, run 'nix log /nix/store/443shclw14zasrpjy4dkgh9wra49vkzy-coq8.15-Certigraph-dev.drv'.

Seem that compcert is required in an older branch?
I tried to fix it by adding COMPCERT_DIR and VST_DIR makeFlags, but it still raised inconsistent assumption error.


fix bug that Decimal256's hash value is not stable (#1574) (#1597)

What problem does this PR solve?
Issue Number: close #xxx
Problem Summary:
What is changed and how it works?
Check List
Tests

[ ] Unit test
[ ] Integration test
[ ] Manual test (add detailed scripts or steps below)
[ ] No code

Side effects

[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[ ] Breaking backward compatibility

Documentation

[ ] Affects user behaviors
[ ] Contains syntax changes
[ ] Contains variable changes
[ ] Contains experimental features
[ ] Changes MySQL compatibility

Release note

None

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

/run-integration-tests

/run-integration-test


HTML5 externs fix: void functions may return non-void

A number of functions are annotated as : Void but return the result of call. In recent versions of the HTMLMediaElement specification play() returns a promise. To avoid errors when the externs are updated I've removed the return.
(See https://github.com/HaxeFoundation/haxe/pull/7354)

Thanks!


Qwen2: PAD token = EOS token?

Current Behavior
Why does qwen2 sets pad token equal to eos token? If this is done and padding is used (likely in most training code) the attention mask is set to -1 for all eos/pad tokens.
There is no way to distinguish eos from pad so everything is attention masked to -1?
Or is qwen2 model/tokenizer code already auto detect first eos and treats as such and only subsequent eos are attention masked to -1?
Expected Behavior
pad_token != eos_token

Closed: https://github.com/QwenLM/Qwen/blob/main/tokenization_note.md


ProcessUserShadowStackPolicy question earlier

thanks for reply to install SDK, but just FYI that ProcessUserShadowStackPolicy is introduced in Win10 version 2004 (which my dev machine isn't at yet)...
https://docs.microsoft.com/en-us/windows/win32/api/winnt/ne-winnt-process_mitigation_policy
ProcessUserShadowStackPolicy | Windows 10, version 2004 and above: The policy regarding user-mode Hardware-enforced Stack Protection for the process.

FYI that ProcessUserShadowStackPolicy is introduced in Win10 version 2004

I don't understand the issue?

FYI that ProcessUserShadowStackPolicy is introduced in Win10 version 2004
which my dev machine isn't at yet

The project supports all versions of Windows both for compilation and at runtime. You can install the Windows 10 SDKs on Windows 7 and compile the code... You just need the latest SDK version installed which is currently 19041 because some types are in that version which are not available in previous versions.


Action generator doesn't respect Ruby file naming

Generator correct Ruby file naming
% bundle exec lotus generate action web AncientBooks#ToggleVisibility
      insert  apps/web/config/routes.rb
      create  spec/web/controllers/ancient_books/toggle_visibility_spec.rb
      create  apps/web/controllers/ancient_books/toggle_visibility.rb
      create  apps/web/views/ancient_books/toggle_visibility.rb
      create  apps/web/templates/ancient_books/toggle_visibility.html.erb
      create  spec/web/views/ancient_books/toggle_visibility_spec.rb

Closes #226

@nguyenngoc2505 This is just great! Thank you! :clap:


Metrics checker

Add a metrics checker to find lines of metrics being changed between two e3sm diags lat-lon runs.

@forsyth2 Hey Ryan, I think this small utility is handy for a quick check. And it potentially can help address #563

@chengzhuzhang I rebased this branch on the latest master with PR #576 to fix black, and pushed the pre-commit fixes.

@forsyth2 Hey Ryan, I have included changes suggested by your review. Since you have approved the PR, so i just went ahead and merge. Let me know if you get a chance to use this tool and we can further improve it.


Cannot run the program (on mac)

Hi,
when I try to install it, after
sudo npm install -g homekit2mqtt

I try to start it with
Anders-Erikssons-Mac-mini:.homebridge anders$ homekit2mqtt
/usr/local/bin/homekit2mqtt: line 1: syntax error near unexpected token `('
/usr/local/bin/homekit2mqtt: line 1: `var pkg = require('./package.json');'

I guess it is something simple but I cannot get it...

Oh, I guess this is my fault. Please try node /usr/local/lib/node_modules/homekit2mqtt/index.js --help this should work. Will fix the link in /usr/local/bin asap.

ok, thanks!

should be fixed in 0.3.0


Add enter event when create workspace

Add enter event to input field and empty string validation when create workspace.

For validation events, see ConfForm.vue.
We are using validation from element-plus.
https://element-plus.org/en-US/component/form.html#validation

@taeyeon-Kim
thx:)
amend b1b6649499707e5544397532c0fba817dee9a984


Fix: Wrong amount calculation when user creates expense in past

Found a major bug while testing, won't be moving forward with this approach. Hence, closing this off.


os x rendering bug

I'm having this bug with go-sdl2 on mac os x catalina, it seems like the renderer has a lot of padding, for example, this red box should be rendered at (0, 0):

after changing the window's position the box is renderer properly, like this:

only happens on my mac machine, tried to reproduce with same code on windows but it works fine there

Hi @xTal96, could I bother you with providing an minimal example code that reproduces this? I simply would like to know if it's an issue with the binding or SDL2 itself.

I updated my os to the latest version and now it won't happen,
closing


Allow for serverside rendering with jsdom

I'd like to use roughjs serverside with jsdom, but rough breaks when the global variable self isn't present. I saw that there are a couple of areas where self is used, surrounded by checks, but the checks themselves break because of a referenceerror.
I updated the checks to not throw an error when self doesn't exist in #68.

There's a lot of different ways to solve this of course, but for example, this a sketch of how the above could be addressed:
const jsdom = require('jsdom');
const rough = require('roughjs');

// Create dom
const { JSDOM } = jsdom;
const dom = new JSDOM();
const document = dom.window.document;

// Pass window to rough
rough.init({
  window: dom.window
})

// Create svg
const svg = document.createElementNS("http://www.w3.org/2000/svg", "svg");
document.body.appendChild(svg);
const rc = rough.svg(svg);

// Draw shapes
const node = rc.rectangle(50, 50, 200, 200, { roughness: 2.8, fill: 'blue' });
svg.appendChild(node);

console.log(svg.outerHTML);

It depends on if you want to break the existing api, if it's important that rough instantiates as soon as possible, etc.
You could also add an extra option to the global config that allows you to pass in window. That seems like a nice way to keep backwards compatibility, at the cost of delaying execution until either .svg or .canvas is called:
const jsdom = require('jsdom');
const rough = require('roughjs');

// Create dom
const { JSDOM } = jsdom;
const dom = new JSDOM();
const document = dom.window.document;

// Create svg
const svg = document.createElementNS("http://www.w3.org/2000/svg", "svg");
document.body.appendChild(svg);
const rc = rough.svg(svg, { window: dom.window });

// Draw shapes
const node = rc.rectangle(50, 50, 200, 200, { roughness: 2.8, fill: 'blue' });
svg.appendChild(node);

console.log(svg.outerHTML);

I think personally I'd prefer the latter.

Or even simpler, just update the bits of code that are using self to not throw an error when self doesn't exist 😋. Sorry for all the noise, was just thinking out loud. I'll send you a PR.

Thanks for the fix. I'll take a look when I get a chance.  Been too swamped with other stuff :/
Btw, have you looked at https://github.com/pshihn/rough/wiki/RoughGenerator and using toPaths with it. Getting raw path information may be more useful than creating a virtual dom. Depending on the situation of course.

I'll take a look when I get a chance. Been too swamped with other stuff :/

No problem, take your time, I'm not in a hurry 😊. As for the changes, I think the only real change is moving the self._roughScript assignment in generator.js. Maybe you want to do that differently, but it made sense to me since that's the only place it's used.
Other than that the changes preserve functionality, but prevent an error from being thrown, so they're just simple fixes.

Btw, have you looked at https://github.com/pshihn/rough/wiki/RoughGenerator and using toPaths with it. Getting raw path information may be more useful than creating a virtual dom. Depending on the situation of course.

That sounds good, I'll take a look at that. Would be great to not have to create a virtual dom 👍

I have ported the code from Javascript to TypeScript. In the process, I have also tried to address this issues. I have not actually tested it yet. Will updated this issue if fixed.

For it's been fixed with the new version 👍


Debugging

After a break in the debugger a program using softdevice ends up in a panic. I understand that  this might be related to the strong requirements of the nordic softdevice.
Please document either that debugging is not possible, or recommend a method to debug that doesn't lead to panics.

Using a debugger will certainly violate the softdevice timing constraints. Generally using defmt logging is the recommended way to debug softdevice apps. PRs to improve the documentation are welcome.


Desktop should show indication when loading

Describe what problem your feature request solves:
The desktop application can take a long time to load, and the spinner in App.vue does not show while loading
Describe the solution you'd like:
Desktop app shows indication of loading
Additional context:

Hey @jgadsden, can you please provide some more information as when is the spinner supposed to show up? when does the loading occur?

When the desktop application is loaded, it can take up to a minute for the application to show the welcome window. This is a relatively long time, and due to the large size of some of the components (probably can do something about that) and not displaying an initial 'wait for it' window

I am still failing to understand where is it taking time to load. I started the desktop application in couple of ways but I didn't find any wait for it or its corresponding screen.
here is how I tested :
In development mode :
https://github.com/OWASP/threat-dragon/assets/94848646/eebb0f12-3db7-4f72-b528-4ff3598fc9c2
Using desktop application :
https://github.com/OWASP/threat-dragon/assets/94848646/ac1121bd-82c7-4b88-962f-9effae205d1b

I am still failing to understand where is it taking time to load. I started the desktop application in couple of ways but I didn't find any wait for it or its corresponding screen.
here is how I tested :
In development mode :
2024-01-07.16-02-03.mp4
Using desktop application :
2024-01-07.16-00-24.mp4

Please guide me through it so I can have a better understanding of what the actually the issue is. A nice walkaround with a video of it from your side could sort things up?
Thanks.

When you install the released application installer and then invoke threat dragon, rather than running from the npm/pnpm command line, there is a delay before displaying the welcome screen

Hey @jgadsden! I have implemented a loading screen that pops up until the actual window loads. Is this what you were talking about?
https://github.com/OWASP/threat-dragon/assets/94848646/7be9a857-5798-433e-bba5-002d96be3171

Hello @raghav1030 ; yes, that is the sort of solution in mind. Thanks for working on this

whatever you think is best, it can be done just for the loader but if you have other ideas then that is all good - ideas gratefully received!

I can submit a pull request with this update for the loading screen. If we receive any future updates for the UI, we can easily incorporate them. Until then, this looks good ig?

yes, please go with it @raghav1030 and create the pull request

reopening this because the current solution does not work


Problems about pretraining

Hello, there is an error in the script 'run_pretrain.py'
when excute the import command
from transformers import BertForCopy, RobertaForCopy
error occurs "Cannot find reference 'BertForCopy' in 'init.py' "
Actually，there is not a model called 'BertForCopy' neither in transformers modules nor in the original transformer package
Did you miss it？

The code is heere:
https://github.com/thunlp/CorefBERT/blob/90c349ca5718118272cf17533fc36c96a5067709/transformers/modeling_bert.py#L1272-L1377

ok, thanks.


Support for Load params

Would like support for Load with multiple Ids
e.g
var ids = new [] { 1, 4, 6, 7 };

var result = session.Load<Member>(ids);

This would result in List<Member>
This can be achieved using:
select data from mt_doc_photo where id = ANY(:arg0 ::int[])
Where :arg0 is a parameter of type Text
Using Npgsql
var command = con.CreateCommand();

command.CommandText = "select data from mt_doc_photo where id = ANY(:arg0 ::int[])";

var name = "arg0";

var parameter = command.CreateParameter();
parameter.ParameterName = name;
parameter.Value = "{90, 92, 93}";
command.Parameters.Add(parameter);

var result = new List<string>();

using (var reader = command.ExecuteReader())
{
    while (reader.Read())
    {
        Console.WriteLine(reader.GetString(0));
    }
}

con.Close();

@phillip-haydon How is that different from what we already do? You can load multiple documents by an array or enumerable of document id's:
http://jasperfx.github.io/marten/documentation/documents/loading/

Ah I guess it's already supported then.
I honestly never looked at the doco for that one, I looked at the code for Load<T> and saw there was no overloads or anything like LoadAll<T> and figured it was not supported.
I think the only difference is that the current API isn't intuitive in my opinion.

Yeah, I agree. What if it's called LoadMany() or something instead?

LoadAll or LoadMany works for me.


tools: Fix various error in Windows native build

Summary
These patch finally raised NuttX Windows native build!

tools/mkdeps:Fix an error caused by do_shquote in Windows native build:
do_shquote in Windows native build will give following error in mkdeps:
sparc-elf-gcc: '-fno-common': No such file or directory sparc-elf-gcc: '-Wall': No such file or directory sparc-elf-gcc: '-Wstrict-prototypes': No such file or directory sparc-elf-gcc: '-Wshadow': No such file or directory sparc-elf-gcc: '-Wundef': No such file or directory sparc-elf-gcc: '-fomit-frame-pointer': No such file or directory sparc-elf-gcc: '-g': No such file or directory sparc-elf-gcc: '-mcpu=leon': No such file or directory sparc-elf-gcc: '-isystem': No such file or directory
tools/Config:Fix some error call DELFILE/DELDIR/CATFILE in Windows native build:
There are so many .ddc in libc that $(call CATFILE, bin/Make.dep, $^) give an error: input line is too long in Windows native build. also $(call DELFILE, $^) give same error.
libc and mm/Makefile:Fix an error caused by backslash in Windows native build:
There is a comment in the libs/libc/Makefile and mm/Makefile of NuttX 8.2 :
# REVISIT: Backslash causes problems in $(COBJS) target DELIM := $(strip /)  The DELIM := $(strip /) was removed thereafter, and resulted an error in $(COBJS) target in Windows native build.
tools/Win.mk:Fix some error in clean_context in Windows native build:
Error use of DIRUNLINK result failure in remove include\arch\board  \include\arch\chip include\arch    $(ARCH_SRC)\board\board  $(ARCH_SRC)\board  $(ARCH_SRC)\chip $(TOPDIR)\drivers\platform
mkdir -p staging Create dir '-p' while creating dir 'staging', and the dir '-p'  cant be remove when make distclean, which result failure of mkdir -p staging  in next bulid.
when use too many libc function, there are too many obj in $(2) and will give an error: too long input line error call ARCHIVE in Windows native build, because @echo "AR (create): ${shell basename $(1)} $(2)"
In pass1dep target, if $(USERDEPDIRS) is empty, there will be an error:
for %%G in () do ( make -C %%G depend )
make: *** [pass1dep] error<PHONE_NUMBER>

Impact
Windows native build
Testing
Configure + Build stm32f4discovery on Windows XP/7 + GnuWin32 + MinGW + kconfig-frontends-win32

why not use (Q)ifexist(1) (del /f /q $(1)) in Windows native build

if $1 contain too many member, if exist $1will give an error: Input line is too long in Windows native build.  this is a windows characteristic:

In Command Prompt, the total length of the following command line can't contain more than 8191 characters;
In a batch file, the total length of the following command line can't contain more than 8191 characters;
https://learn.microsoft.com/en-US/troubleshoot/windows-client/shell-experience/command-line-string-limitation
https://learn.microsoft.com/en-US/troubleshoot/windows-client/shell-experience/command-line-string-limitation

@zouboan please rebase your change to remove the merging patch and the conflict.

it seems the conflict was resolved?

I move the commit related to conflict to separate #7658 ,i think this PR is all right now @xiaoxiang781216

@zouboan  In the ubuntu environment, there are two separators for some information compiled and output using Make V=1
Is it necessary? Is there a way to remove
arm-none-eabi-ar rcs  libc.a    bin//lib_assert.o  bin//lib_builtin_getname.o  bin//lib_builtin_isavail.o  bin//lib_builtin_forindex.o


Move files to the src/ directory, add a package.json file

This will enable us to pull in esbuild/tsc & a dev server as development-only dependencies, and have tests.

Land your CL first, then I'll rebase this 😃. After that I'll be able to open a pull request for a reasonably clean dev environment.


Proposal to rename and simplify config

Rename package from @zakodium/eslint-config to eslint-config-zakodium. This will allow to reference it more simply in our ESLint files: zakodium/react instead of @zakodium/eslint-config/react.
Simplify config by removing "bundles". Only export "base" configs (for example, zakodium/adonis would just be the Adonis-specific config and would have to be explicitly combined with zakodium/typescript).

@zakodium/admin
WDYT?

SGTM. Easier names and compose yourself 👍🏻

Done.


Inconsistent node exits

Occasionally, after running several nodes (specific case was 4 from script), exit command will not properly call stop on all running nodes.

Having not experienced this since reported, closing this issue until it reoccurs.


Feature Request; Automatically create HM/CM/Expert variant competition

I find myself everytime i make a competition for my clan for a raid creating two competitions to track, for example, TOA normals AND TOA expert, therefore having to go through the create competition twice.
Proposal is to create a toggle switch or check box of some sort that, on competition create, make an identical competition with the same time parameters for the other raid variant.
This concept works well for group competitions but haven't thought it out for team competitions.
I agree it's a little confusing with the word "include" because all it's doing it automatically making the second competition on your behalf and then they are tracked separately.
Made a little POC for context:

This should be solved better with a fully fledged feature to handle multi-metric competitions


Display Currently Active Dropdown Option as Selected

For dropdown settings, show the currently active (user-chosen) one as the selected one on the UI.
Fixes #216

Issue Description
When dropdown options load in settings, the user-selected options are not shown as the currently selected item.
Steps to Reproduce
We can use the "Capitalize Headings > Style" option to demonstrate.

Open Obsidian Linter settings, and scroll down to the "Capitalize Headings" section
For the "Style" dropdown, choose a setting that's different from what's currently enabled. For example, if "Title Case" is currently your preferred setting, choose "All Caps".
Close the settings page. (Optional: test that the linter is now changing headings to "ALL CAPS").
Re-open the settings page, and scroll down to "Capitalize Headings"

Expected: that the newly selected option "ALL CAPS" is chosen as the Style (i.e. with a checkmark), and is currently shown
Actual: "Title Case" is still chosen as the Style, and it has the checkmark.
(see full details in #216)

Testing

Manual testing was done to verify this change
Any guidance on how / where we can add unit tests for this would be appreciated

Before Fix
The value shown in the drop-down always defaults to "Title Case", despite the currently selected user setting
https://user-images.githubusercontent.com/1861055/172000985-85e229c9-f027-42dd-a622-ff57572dfa85.mp4
After Fix
The currently selected user setting is now displayed as the active option in the dropdown
https://user-images.githubusercontent.com/1861055/172001033-8627ee0f-0b54-434c-95f4-7d54854e8ac7.mp4

Thanks for the detailed documentation and fix!


Add Sliding tabs while transiting from Latest Tech to Display Store

Hey @praveenscience I would like to fix the animation as it does not provide a smooth experience while traversing from Latest Tech to Display Store, kindly assign this issue to me !✌

Sure, thanks, make sure you follow the Prettier rules. Looking forward. 💪🏻
Why did you mention GSSoC 23? I specifically asked folks not to mention it right?

Sorry for the inconvenience @praveenscience , will keep note of it ✅

@dakshsinghrathore What's the update ? Final call...


Add filters for search results based on kind.

This allows you to search defs, theorems, etc independently.

Very cool, thank you!


Implement new Wallet Settings actions bar (save, cancel)

Detailed Description
Users should be able to save changes to wallet configurations without launching the wallet.
Context
It's not convenient or obvious that you need to launch a wallet in order to save changes to it's configuration.
Possible Implementation
Implement new Lightning Payment Received modal based on Zeplin design:
zpl://screen?sid=5c065f4f7324832ffb782ff2&pid=5bacdbad33af886d419f1b34

Closed via #1453


Provisioning Elasticsearch role & user resources

Overview
We'd like to provision Elasticsearch users and roles as part of our infrastructure management.
We've used local-exec provisioners calling out to scripts to do this for us so far, but understanding when a particular provisioner will be called and ensuring proper resource management using local-exec provisioners is awkward. It would be very useful to have native integration in this provisioner.
I would imagine this implemented as ec_user and ec_role resources, referencing a particular ec_deployment.
Is this something that is on the road map for this project?

Hi @kenoir, thanks for opening the issue.
These APIs belong to the Elasticsearch API and are not in scope for the EC Terraform provider, however they are in scope for an Elasticsearch Terraform provider but we do not have one at the moment I'm afraid.
local-exec provisioners are not great, but we do not plan to support Elasticsearch APIs through the Elastic Cloud API, which this particular provider uses.

Thanks for clarifying.


File V3000 format with Atom NOT List is not loaded

Affected version: Release/2.1
Steps to reproduce:

Open the attached file - V3000 with Atom NOT List.  29DHIS_prohibitatoms.zip

Actual result:
The file is not loaded. Nothing happens.

Expected result:
The file is loaded.

Status: PASS
Build: Release 2.1
OS/Browser: Win10/Chrome
Actual result:
The file is loaded and displayed.


New Device: APOSEN A550 Vacuum

Hello, just picked up this device from a thrift store and would love to have it integrated into my home assistant setup via a local api. I'm not too familiar with this project or robot vacuums in general so I'm not sure how difficult it would be to add support, but I'm happy to provide any device information needed.

In theory if it is tuya based it can be supported.  First you'll need the device id and local key (this information should be kept private) to do any local control of the device.  If you try to then configure the device in this integration with this information, and the device is not supported, there will be a warning message in the log with a snapshot of the current device status, which is a big part of what is needed to add support.  But better information can be obtained from the developer portal at iot.tuya.com, if you sign up for an account there (which you need to for the official tuya cloud integration, and also some of the methods of obtaining the local key), then under Cloud there is an API explorer.  It has recently changed, but one of the last categories is called "Device Control" (it used to be Smart Home Management), and the last function under there "Get Device Specification Attribute" gives more detailed information on most of the functions available.  Probably some dps you see reported in the log message will be missing from there so both sets of information are needed, but we can figure those out later, or just leave them as unknown attributes so they are available for observation so someone can figure out their meaning in future.

Thanks for the response. I've already gone through the setup guide and acquired the local api but home assistant just detects it as a simple switch. Navigating to the device specification attribute gave me the following information:
{
  "result": {
    "category": "sd",
    "functions": [
      {
        "code": "power",
        "dp_id": 1,
        "type": "Boolean",
        "values": "{}"
      },
      {
        "code": "power_go",
        "dp_id": 2,
        "type": "Boolean",
        "values": "{}"
      },
      {
        "code": "mode",
        "dp_id": 3,
        "type": "Enum",
        "values": "{\"range\":[\"standby\",\"smart\",\"wall_follow\",\"spiral\",\"chargego\",\"random\"]}"
      },
      {
        "code": "direction_control",
        "dp_id": 4,
        "type": "Enum",
        "values": "{\"range\":[\"backward\",\"turn_left\",\"turn_right\",\"stop\"]}"
      },
      {
        "code": "seek",
        "dp_id": 13,
        "type": "Boolean",
        "values": "{}"
      }
    ],
    "status": [
      {
        "code": "power",
        "dp_id": 1,
        "type": "Boolean",
        "values": "{}"
      },
      {
        "code": "power_go",
        "dp_id": 2,
        "type": "Boolean",
        "values": "{}"
      },
      {
        "code": "mode",
        "dp_id": 3,
        "type": "Enum",
        "values": "{\"range\":[\"standby\",\"smart\",\"wall_follow\",\"spiral\",\"chargego\",\"random\"]}"
      },
      {
        "code": "direction_control",
        "dp_id": 4,
        "type": "Enum",
        "values": "{\"range\":[\"backward\",\"turn_left\",\"turn_right\",\"stop\"]}"
      },
      {
        "code": "electricity_left",
        "dp_id": 6,
        "type": "Integer",
        "values": "{\"unit\":\"%\",\"min\":0,\"max\":100,\"scale\":0,\"step\":1}"
      },
      {
        "code": "seek",
        "dp_id": 13,
        "type": "Boolean",
        "values": "{}"
      },
      {
        "code": "clean_record",
        "dp_id": 15,
        "type": "String",
        "values": "{\"maxlen\":255}"
      },
      {
        "code": "clean_area",
        "dp_id": 16,
        "type": "Integer",
        "values": "{\"unit\":\"㎡\",\"min\":1,\"max\":9999,\"scale\":0,\"step\":1}"
      },
      {
        "code": "clean_time",
        "dp_id": 17,
        "type": "Integer",
        "values": "{\"unit\":\"分\",\"min\":0,\"max\":9999,\"scale\":0,\"step\":1}"
      },
      {
        "code": "fault",
        "dp_id": 18,
        "type": "Bitmap",
        "values": "{\"label\":[\"edge_sweep\",\"middle_sweep\",\"left_wheel\",\"right_wheel\",\"garbage_box\",\"land_check\",\"collision\"]}"
      }
    ]
  },
  "success": true,
  "t":<PHONE_NUMBER>611,
  "tid": "d4df3f19e47311ecb42622d60ed3c2d7"
}

And here is the error message that shows up in the logs when I try adding the vacuum in home assistant:
2022-06-05 02:07:10 WARNING (MainThread) [custom_components.tuya_local.config_flow] Device matches simple_switch with quality of 7%. DPS: {'1': True, '2': True, '3': 'smart', '4': 'foward', '5': '1', '6': 86, '13': False,>
2022-06-05 02:07:10 WARNING (MainThread) [custom_components.tuya_local.config_flow] Report this to https://github.com/make-all/tuya-local/issues/

Can you check that log message again?  There seems to be a continuation arrow at the end that is indicating there is more, maybe this is a new feature in 2022.6, but there should be some way to expand that to see the rest, or maybe you need to look in the full log.

Ah I was reading the log through an ssh session, I think the terminal probably cut it off. Here's the full message:
2022-06-05 02:07:10 WARNING (MainThread) [custom_components.tuya_local.config_flow] Device matches simple_switch with quality of 7%. DPS: {'1': True, '2': True, '3': 'smart', '4': 'foward', '5': '1', '6': 86, '13': False, '16': 4, '17': 10, '18': 0, '101': 'nar', '102': 31, '103': 175, '104': 0, 'updated_at':<PHONE_NUMBER>.0133498}
2022-06-05 02:07:10 WARNING (MainThread) [custom_components.tuya_local.config_flow] Report this to https://github.com/make-all/tuya-local/issues/

Since this is almost identical to Lefant M213, I have removed some of the extra unknown attributes from that config to make it compatible.

It does seem to let me add the device to home assistant now, but I'm not really sure what to do with it from here. It shows up as the entity "automation.vacuum", and attempting to add it to the dashboard only shows a simple on / off switch, which doesn't seem to do anything. Clicking the switch does not turn on the vacuum nor does it write anything to the logs.

My bad, I accidentally selected the entity for my vacuum automation task instead of the vacuum itself. Selecting the Lefant device during setup then adding the correct entity to the dashboard seems to be working perfectly. It also works well with this custom vacuum card: https://github.com/denysdovhan/vacuum-card


Custom operations generated files doesn't import custom scalars

When using custom operations, the files generated doesn't include the import statements for custom scalars that have been defined.

Where are you expecting the imports where they don't appear? Theres a test in clients/custom_scalars that tests generating them and here's an example where it's imported: https://github.com/mirumee/ariadne-codegen/blob/11bfe35bd62b2489927e0e93c6891bccc29c7f37/tests/main/clients/custom_scalars/expected_client/client.py#L5

Someone deleted their comment, but to give some more context, I have this in my pyproject.toml:
[tool.ariadne-codegen.scalars.DateTime]
type = "pydantic.AwareDatetime"

When not using custom operations, this works great. However when it generates the files for the custom operations, this does not get imported in those spesific files, so it is impossible to import the module without errors.


add style prop support

closes #13

@hagmic This is fantastic. I'll merge and roll this out with next release. Thanks much!
Maybe we should add css prop too for emotion.


Improve test coverage

Need to add tests to crud.py

Current coverage is 51% (73% for cruds.py).

Thank you so much @telenieko


[Gssoc21] News section is not responsive

Issue Type:
[x] Bug Report
[  ]  Feature Request
[  ] Documentaion
Describe the bug

@arpit456jain  nice, you can work on this, send a PR . Discuss this here with @SnehaVeerakumar

@arpit456jain I can fix it, thank you for spotting it out. Can you raise an issue regarding a separate page for any of the section in navigation bar !!
That would be really helpful.

@arpit456jain I can fix it, thank you for spotting it out. Can you raise an issue regarding a separate page for any of the section in navigation bar !!
That would be really helpful.

I can make a separate page for contact section

@arpit456jain I can fix it, thank you for spotting it out. Can you raise an issue regarding a separate page for any of the section in navigation bar !!
That would be really helpful.

I can make a separate page for contact section

"About us" page please. Level-3 can be given for that.
Contact us is already in progress

@arpit456jain I can fix it, thank you for spotting it out. Can you raise an issue regarding a separate page for any of the section in navigation bar !!
That would be really helpful.

I can make a separate page for contact section

"About us" page please. Level-3 can be given for that.
Contact us is already in progress

ok

I

@arpit456jain I can fix it, thank you for spotting it out. Can you raise an issue regarding a separate page for any of the section in navigation bar !!
That would be really helpful.

I can make a separate page for contact section

"About us" page please. Level-3 can be given for that.
Contact us is already in progress

@himanshu007-creator  I had raised a issue for this. Can i close this issue now?

@arpit456jain Yes, go ahead.


[105] Await_response not failing when payload does not contain key

Closes #105.

Maybe this also want a changelog 🤷 ?

Maybe this also wants a changelog 🤷 ?

IMO it's not user-facing, so I'd not write one


Add a license to the repository

I recently came across your project getting-started and I noticed that the repository github.com/ionq-samples/getting-started currently does not have a license specified.
Why Adding a License is Important?
As GitHub documents

Public repositories on GitHub are often used to share open source software. For your repository to truly be open source, you'll need to license it so that others are free to use, change, and distribute the software.

Would be possible to add a valid license to this repository? Probably something really OpenSource such as the Apache 2.0 license would fit really well.
Thanks!

thank you! we agree and added apache 2.0

Thanks!


increase speed performance !

hi ,
i have a fresh laravel's instalation , for the welcome page , the response time is 700ms oO ? so slowly ?
are thez solution for increase response time of the application ? or it's a framework issue ?
not a volume acces issue with virtualB  ?

This is a Docker issue, I don't think we can do anything with LaraDock to speed it up!!


Map Layers - Reference Layers

Miro reference here
Map Layer feature contains a legend for the PLSS point layer, and reference layers. PLSS points are always on, reference layers can be turned on/off.

[ ]  Landownership

[ ] Parcels with Parcel ID as label

[ ] BLM CadNSDI

All layers within the version 2.2 database

Cadastral team to create map resource and define layer symbology

Hover cursor on question mark for layer legend

this needs to be reworked some

closed in favor of #77, #78, #79, #80


Shutdown fails because new peer connects

It looks like the connection might be started by the auto-connect background thread.
^CShutting down Squeaknode...
INFO:squeaknode.admin.webapp.app:Stopping SqueakAdminWebServer....

INFO:squeaknode.admin.webapp.app:Stopped SqueakAdminWebServer.
INFO:squeaknode.admin.squeak_admin_server_servicer:Stopping SubscribeSqueakDisplays.
INFO:squeaknode.admin.squeak_admin_server_servicer:Stopped SqueakAdminServerServicer.
INFO:squeaknode.network.peer_server:Stopping peer server listener thread...
INFO:squeaknode.network.peer_server:Stopped accepting incoming connections.
INFO:squeaknode.network.peer:Stopping peer socket: <socket.socket fd=24, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('<IP_ADDRESS>', 47842), raddr=('<IP_ADDRESS>', 18555)>
ERROR:squeaknode.network.peer:revc_data is None
INFO:squeaknode.network.peer:Failed to receive msg from <squeaknode.network.peer.MessageReceiver object at 0x7fe0d53e7a60>
INFO:squeaknode.network.peer:Received msg None from Peer(PeerAddress(host='<IP_ADDRESS>', port=18555))
INFO:squeaknode.network.peer_message_handler:Finished handle_msgs
INFO:squeaknode.network.connection_manager:Current number of peers 0
INFO:squeaknode.network.connection_manager:Current peers:--------
INFO:squeaknode.network.connection_manager:--------------
INFO:squeaknode.network.peer:Stopping peer socket: <socket.socket fd=24, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('<IP_ADDRESS>', 47842), raddr=('<IP_ADDRESS>', 18555)>
INFO:squeaknode.node.process_received_payments_worker:Stopping ProcessReceivedPaymentsWorker...
INFO:squeaknode.node.payment_processor:Stopping payment processor task.

git stat	INFO:squeaknode.network.peer_client:Connecting to peer with address PeerAddress(host='<IP_ADDRESS>', port=18555)
INFO:squeaknode.network.peer_client:Making connection to PeerAddress(host='<IP_ADDRESS>', port=18555)
INFO:squeaknode.network.peer_client:Trying to connect socket to PeerAddress(host='<IP_ADDRESS>', port=18555)
INFO:squeaknode.network.peer:Sending msg msg_version(nVersion=70002 nServices=1 nTime=Sun Sep  5 21:30:33 2021 addrTo=CAddress(nTime=0 nServices=1 ip=<IP_ADDRESS> port=18555) addrFrom=CAddress(nTime=0 nServices=1 ip=<IP_ADDRESS> port=18555) nNonce=0xF9DB2622555F273A strSubVer=b'/python-squeak:0.7.3/' nStartingHeight=-1 fRelay=True) to Peer(PeerAddress(host='<IP_ADDRESS>', port=18555))
INFO:squeaknode.network.peer:Received msg msg_verack() from Peer(PeerAddress(host='<IP_ADDRESS>', port=18555))
INFO:squeaknode.network.peer:Received msg msg_version(nVersion=70002 nServices=1 nTime=Sun Sep  5 21:30:33 2021 addrTo=CAddress(nTime=0 nServices=1 ip=<IP_ADDRESS> port=39218) addrFrom=CAddress(nTime=0 nServices=1 ip=<IP_ADDRESS> port=18555) nNonce=0xA1DF085AA5F40607 strSubVer=b'/python-squeak:0.7.3/' nStartingHeight=-1 fRelay=1) from Peer(PeerAddress(host='<IP_ADDRESS>', port=18555))
INFO:squeaknode.network.peer:Sending msg msg_verack() to Peer(PeerAddress(host='<IP_ADDRESS>', port=18555))
INFO:squeaknode.network.connection_manager:Current number of peers 1
INFO:squeaknode.network.connection_manager:Current peers:--------
INFO:squeaknode.network.connection_manager:Peer(PeerAddress(host='<IP_ADDRESS>', port=18555))
INFO:squeaknode.network.connection_manager:--------------
INFO:squeaknode.network.peer:Sending msg msg_getaddr() to Peer(PeerAddress(host='<IP_ADDRESS>', port=18555))

The peer connection should not be accepted after shutdown, but that is not the root cause here.
The new "update squeak subscription" thread and "update secret key subscription" thread are still running after shutdown.

https://github.com/yzernik/squeaknode/pull/1207


Prepare PoC for RBAC execution support

Description
Changes proposed in this pull request:

Prepare PoC for RBAC execution support

This PR contains all source code changes related to RBAC. For instructions, see PR https://github.com/kubeshop/botkube/pull/910.
Video
https://user-images.githubusercontent.com/7155799/210326564-64210b6e-22d2-4200-88bb-1d6b342a404c.mp4
Related issue(s)
#873

Closing as it won't be merged to main.