k3s kube-system pods failure when using --bind-address

Environmental Info:
K3s Version:
latest version v1.30.3+k3s1

Node(s) CPU architecture, OS, and Version:
Just one master node x86_64 Debian 11

Cluster Configuration:

Describe the bug:
I have several servers that each has its own private IP tunneled over VPN
I would like to setup the master node and bind the k3s address to the VPN IP e.g <IP_ADDRESS> (a layer 2)
If I use the --bind-address=<IP_ADDRESS> the setup is done with no error but kube-system pods fail
kk get pod -n kube-system
NAME                                      READY   STATUS             RESTARTS      AGE
coredns-576bfc4dc7-fjlpk                  0/1     Running            0             97s
local-path-provisioner-6795b5f9d8-js27t   0/1     CrashLoopBackOff   3 (49s ago)   97s
metrics-server-557ff575fb-p8thv           0/1     CrashLoopBackOff   3 (40s ago)   97s

and see error in k3s service log
Aug 03 20:13:23 204 k3s[1387]: I0803 20:13:23.770137    1387 garbagecollector.go:826] "failed to discover some groups" groups="<internal error: json: unsupported type: map[schema.GroupVersion]error>"
Aug 03 20:13:23 204 k3s[1387]: time="2024-08-03T20:13:23+03:30" level=info msg="Connecting to proxy" url="wss://<IP_ADDRESS>:6443/v1-k3s/connect"
Aug 03 20:13:23 204 k3s[1387]: time="2024-08-03T20:13:23+03:30" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp <IP_ADDRESS>:6443: connect: connection refused"
Aug 03 20:13:23 204 k3s[1387]: time="2024-08-03T20:13:23+03:30" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp <IP_ADDRESS>:6443: connect: connection refused" url="wss://<IP_ADDRESS>:6443/v1-k3s/connect"

next I uninstall k3s and started a new setup with both

--bind-address=<IP_ADDRESS>
--advertise-address=<IP_ADDRESS>

Then kube-system pod got working (expect the metric)
kk get pods -n kube-system
NAME                                      READY   STATUS    RESTARTS       AGE
coredns-576bfc4dc7-fjlpk                  1/1     Running   0              10m
local-path-provisioner-6795b5f9d8-js27t   1/1     Running   11 (58s ago)   10m
metrics-server-557ff575fb-p8thv           0/1     Running   11 (58s ago)   10m

but still I see metric server error log
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  36s                default-scheduler  Successfully assigned kube-system/metrics-server-557ff575fb-ccrxq to 204
  Normal   Pulling    33s                kubelet            Pulling image "rancher/mirrored-metrics-server:v0.7.0"
  Normal   Pulled     30s                kubelet            Successfully pulled image "rancher/mirrored-metrics-server:v0.7.0" in 3.477s (3.477s including waiting). Image size: 19434712 bytes.
  Normal   Created    30s                kubelet            Created container metrics-server
  Normal   Started    30s                kubelet            Started container metrics-server
  Warning  Unhealthy  29s (x2 over 29s)  kubelet            Readiness probe failed: Get "https://<IP_ADDRESS>:10250/readyz": dial tcp <IP_ADDRESS>:10250: connect: connection refused
  Warning  Unhealthy  1s (x16 over 28s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500

Steps To Reproduce:
run
curl -sfL https://get.k3s.io | sh -s - server --cluster-init --bind-address=<IP_ADDRESS> --disable=traefik

or
curl -sfL https://get.k3s.io | sh -s - server --cluster-init --bind-address=<IP_ADDRESS> --advertise-address=<IP_ADDRESS> --disable=traefik

Expected behavior:
I need a working cluster without exposed api-server to public Internet, I run k3s on VMs that have public IPs but all VMs also connected via VPN and each one has its own private IP (it is not a cloud)

Actual behavior:
from other location (servers connected the VPN network) I can access (e.g from home) to the cluster via private IP (e.g <IP_ADDRESS>) but seems there is a bug or I am setting it up incorrectly

Additional context / logs:
the kube-config address:
server: https://<IP_ADDRESS>:6443

but etcd configs shows the VM public IP has been set
/var/lib/rancher/k3s/server/db/etcd/config:advertise-client-urls: https://<IP_ADDRESS>:2379
/var/lib/rancher/k3s/server/db/etcd/config:initial-advertise-peer-urls: https://<IP_ADDRESS>:2380
/var/lib/rancher/k3s/server/db/etcd/config:initial-cluster: 204-5f774753=https://<IP_ADDRESS>:2380
/var/lib/rancher/k3s/server/db/etcd/config:listen-client-urls: https://<IP_ADDRESS>:2379,https://<IP_ADDRESS>:2379
/var/lib/rancher/k3s/server/db/etcd/config:listen-peer-urls: https://<IP_ADDRESS>:2380,https://<IP_ADDRESS>:2380
etc
etc

I did a new test with --bind-address and see the same error
kubectl get pods -n kube-system
NAME                                      READY   STATUS             RESTARTS        AGE
coredns-576bfc4dc7-9k8m2                  0/1     Running            0               20m
helm-install-traefik-crd-glb9q            0/1     CrashLoopBackOff   8 (4m28s ago)   20m
helm-install-traefik-xsmml                0/1     CrashLoopBackOff   8 (4m42s ago)   20m
local-path-provisioner-6795b5f9d8-vqx5j   0/1     CrashLoopBackOff   8 (4m38s ago)   20m
metrics-server-557ff575fb-5k6w8           0/1     CrashLoopBackOff   8 (4m35s ago)   20m

further testing shows that for example this pod metrics-server-557ff575fb-5k6w8 error is because of TCP failure .
I can ping the pods IP but cannot netcat it
describe pod
Successfully assigned kube-system/metrics-server-557ff575fb-5k6w8 to 208
Pulling image "rancher/mirrored-metrics-server:v0.7.0"
Successfully pulled image "rancher/mirrored-metrics-server:v0.7.0" in 6.911s (6.911s including waiting). Image size: 19434712 bytes.
Readiness probe failed: Get "https://<IP_ADDRESS>:10250/readyz": dial tcp <IP_ADDRESS>:10250: connect: connection refused
Readiness probe failed: Get "https://<IP_ADDRESS>:10250/readyz": read tcp <IP_ADDRESS>:55902-><IP_ADDRESS>:10250: read: connection reset by peer
Created container metrics-server
Started container metrics-server
Container image "rancher/mirrored-metrics-server:v0.7.0" already present on machine
Readiness probe failed: Get "https://<IP_ADDRESS>:10250/readyz": read tcp <IP_ADDRESS>:57652-><IP_ADDRESS>:10250: read: connection reset by peer
Back-off restarting failed container metrics-server in pod metrics-server-557ff575fb-5k6w8_kube-system(a8c21f81-0a9e-4df5-967b-899035bb3236)

ping (okay)
ping -c4 <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=64 time=0.123 ms
64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=64 time=0.108 ms
64 bytes from <IP_ADDRESS>: icmp_seq=3 ttl=64 time=0.110 ms
64 bytes from <IP_ADDRESS>: icmp_seq=4 ttl=64 time=0.097 ms

netcat (fails)
nc -v <IP_ADDRESS> 10250
nc: connect to <IP_ADDRESS> port 10250 (tcp) failed: Connection refused

Have you tried just setting --node-ip and --flannel-iface instead of mucking about with  --advertise-address and --bind-address?

it is not MUCKING
When we are learning anything we have to go with trail-and-error
I am running k8s but recently I am looking for a lighter distro like yours to have multi-region multi-cluster dual-stack  (did not do it before)
For security reason and using my own overlay network (VPN) I waned all services listen on private IPs, thus I tried --bind-address so the cluster be configured for cross node communication via VPN network and no port be opened on any other IPs or Interfaces (just on VPN IP), I do not want any services listen to public IPs and I thought --bind-address is for this purpose
Yes after seeing errors , I tried the other way. I tested --flannel-iface and --node-external-ip , Because my setup is on-perm (no cloud) this way I can have flannel running on a private network and ingress-controller working with public ip (set by --node-external-ip)
And for disallowing open ports of k3s, I tested nftables and faced some issues that I mentioned here k3s nftables rule of ingress controller bypasses host nftables #10693
Any other software I have in mind (nginx, haproxy, ssh, etc) , when we set an IP to bind to, they listen on that IP and working fine. This is a common expectation of any command or option to bind an address. Thus if I did it wrong and it is not a bug excuse me for such a silly thing, you can close the issue. If it is a bug , let it be open
Thank you
I have not tried --node-ip yet , but I will try and update here

I am looking for a lighter distro like yours to have multi-region multi-cluster dual-stack

If this is what you're going for, I would suggest reading the docs at https://docs.k3s.io/networking/distributed-multicloud#embedded-k3s-multicloud-solution - hybrid clusters are hard to get right, and using the integrated wireguard/tailscale integration is your best bet.


Web -  "Uh-oh, something went wrong!" when trying to access the Receipt scan in progress…  chat  using the link

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Request money using UserA
Go to scan tab
Upload a receipt
Choose UserB
Click request
Click on the receipt and copy the link
Connect to the app using UserB and past the link

Expected Result:
"Hmm... it's not here" page should appear
Actual Result:
Uh-oh, something went wrong!" page is displayed
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: <IP_ADDRESS>
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
scan-progress.webm
https://github.com/Expensify/App/assets/93399543/4fd28139-559d-4022-b6fb-d497243ab680
Expensify/Expensify Issue URL:
Issue reported by: @hichamcc
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1693739581173179
View all open jobs on GitHub

I can't reproduce this one - gonna close!


Automated cherry pick of #1735: add verify storage mount point

Cherry pick of #1735 on release/2.10.0.
#1735: add verify storage mount point

Codecov Report

:exclamation: No coverage uploaded for pull request base (release/2.10.0@3b56ca9). Click here to learn what that means.
The diff coverage is 0%.

@@               Coverage Diff                @@
##             release/2.10.0   #1737   +/-   ##
================================================
  Coverage                  ?   7.67%
================================================
  Files                     ?     410
  Lines                     ?   62467
  Branches                  ?       0
================================================
  Hits                      ?    4796
  Misses                    ?   57261
  Partials                  ?     410

Flag
Coverage Δ

#aFlag
7.67% <0%> (?)

Impacted Files
Coverage Δ

pkg/compute/models/hostdrivers.go
18.18% <ø> (ø)

pkg/compute/models/hoststorages.go
0.64% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3b56ca9...9c14f2f. Read the comment docs.

/approve
/lgtm


feat(native-filters): Auto apply changes in FiltersConfigModal

SUMMARY

After user changes filters in FiltersConfig modal and clicks it automatically apply new changes to dashboard without need to click apply

Disable controls that affects dataMask in FiltersConfig modal until Default Value populates

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

https://user-images.githubusercontent.com/56388545/117250783-63338380-ae4c-11eb-83a1-08a970613180.mov
TEST PLAN

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

@simcha90 please include video in the PR🙏

@simcha90 is this PR only code refactoring? please include video in the PR if there's UI changes 🙏

done

@simcha90 @villebro Hi both, i would highly appreciate we look at PRs that are tagged "hold" together before merging.

My apologies, this was not intentional, I had missed this label 🙁  I'll look into adding a CI workflow to block merging if any restricted labels are present asap.


While running Testcafe-BDD test suite, the test run closes completely after running first feature

The test automation suite was developed using Test cafe in BDD framework.
The issue faced is that after running my first feature file, the test run closes completely,testcafe version used is v0.23.2.Issue is also seen in the same version.
While downgrading test cafe version v0.20.5.The suite run perfectly, but no snippets where being generated for my features which has no step definition, still the suite runs completely
Issue is seen across all platforms and browsers.
What is the Current behavior?
Currently the test run completes after running my first feature.
What is the Expected behavior?
I have almost 7 features and expecting to run all features and also to generate snippets for features which have no step definitions
The suite has been attached with issue PFA.Issue is seen while running attached suite.
Steps to Reproduce:
Go to project folder

run- npm install
2.Once dependencies are downloaded, run - npm run test-firefox
Suite fails after first feature run.

Error:
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>test-chrome: cucumber-js  --world-parameters "{\"browser\": \"chrome\"}"
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>test-chrome script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
Your Environment details:

testcafe version:  v0.23.2
node.js version: v10.10.0
command-line arguments: ' npm run test-{{browserName}}'
browser name and version:  safari v 11.1.2
platform and version:       masOS High Sierra v10.13.6

connect-ui-automation.zip

The test automation suite was developed using Test cafe in BDD framework.
The issue faced is that after running my first feature file, the test run closes completely,testcafe version used is v0.23.2.Issue is also seen in the same version.
While downgrading test cafe version v0.20.5.The suite run perfectly, but no snippets where being generated for my features which has no step definition, still the suite runs completely
Issue is seen across all platforms and browsers.
What is the Current behavior?
Currently the test run completes after running my first feature.
What is the Expected behavior?
I have almost 7 features and expecting to run all features and also to generate snippets for features which have no step definitions
The suite has been attached with issue PFA.Issue is seen while running attached suite.
Steps to Reproduce:
Go to project folder

run- npm install
2.Once depend ices are downloaded, run - npm run test-firefox
Suite fails after first feature run.

Error:
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>test-chrome: cucumber-js  --world-parameters "{\"browser\": \"chrome\"}"
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>test-chrome script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
Your Environment details:

testcafe version:  v0.23.2
node.js version: v10.10.0
command-line arguments: ' npm run test-{{browserName}}'
browser name and version:  safari v 11.1.2
platform and version:       masOS High Sierra v10.13.6

connect-ui-automation.zip

@shariqMohammed92
I checked your test project and I can't reproduce this error after upgrading TestCafe to the latest alpha version. Please try
"testcafe ":" 0.23.3-alpha.1"

and let us know if it solves the issue.


fix: add FPC to allowlist

Fix for test failing on master. One of my prior PRs that added validation went in and this test needed to updated to work with those changes :)

#5464  👈
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @alexghr and the rest of your teammates on  Graphite

Benchmark results
No metrics with a significant change found.

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Values are compared against data from master at commit ef7bf79b and shown if the difference exceeds 1%.
L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
8 txs
32 txs
64 txs

l1_rollup_calldata_size_in_bytes
676
676
676

l1_rollup_calldata_gas
6,412
6,400
6,412

l1_rollup_execution_gas
585,745
585,733
585,745

l2_block_processing_time_in_ms
1,311
4,704 (-1%)
9,256 (-4%)

note_successful_decrypting_time_in_ms
186 (-2%)
560 (+15%)
966 (-8%)

note_trial_decrypting_time_in_ms
89.1 (-8%)
79.4 (-24%)
96.6 (+46%)

l2_block_building_time_in_ms
13,775 (+1%)
50,524 (+1%)
100,278

l2_block_rollup_simulation_time_in_ms
7,920
27,649
54,569 (-2%)

l2_block_public_tx_process_time_in_ms
5,832 (+2%)
22,818 (+2%)
45,607 (+1%)

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric
5 blocks
10 blocks

node_history_sync_time_in_ms
14,135
27,319 (+2%)

note_history_successful_decrypting_time_in_ms
1,230 (-4%)
2,527 (+2%)

note_history_trial_decrypting_time_in_ms
68.2 (-30%)
182 (+19%)

node_database_size_in_bytes
18,681,936
35,106,896

pxe_database_size_in_bytes
29,859
59,414

Circuits stats
Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit
circuit_simulation_time_in_ms
circuit_input_size_in_bytes
circuit_output_size_in_bytes

private-kernel-init
229 (-1%)
44,379
28,214

private-kernel-ordering
204 (-1%)
52,880
14,296

base-parity
3,573 (+1%)
128
311

root-parity
1,316 (+4%)
1,244
311

base-rollup
16,535 (+1%)
165,760
861

root-rollup
49.7
4,359
725

private-kernel-inner
298 (-1%)
73,795
28,214

public-kernel-app-logic
122 (+2%)
35,252
28,217

public-kernel-tail
165 (+1%)
40,928
28,217

merge-rollup
8.35
2,568
861

public-kernel-teardown
117 (+1%)
35,252
28,217

public-kernel-setup
116 (+1%)
35,252
28,217

Tree insertion stats
The duration to insert a fixed batch of leaves into each tree type.

Metric
1 leaves
16 leaves
64 leaves
128 leaves
512 leaves
1024 leaves
2048 leaves
4096 leaves
32 leaves

batch_insert_into_append_only_tree_16_depth_ms
10.2 (+1%)
16.1
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_count
16.8
31.6
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_ms
0.592 (+1%)
0.496
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_32_depth_ms
N/A
N/A
47.4 (-1%)
73.0 (-5%)
232
447
876 (+1%)
1,740 (-1%)
N/A

batch_insert_into_append_only_tree_32_depth_hash_count
N/A
N/A
96.0
159
543
1,055
2,079
4,127
N/A

batch_insert_into_append_only_tree_32_depth_hash_ms
N/A
N/A
0.485 (-1%)
0.449 (-5%)
0.422
0.417
0.414 (+1%)
0.416
N/A

batch_insert_into_indexed_tree_20_depth_ms
N/A
N/A
55.4 (+1%)
109 (+2%)
341 (+1%)
671 (+2%)
1,314 (+1%)
2,623 (-1%)
N/A

batch_insert_into_indexed_tree_20_depth_hash_count
N/A
N/A
105
207
691
1,363
2,707
5,395
N/A

batch_insert_into_indexed_tree_20_depth_hash_ms
N/A
N/A
0.488 (+2%)
0.491 (+1%)
0.463 (+1%)
0.461 (+1%)
0.456 (+1%)
0.455 (-1%)
N/A

batch_insert_into_indexed_tree_40_depth_ms
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
62.1 (+1%)

batch_insert_into_indexed_tree_40_depth_hash_count
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
109

batch_insert_into_indexed_tree_40_depth_hash_ms
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
0.541 (+1%)

Miscellaneous
Transaction sizes based on how many contract classes are registered in the tx.

Metric
0 registered classes
1 registered classes

tx_size_in_bytes
14,968
495,320

Transaction size based on fee payment method

Metric
none fee payment method
native fee payment method
fpc_public fee payment method
fpc_private fee payment method

tx_with_fee_size_in_bytes
773
905
1,161
1,377

Transaction processing duration by data writes.

Metric
0 new note hashes
1 new note hashes
2 new note hashes

tx_pxe_processing_time_ms
2,094 (+1%)
1,381 (-1%)
4,708

Metric
0 public data writes
1 public data writes
2 public data writes
3 public data writes
4 public data writes
5 public data writes
8 public data writes

tx_sequencer_processing_time_ms
12.1 (+1%)
706 (+1%)
531 (+2%)
1,448 (+2%)
582 (+1%)
2,494 (+1%)
670


Jetson orin agx

Hello,
its possible to install it on Jetson orin agx ? (its nvidia tpu) ubuntu aarch64 (arm64) ?
if yes, can you explain me how ??
thanks you

yes it is possible - altough I'm running it without docker. I'll add this to my to-do as I have few notes already that can be converted to docs instead

yes it is possible, I've tried this out on a board I have here - altough I'm running it without docker. I'll add this to my to-do as I have few notes already that can be converted to docs instead

Oh, thank you, that would be great, because I have to do a lab assignment on this for school, and we only have Jetson AGX Orin devices, running on aarch64 (arm64) with Ubuntu 20.04 (5.10.120-tegra), and using a PyTorch version specific to the Jetson. So, this is complicated.
Thank you very much, I look forward to your response eagerly. thanks :D

hello, I forgot to specify the PyTorch version that works with JetPack 5.1 on Nvidia's Jetson AGX devices. Here it is: export https://developer.download.nvidia.cn/compute/redist/jp/v511/pytorch/torch-2.0.0+nv23.05-cp38-cp38-linux_aarch64.whl.
sorry, thanks you very much

I apologize if I wasn't clear earlier. To summarize:
I'm trying to install localai on an NVIDIA Jetson AGX Orin.
This device operates on Ubuntu 20.04 (tegra 5.10.120), which is an ARM64 version.
The Jetson runs on Python 3.8, and I cannot upgrade to a newer version like Python 3.10 due to specific dependencies on this platform.
It uses a specific version of PyTorch that requires Python 3.8 and is optimized for CUDA on this platform. You can find this version of PyTorch here: [PyTorch link](https://developer.download.nvidia.cn/compute/redist/jp/v511/pytorch/torch-2.0.0+nv23.05-cp38-cp38-linux_aarch64.whl).
I aim to get guidance on making localai work with this particular PyTorch version and the Jetson.

Is it possible to add support for this configuration? If not, could you guide me through the necessary steps to extend the support on my own, such as creating a PR?
Thank you for your understanding.

Please leave it open - I'd like to get at this as I have some spare time

Oh okay, im sorry.
thanks you very much for you're help

@mudler any updates on running LocalAI within Docker on Jetson Orin?  Many thanks!

@FutureProofHomes there are no container images, however, I made it to work also on the Orin board. As @ToeiRei mentioned here, you can either build the core image, or just build it from source.
In my case, I build it from source on the board and setup the python environment manually. I've just added some sparse support to non-conda environment with my work in #1653 , should be easily extensible for diffusers generically. However, in the case of Intel GPUs it assumes the dependencies are already installed.
For the Orin AGX board, this is quite needed, as you need a specific torch version for the board, that limits slightly setup/automation for now, but maybe we can collect the requirements for the specific case. I don't have access to my board now, but will have next week - I can likely check what's installed there and report here.

From my notes: I've installed pytorch from here: https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048
however, at the time of setup there was only a compatible diffuser version, which was https://github.com/huggingface/transformers/releases/tag/v4.32.0 and diffusers-0.20.2 (didn't tested other versions)
I had also to set:
LD_PRELOAD=/lib/aarch64-linux-gnu/libGLdispatch.so.0
LD_LIBRARY_PATH=/usr/local/cuda/lib64/

while running the binary (for pytorch and diffusers to work).
In case ggml/llama.cpp compilation fails, here is a patch that I usually apply:
diff --git a/ggml.h b/ggml.h
index 255541d..368ac2e 100644
--- a/ggml.h
+++ b/ggml.h
@@ -212,9 +212,11 @@
 extern "C" {
 #endif

-#ifdef __ARM_NEON
+#if defined(__ARM_NEON) && !defined(__CUDACC__)
     // we use the built-in 16-bit float type
     typedef __fp16 ggml_fp16_t;
+#elif defined(__ARM_NEON) && defined(__CUDACC__)
+    typedef half ggml_fp16_t;
 #else
     typedef uint16_t ggml_fp16_t;
 #endif

Note you need cmake (quite a recent version), I'm not sure if it was not available, so I have this in my notes too:
# Install CMAKE:
wget https://github.com/Kitware/CMake/releases/download/v3.26.4/cmake-3.26.4.tar.gz
tar xvf cmake-3.26.4.tar.gz
cd cmake-3.26.4
./configure
make
make install
# Install golang
wget https://go.dev/dl/go1.20.4.linux-arm64.tar.gz
rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.20.4.linux-arm64.tar.gz
# Build LocalAI
export PATH=/usr/local/go/bin:$PATH
export PATH=/usr/local/cuda/bin/:$PATH
make BUILD_TYPE=cublas build

@FutureProofHomes Small hint: you'd have to actually remove the conda install around line 67 in the Dockerfile to actually get past that point.

Thanks for the tips everyone.  I really want/need to get this up and running inside a container.   I'm gonna keep plugging away and if I pull this off I'll share the solution.  For anyone who has pulled this off, please share!

Okay, I set my Dockerfile to only build core and the container runs perfectly, but only powered by Jetson's CPU (no GPU).
ARG IMAGE_TYPE=core

By the way, will the core build support OpenAI Functions?  Probably not, huh?
I then set out to enable GPU support by setting LD_LIBRARY,  LD_LIBRARY_PATH  & BUILD_TYPE in my .env file:
## Set number of threads.
## Note: prefer the number of physical cores. Overbooking the CPU degrades performance notably.
# THREADS=14

## Specify a different bind address (defaults to ":8080")
# ADDRESS=<IP_ADDRESS>:8080

## Default models context size
# CONTEXT_SIZE=512
#
## Define galleries.
## models will to install will be visible in `/models/available`
# GALLERIES=[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}]

## CORS settings
# CORS=true
# CORS_ALLOW_ORIGINS=*

## Default path for models
#
MODELS_PATH=/models

## Enable debug mode
DEBUG=true

## Disables COMPEL (Diffusers)
# COMPEL=0

## Enable/Disable single backend (useful if only one GPU is available)
# SINGLE_ACTIVE_BACKEND=true

## Specify a build type. Available: cublas, openblas, clblas.
## cuBLAS: This is a GPU-accelerated version of the complete standard BLAS (Basic Linear Algebra Subprograms) library. It's provided by Nvidia and is part of their CUDA toolkit.
## OpenBLAS: This is an open-source implementation of the BLAS library that aims to provide highly optimized code for various platforms. It includes support for multi-threading and can be compiled to use hardware-specific features for additional performance. OpenBLAS can run on many kinds of hardware, including CPUs from Intel, AMD, and ARM.
## clBLAS:   This is an open-source implementation of the BLAS library that uses OpenCL, a framework for writing programs that execute across heterogeneous platforms consisting of CPUs, GPUs, and other processors. clBLAS is designed to take advantage of the parallel computing power of GPUs but can also run on any hardware that supports OpenCL. This includes hardware from different vendors like Nvidia, AMD, and Intel.
BUILD_TYPE=cublas

## Uncomment and set to true to enable rebuilding from source
REBUILD=true

## Enable go tags, available: stablediffusion, tts
## stablediffusion: image generation with stablediffusion
## tts: enables text-to-speech with go-piper
## (requires REBUILD=true)
#
# GO_TAGS=stablediffusion

## Path where to store generated images
# IMAGE_PATH=/tmp

## Specify a default upload limit in MB (whisper)
# UPLOAD_LIMIT

## List of external GRPC backends (note on the container image this variable is already set to use extra backends available in extra/)
# EXTERNAL_GRPC_BACKENDS=my-backend:<IP_ADDRESS>:9000,my-backend2:/usr/bin/backend.py

### Advanced settings ###
### Those are not really used by LocalAI, but from components in the stack ###
##
### Preload libraries
#LD_PRELOAD=
LD_PRELOAD=/lib/aarch64-linux-gnu/libGLdispatch.so.0
LD_LIBRARY_PATH=/usr/local/cuda/lib64/

### Huggingface cache for models
# HUGGINGFACE_HUB_CACHE=/usr/local/huggingface

### Python backends GRPC max workers
### Default number of workers for GRPC Python backends.
### This actually controls wether a backend can process multiple requests or not.
# PYTHON_GRPC_MAX_WORKERS=1

### Define the number of parallel LLAMA.cpp workers (Defaults to 1)
# LLAMACPP_PARALLEL=1

### Enable to run parallel requests
# PARALLEL_REQUESTS=true

### Watchdog settings
###
# Enables watchdog to kill backends that are inactive for too much time
# WATCHDOG_IDLE=true
#
# Enables watchdog to kill backends that are busy for too much time
# WATCHDOG_BUSY=true
#
# Time in duration format (e.g. 1h30m) after which a backend is considered idle
# WATCHDOG_IDLE_TIMEOUT=5m
#
# Time in duration format (e.g. 1h30m) after which a backend is considered busy
# WATCHDOG_BUSY_TIMEOUT=5m

Here is my models/luna.yml file.  By the way, should I set f16: true?  I did try this and it didn't solve my blocker though.
name: luna
parameters:
  model: luna-ai-llama2-uncensored.Q6_K.gguf
  top_k: 90
  temperature: 0.2
  top_p: 0.7
context_size: 4096
threads: 6
gpu_layers: 50
f16: false
mmap: true
backend: llama
roles:
  assistant: 'ASSISTANT:'
  system: 'SYSTEM:'
  user: 'USER:'
template:
  chat: lunademo-chat
  completion: lunademo-completion

I then run docker compose up -d --pull always and the container starts up.  When I check docker logs -f localai-api-1 i see the following error upon build:
make: *** [Makefile:514: backend-assets/grpc/bert-embeddings] Error 1
go mod edit -replace github.com/nomic-ai/gpt4all/gpt4all-bindings/golang=/build/sources/gpt4all/gpt4all-bindings/golang
go mod edit -replace github.com/donomii/go-rwkv.cpp=/build/sources/go-rwkv
go mod edit -replace github.com/ggerganov/whisper.cpp=/build/sources/whisper.cpp
go mod edit -replace github.com/ggerganov/whisper.cpp/bindings/go=/build/sources/whisper.cpp/bindings/go
go mod edit -replace github.com/go-skynet/go-bert.cpp=/build/sources/go-bert
go mod edit -replace github.com/mudler/go-stable-diffusion=/build/sources/go-stable-diffusion
go mod edit -replace github.com/M0Rf30/go-tiny-dream=/build/sources/go-tiny-dream
go mod edit -replace github.com/mudler/go-piper=/build/sources/go-piper
go mod download
touch prepare-sources
touch prepare
CGO_LDFLAGS="-lcublas -lcudart -L/usr/local/cuda/lib64/" C_INCLUDE_PATH=/build/sources/go-bert LIBRARY_PATH=/build/sources/go-bert \
go build -ldflags "-X "github.com/go-skynet/LocalAI/internal.Version=v2.9.0-39-ge022b59" -X "github.com/go-skynet/LocalAI/internal.Commit=e022b5959ea409586bcead3473bbe8c180b9d2bf"" -tags "" -o backend-assets/grpc/bert-embeddings ./backend/go/llm/bert/
# github.com/go-skynet/go-bert.cpp
In file included from gobert.cpp:6:
sources/go-bert/bert.cpp/bert.cpp: In function 'bert_ctx* bert_load_from_file(const char*)':
sources/go-bert/bert.cpp/bert.cpp:610:89: warning: format '%lld' expects argument of type 'long long int', but argument 5 has type 'int64_t' {aka 'long int'} [-Wformat=]
  610 |                 fprintf(stderr, "%s: tensor '%s' has wrong shape in model file: got [%lld, %lld], expected [%lld, %lld]\n",
      |                                                                                ~~~^
      |                                                                                |
      |                                                                                long long int
      |                                                                                %ld
  611 |                         __func__, name.data(), tensor->ne[0], tensor->ne[1], ne[0], ne[1]);
      |                                                ~~~~~~~~~~~~~
      |                                                            |
      |                                                            int64_t {aka long int}
sources/go-bert/bert.cpp/bert.cpp:610:95: warning: format '%lld' expects argument of type 'long long int', but argument 6 has type 'int64_t' {aka 'long int'} [-Wformat=]
  610 |                 fprintf(stderr, "%s: tensor '%s' has wrong shape in model file: got [%lld, %lld], expected [%lld, %lld]\n",
      |                                                                                ~~~^
      |                                                                                |
      |                                                                                long long int
      |                                                                                %ld
  611 |                         __func__, name.data(), tensor->ne[0], tensor->ne[1], ne[0], ne[1]);
      |                                                               ~~~~~~~~~~~~~
      |                                                                           |
      |                                                                           int64_t {aka long int}
sources/go-bert/bert.cpp/bert.cpp:610:112: warning: format '%lld' expects argument of type 'long long int', but argument 7 has type 'int64_t' {aka 'long int'} [-Wformat=]
  610 |                 fprintf(stderr, "%s: tensor '%s' has wrong shape in model file: got [%lld, %lld], expected [%lld, %lld]\n",
      |                                                                                ~~~^
      |                                                                                |
      |                                                                                long long int
      |                                                                                %ld
  611 |                         __func__, name.data(), tensor->ne[0], tensor->ne[1], ne[0], ne[1]);
      |                                                                              ~~~~~
      |                                                                                |
      |                                                                                int64_t {aka long int}
sources/go-bert/bert.cpp/bert.cpp:610:118: warning: format '%lld' expects argument of type 'long long int', but argument 8 has type 'int64_t' {aka 'long int'} [-Wformat=]
  610 |                 fprintf(stderr, "%s: tensor '%s' has wrong shape in model file: got [%lld, %lld], expected [%lld, %lld]\n",
      |                                                                                ~~~^
      |                                                                                |
      |                                                                                long long int
      |                                                                                %ld
  611 |                         __func__, name.data(), tensor->ne[0], tensor->ne[1], ne[0], ne[1]);
      |                                                                                ~~~~~
      |                                                                                |
      |                                                                                int64_t {aka long int}
sources/go-bert/bert.cpp/bert.cpp:624:37: warning: format '%lld' expects argument of type 'long long int', but argument 3 has type 'int64_t' {aka 'long int'} [-Wformat=]
  624 |                 printf("%24s - [%5lld, %5lld], type = %6s, %6.2f MB, %9zu bytes\n", name.data(), ne[0], ne[1], ftype_str[ftype], ggml_nbytes(tensor) / 1024.0 / 1024.0, ggml_nbytes(tensor));
      |                                 ~~~~^                                                            ~~~~~
      |                                     |                                                                |
      |                                     long long int                                                    int64_t {aka long int}
      |                                 %5ld
sources/go-bert/bert.cpp/bert.cpp:624:44: warning: format '%lld' expects argument of type 'long long int', but argument 4 has type 'int64_t' {aka 'long int'} [-Wformat=]
  624 |                 printf("%24s - [%5lld, %5lld], type = %6s, %6.2f MB, %9zu bytes\n", name.data(), ne[0], ne[1], ftype_str[ftype], ggml_nbytes(tensor) / 1024.0 / 1024.0, ggml_nbytes(tensor));
      |                                        ~~~~^                                                            ~~~~~
      |                                            |                                                                |
      |                                            long long int                                                    int64_t {aka long int}
      |                                        %5ld
sources/go-bert/bert.cpp/bert.cpp:655:101: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 6 has type 'long unsigned int' [-Wformat=]
  655 |                 fprintf(stderr, "%s: tensor '%s' has wrong size in model file: got %zu, expected %llu\n",
      |                                                                                ~~~^
      |                                                                                |
      |                                                                                long long unsigned int
      |                                                                                %lu
  656 |                         __func__, name.data(), ggml_nbytes(tensor), nelements * bpe);
      |                                                                     ~~~~~~~~~~~~~~~
      |                                                                               |
      |                                                                               long unsigned int
sources/go-bert/bert.cpp/bert.cpp:692:56: warning: format '%lld' expects argument of type 'long long int', but argument 4 has type 'int64_t' {aka 'long int'} [-Wformat=]
  692 |     printf("%s: mem_per_token %zd KB, mem_per_input %lld MB\n", __func__, new_bert->mem_per_token / (1 << 10), new_bert->mem_per_input / (1 << 20));
      |                                                     ~~~^                                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                        |                                                                               |
      |                                                        long long int                                                                   int64_t {aka long int}
      |                                                     %ld
# github.com/go-skynet/LocalAI/backend/go/llm/bert
/usr/local/go/pkg/tool/linux_arm64/link: running g++ failed: exit status 1
/usr/bin/ld: cannot find -lcublas: No such file or directory
/usr/bin/ld: cannot find -lcudart: No such file or directory
/usr/bin/ld: cannot find -lcublas: No such file or directory
/usr/bin/ld: cannot find -lcudart: No such file or directory
/usr/bin/ld: cannot find -lcublas: No such file or directory
/usr/bin/ld: cannot find -lcudart: No such file or directory
/usr/bin/ld: cannot find -lcublas: No such file or directory
/usr/bin/ld: cannot find -lcudart: No such file or directory
collect2: error: ld returned 1 exit status

make: *** [Makefile:514: backend-assets/grpc/bert-embeddings] Error 1

In summary, the only way I can get the container to build and run is if I comment out BUILD_TYPE=cublas from my .env and run via CPU.  Thanks for any tips on next steps @mudler.

Just asking: You have the nvidia docker parts done and all that?

Just asking: You have the nvidia docker parts done and all that?

@ToeiRei, I have successful run stable diffusion and llama.cpp from https://github.com/dusty-nv/jetson-containers repo, so I think I have all the prerequisites in place.
Anything specific you'd like me to check?  Thanks for help, btw.

FYI - this is the same issue I'm having: https://github.com/mudler/LocalAI/discussions/601.

Another bit of info:

Figured it out.  I needed to add the following to my bash profile.  Then rebuilt the container with no cache.
export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda/lib64\
                         ${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

Disregard the above.  I did NOT figure it out.  The hunt continues. Same error.   -I./bert.cpp/ggml/include/ggml/ -I./bert.cpp/ -I -O3 -DNDEBUG -std=c11 -fPIC -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -Wno-unused-function -pthread -mcpu=native
I CXXFLAGS: -I. -I./bert.cpp/ggml/include/ggml/ -I./bert.cpp/ -O3 -DNDEBUG -std=c++17 -fPIC -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -mcpu=native
I LDFLAGS:
I CMAKEFLAGS:
I CC:       cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
I CXX:      g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

make[1]: 'libgobert.a' is up to date.
make[1]: Leaving directory '/build/sources/go-bert'
CGO_LDFLAGS="-lcublas -lcudart -L/usr/local/cuda/lib64/" C_INCLUDE_PATH=/build/sources/go-bert LIBRARY_PATH=/build/sources/go-bert \
go build -ldflags "-X "github.com/go-skynet/LocalAI/internal.Version=v2.9.0-39-ge022b59" -X "github.com/go-skynet/LocalAI/internal.Commit=e022b5959ea409586bcead3473bbe8c180b9d2bf"" -tags "" -o backend-assets/grpc/bert-embeddings ./backend/go/llm/bert/
# github.com/go-skynet/go-bert.cpp
In file included from gobert.cpp:6:
sources/go-bert/bert.cpp/bert.cpp: In function 'bert_ctx* bert_load_from_file(const char*)':
sources/go-bert/bert.cpp/bert.cpp:610:89: warning: format '%lld' expects argument of type 'long long int', but argument 5 has type 'int64_t' {aka 'long int'} [-Wformat=]
  610 |                 fprintf(stderr, "%s: tensor '%s' has wrong shape in model file: got [%lld, %lld], expected [%lld, %lld]\n",
      |                                                                                ~~~^
      |                                                                                |
      |                                                                                long long int
      |                                                                                %ld
  611 |                         __func__, name.data(), tensor->ne[0], tensor->ne[1], ne[0], ne[1]);
      |                                                ~~~~~~~~~~~~~
      |                                                            |
      |                                                            int64_t {aka long int}
sources/go-bert/bert.cpp/bert.cpp:610:95: warning: format '%lld' expects argument of type 'long long int', but argument 6 has type 'int64_t' {aka 'long int'} [-Wformat=]
  610 |                 fprintf(stderr, "%s: tensor '%s' has wrong shape in model file: got [%lld, %lld], expected [%lld, %lld]\n",
      |                                                                                ~~~^
      |                                                                                |
      |                                                                                long long int
      |                                                                                %ld
  611 |                         __func__, name.data(), tensor->ne[0], tensor->ne[1], ne[0], ne[1]);
      |                                                               ~~~~~~~~~~~~~
      |                                                                           |
      |                                                                           int64_t {aka long int}
sources/go-bert/bert.cpp/bert.cpp:610:112: warning: format '%lld' expects argument of type 'long long int', but argument 7 has type 'int64_t' {aka 'long int'} [-Wformat=]
  610 |                 fprintf(stderr, "%s: tensor '%s' has wrong shape in model file: got [%lld, %lld], expected [%lld, %lld]\n",
      |                                                                                ~~~^
      |                                                                                |
      |                                                                                long long int
      |                                                                                %ld
  611 |                         __func__, name.data(), tensor->ne[0], tensor->ne[1], ne[0], ne[1]);
      |                                                                              ~~~~~
      |                                                                                |
      |                                                                                int64_t {aka long int}
sources/go-bert/bert.cpp/bert.cpp:610:118: warning: format '%lld' expects argument of type 'long long int', but argument 8 has type 'int64_t' {aka 'long int'} [-Wformat=]
  610 |                 fprintf(stderr, "%s: tensor '%s' has wrong shape in model file: got [%lld, %lld], expected [%lld, %lld]\n",
      |                                                                                ~~~^
      |                                                                                |
      |                                                                                long long int
      |                                                                                %ld
  611 |                         __func__, name.data(), tensor->ne[0], tensor->ne[1], ne[0], ne[1]);
      |                                                                                ~~~~~
      |                                                                                |
      |                                                                                int64_t {aka long int}
sources/go-bert/bert.cpp/bert.cpp:624:37: warning: format '%lld' expects argument of type 'long long int', but argument 3 has type 'int64_t' {aka 'long int'} [-Wformat=]
  624 |                 printf("%24s - [%5lld, %5lld], type = %6s, %6.2f MB, %9zu bytes\n", name.data(), ne[0], ne[1], ftype_str[ftype], ggml_nbytes(tensor) / 1024.0 / 1024.0, ggml_nbytes(tensor));
      |                                 ~~~~^                                                            ~~~~~
      |                                     |                                                                |
      |                                     long long int                                                    int64_t {aka long int}
      |                                 %5ld
sources/go-bert/bert.cpp/bert.cpp:624:44: warning: format '%lld' expects argument of type 'long long int', but argument 4 has type 'int64_t' {aka 'long int'} [-Wformat=]
  624 |                 printf("%24s - [%5lld, %5lld], type = %6s, %6.2f MB, %9zu bytes\n", name.data(), ne[0], ne[1], ftype_str[ftype], ggml_nbytes(tensor) / 1024.0 / 1024.0, ggml_nbytes(tensor));
      |                                        ~~~~^                                                            ~~~~~
      |                                            |                                                                |
      |                                            long long int                                                    int64_t {aka long int}
      |                                        %5ld
sources/go-bert/bert.cpp/bert.cpp:655:101: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 6 has type 'long unsigned int' [-Wformat=]
  655 |                 fprintf(stderr, "%s: tensor '%s' has wrong size in model file: got %zu, expected %llu\n",
      |                                                                                ~~~^
      |                                                                                |
      |                                                                                long long unsigned int
      |                                                                                %lu
  656 |                         __func__, name.data(), ggml_nbytes(tensor), nelements * bpe);
      |                                                                     ~~~~~~~~~~~~~~~
      |                                                                               |
      |                                                                               long unsigned int
sources/go-bert/bert.cpp/bert.cpp:692:56: warning: format '%lld' expects argument of type 'long long int', but argument 4 has type 'int64_t' {aka 'long int'} [-Wformat=]
  692 |     printf("%s: mem_per_token %zd KB, mem_per_input %lld MB\n", __func__, new_bert->mem_per_token / (1 << 10), new_bert->mem_per_input / (1 << 20));
      |                                                     ~~~^                                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                        |                                                                               |
      |                                                        long long int                                                                   int64_t {aka long int}
      |                                                     %ld
# github.com/go-skynet/LocalAI/backend/go/llm/bert
/usr/local/go/pkg/tool/linux_arm64/link: running g++ failed: exit status 1
/usr/bin/ld: cannot find -lcublas: No such file or directory
/usr/bin/ld: cannot find -lcudart: No such file or directory
/usr/bin/ld: cannot find -lcublas: No such file or directory
/usr/bin/ld: cannot find -lcudart: No such file or directory
/usr/bin/ld: cannot find -lcublas: No such file or directory
/usr/bin/ld: cannot find -lcudart: No such file or directory
/usr/bin/ld: cannot find -lcublas: No such file or directory
/usr/bin/ld: cannot find -lcudart: No such file or directory
collect2: error: ld returned 1 exit status

make: *** [Makefile:515: backend-assets/grpc/bert-embeddings] Error 1

Okay, I've gotten further.  I updated my Dockerfile to use Nvidia's recommended CUDA base image for Jetson.  I also updated ARG CUDA_MAJOR_VERSION to 12 since that is what comes built-in.
# Set to core only.
ARG IMAGE_TYPE=core

#Use specific Nvidia base image instead of vanilla Ubuntu
#ARG BASE_IMAGE=ubuntu:22.04
ARG BASE_IMAGE=nvcr.io/nvidia/l4t-cuda:12.2.2-devel-arm64-ubuntu22.04

# extras or core
FROM ${BASE_IMAGE} as requirements-core

USER root

ARG GO_VERSION=1.21.7
ARG BUILD_TYPE
#Set CUDA version to 12 since that is what comes with Nvidia base imaage/Jetson OS.
ARG CUDA_MAJOR_VERSION=12
ARG CUDA_MINOR_VERSION=7
ARG TARGETARCH
ARG TARGETVARIANT

ENV BUILD_TYPE=${BUILD_TYPE}
ENV DEBIAN_FRONTEND=noninteractive
ENV EXTERNAL_GRPC_BACKENDS="coqui:/build/backend/python/coqui/run.sh,huggingface-embeddings:/build/backend/python/se>

ARG GO_TAGS="stablediffusion tinydream tts"

RUN apt-get update && \
    apt-get install -y ca-certificates curl patch pip cmake git && apt-get clean

It's worth mentioning that if I just run docker compose up -d again that LocalAI starts:
I local-ai build info:
I BUILD_TYPE: cublas
I GO_TAGS: stablediffusion
I LD_FLAGS: -X "github.com/go-skynet/LocalAI/internal.Version=v2.9.0-39-ge022b59" -X "github.com/go-skynet/LocalAI/internal.Commit=e022b5959ea409586bcead3473bbe8c180b9d2bf"
CGO_LDFLAGS="-lcublas -lcudart -L/usr/local/cuda/lib64/" go build -ldflags "-X "github.com/go-skynet/LocalAI/internal.Version=v2.9.0-39-ge022b59" -X "github.com/go-skynet/LocalAI/internal.Commit=e022b5959ea409586bcead3473bbe8c180b9d2bf"" -tags "stablediffusion" -o local-ai ./
7:07AM DBG no galleries to load
7:07AM INF Starting LocalAI using 6 threads, with models path: /models
7:07AM INF LocalAI version: v2.9.0-39-ge022b59 (e022b5959ea409586bcead3473bbe8c180b9d2bf)
7:07AM WRN [startup] failed resolving model '/usr/bin/local-ai'
7:07AM INF Preloading models from /models
7:07AM INF Model name: luna
7:07AM DBG Model: luna (config: {PredictionOptions:{Model:luna-ai-llama2-uncensored.Q6_K.gguf Language: N:0 TopP:0.7 TopK:90 Temperature:0.2 Maxtokens:0 Echo:false Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 TypicalP:0 Seed:0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:luna F16:false Threads:6 Debug:false Roles:map[assistant:ASSISTANT: system:SYSTEM: user:USER:] Embeddings:false Backend:llama TemplateConfig:{Chat:lunademo-chat ChatMessage: Completion:lunademo-completion Edit: Functions:} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: FunctionsConfig:{DisableNoAction:false NoActionFunctionName: NoActionDescriptionName: ParallelCalls:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:1 MMap:true MMlock:false LowVRAM:false Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:1024 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 MMProj: RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} VallE:{AudioPath:} CUDA:false DownloadFiles:[] Description: Usage:})
7:07AM DBG Extracting backend assets files to /tmp/localai/backend_data
7:07AM INF core/startup process completed!
7:07AM DBG No uploadedFiles file found at /tmp/localai/upload/uploadedFiles.json

 ┌───────────────────────────────────────────────────┐
 │                   Fiber v2.50.0                   │
 │               http://<IP_ADDRESS>:8080               │
 │       (bound on host <IP_ADDRESS> and port 8080)       │
 │                                                   │
 │ Handlers ........... 105  Processes ........... 1 │
 │ Prefork ....... Disabled  PID ............... 167 │
 └───────────────────────────────────────────────────┘

[<IP_ADDRESS>]:54492 200 - GET /readyz

But then when I hit the completions endpoint with:
curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
     "model": "luna",
     "prompt": "How fast is light?",
     "temperature": 0.1 }'

I get the below error:
7:10AM DBG Request received: {"model":"luna","language":"","n":0,"top_p":0,"top_k":0,"temperature":0.1,"max_tokens":0,"echo":false,"batch":0,"f16":false,"ignore_eos":false,"repeat_penalty":0,"n_keep":0,"mirostat_eta":0,"mirostat_tau":0,"mirostat":0,"frequency_penalty":0,"tfz":0,"typical_p":0,"seed":0,"negative_prompt":"","rope_freq_base":0,"rope_freq_scale":0,"negative_prompt_scale":0,"use_fast_tokenizer":false,"clip_skip":0,"tokenizer":"","file":"","response_format":{},"size":"","prompt":"How fast is light?","instruction":"","input":null,"stop":null,"messages":null,"functions":null,"function_call":null,"stream":false,"mode":0,"step":0,"grammar":"","grammar_json_functions":null,"backend":"","model_base_name":""}
7:10AM DBG `input`: &{PredictionOptions:{Model:luna Language: N:0 TopP:0 TopK:0 Temperature:0.1 Maxtokens:0 Echo:false Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 TypicalP:0 Seed:0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Context:context.Background.WithCancel Cancel:0xbaa10 File: ResponseFormat:{Type:} Size: Prompt:How fast is light? Instruction: Input:<nil> Stop:<nil> Messages:[] Functions:[] FunctionCall:<nil> Tools:[] ToolsChoice:<nil> Stream:false Mode:0 Step:0 Grammar: JSONFunctionGrammarObject:<nil> Backend: ModelBaseName:}
7:10AM DBG Parameter Config: &{PredictionOptions:{Model:luna-ai-llama2-uncensored.Q6_K.gguf Language: N:0 TopP:0.7 TopK:90 Temperature:0.1 Maxtokens:0 Echo:false Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 TypicalP:0 Seed:0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:luna F16:false Threads:6 Debug:true Roles:map[assistant:ASSISTANT: system:SYSTEM: user:USER:] Embeddings:false Backend:llama TemplateConfig:{Chat:lunademo-chat ChatMessage: Completion:lunademo-completion Edit: Functions:} PromptStrings:[How fast is light?] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: FunctionsConfig:{DisableNoAction:false NoActionFunctionName: NoActionDescriptionName: ParallelCalls:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:1 MMap:true MMlock:false LowVRAM:false Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:1024 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 MMProj: RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} VallE:{AudioPath:} CUDA:false DownloadFiles:[] Description: Usage:}
7:10AM DBG Template found, input modified to: Complete the following sentence: How fast is light?

7:10AM INF Loading model 'luna-ai-llama2-uncensored.Q6_K.gguf' with backend llama
7:10AM DBG llama-cpp is an alias of llama-cpp
7:10AM DBG Loading model in memory from file: /models/luna-ai-llama2-uncensored.Q6_K.gguf
7:10AM DBG Loading Model luna-ai-llama2-uncensored.Q6_K.gguf with gRPC (file: /models/luna-ai-llama2-uncensored.Q6_K.gguf) (backend: llama-cpp): {backendString:llama model:luna-ai-llama2-uncensored.Q6_K.gguf threads:6 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0x4000234600 externalBackends:map[autogptq:/build/backend/python/autogptq/run.sh bark:/build/backend/python/bark/run.sh coqui:/build/backend/python/coqui/run.sh diffusers:/build/backend/python/diffusers/run.sh exllama:/build/backend/python/exllama/run.sh exllama2:/build/backend/python/exllama2/run.sh huggingface-embeddings:/build/backend/python/sentencetransformers/run.sh mamba:/build/backend/python/mamba/run.sh petals:/build/backend/python/petals/run.sh sentencetransformers:/build/backend/python/sentencetransformers/run.sh transformers:/build/backend/python/transformers/run.sh transformers-musicgen:/build/backend/python/transformers-musicgen/run.sh vall-e-x:/build/backend/python/vall-e-x/run.sh vllm:/build/backend/python/vllm/run.sh] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
7:10AM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/llama-cpp
7:10AM DBG GRPC Service for luna-ai-llama2-uncensored.Q6_K.gguf will be running at: '<IP_ADDRESS>:46509'
7:10AM DBG GRPC Service state dir: /tmp/go-processmanager1266993544
7:10AM DBG GRPC Service Started
7:10AM DBG GRPC(luna-ai-llama2-uncensored.Q6_K.gguf-<IP_ADDRESS>:46509): stdout Server listening on <IP_ADDRESS>:46509
7:10AM DBG GRPC Service Ready
7:10AM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:luna-ai-llama2-uncensored.Q6_K.gguf ContextSize:1024 Seed:0 NBatch:512 F16Memory:false MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:1 MainGPU: TensorSplit: Threads:6 LibrarySearchPath: RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/models/luna-ai-llama2-uncensored.Q6_K.gguf Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type:}
7:10AM DBG GRPC(luna-ai-llama2-uncensored.Q6_K.gguf-<IP_ADDRESS>:46509): stderr ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
7:10AM DBG GRPC(luna-ai-llama2-uncensored.Q6_K.gguf-<IP_ADDRESS>:46509): stderr ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
7:10AM DBG GRPC(luna-ai-llama2-uncensored.Q6_K.gguf-<IP_ADDRESS>:46509): stderr ggml_init_cublas: found 1 CUDA devices:
7:10AM DBG GRPC(luna-ai-llama2-uncensored.Q6_K.gguf-<IP_ADDRESS>:46509): stderr   Device 0: Orin, compute capability 8.7, VMM: yes
7:10AM DBG GRPC(luna-ai-llama2-uncensored.Q6_K.gguf-<IP_ADDRESS>:46509): stderr CUDA error: the resource allocation failed
7:10AM DBG GRPC(luna-ai-llama2-uncensored.Q6_K.gguf-<IP_ADDRESS>:46509): stderr   current device: 0, in function ggml_init_cublas at /build/backend/cpp/llama/llama.cpp/ggml-cuda.cu:8831
7:10AM DBG GRPC(luna-ai-llama2-uncensored.Q6_K.gguf-<IP_ADDRESS>:46509): stderr   cublasCreate_v2(&g_cublas_handles[id])
7:10AM DBG GRPC(luna-ai-llama2-uncensored.Q6_K.gguf-<IP_ADDRESS>:46509): stderr GGML_ASSERT: /build/backend/cpp/llama/llama.cpp/ggml-cuda.cu:255: !"CUDA error"
[<IP_ADDRESS>]:45940 500 - POST /v1/completions

@mudler any tips to get this working?  Thnx! ^^^^

Watching this thread with interest as keen to pick up a Jetson Orin Nano 8GB for low power always available local AI workloads - primarily a voice pipeline from Home Assistant. So hopefully whatever you figure out here @FutureProofHomes will apply to the Orin Nano. Keep at it!

Okay, I've gotten further.
I updated my Dockerfile to use Nvidia's recommended CUDA base image for Jetson. I also updated ARG CUDA_MAJOR_VERSION to 12 since that is what comes built-in Nvidia's base image. NOTE: I'm only buildling Core.
Dockerfile:
# Set to core only.
ARG IMAGE_TYPE=core

#Use specific Nvidia base image instead of vanilla Ubuntu
#ARG BASE_IMAGE=ubuntu:22.04
ARG BASE_IMAGE=nvcr.io/nvidia/l4t-cuda:12.2.2-devel-arm64-ubuntu22.04

# extras or core
FROM ${BASE_IMAGE} as requirements-core

USER root

ARG GO_VERSION=1.21.7
ARG BUILD_TYPE
#Set CUDA version to 12 since that is what comes with Nvidia base imaage/Jetson OS.
ARG CUDA_MAJOR_VERSION=12
ARG CUDA_MINOR_VERSION=7
ARG TARGETARCH
ARG TARGETVARIANT

ENV BUILD_TYPE=${BUILD_TYPE}
ENV DEBIAN_FRONTEND=noninteractive
ENV EXTERNAL_GRPC_BACKENDS="coqui:/build/backend/python/coqui/run.sh,huggingface-embeddings:/build/backend/python/se>

ARG GO_TAGS="stablediffusion tinydream tts"

RUN apt-get update && \
    apt-get install -y ca-certificates curl patch pip cmake git && apt-get clean
.
.
.
.

Below is my docker-compose file:
version: '3.6'

services:
  api:
    image: localai:latest
    build:
      context: .
      dockerfile: Dockerfile
      platforms:
        - "linux/arm64"
    ports:
      - 8080:8080
    env_file:
      - .env
    volumes:
      - ./models:/models:cached
      - ./images/:/tmp/generated/images/
    command: ["/usr/bin/local-ai" ]
    runtime: nvidia

Below is my .env file:
## Set number of threads.
## Note: prefer the number of physical cores. Overbooking the CPU degrades performance notably.
# THREADS=14

## Specify a different bind address (defaults to ":8080")
# ADDRESS=<IP_ADDRESS>:8080

## Default models context size
# CONTEXT_SIZE=512
#
## Define galleries.
## models will to install will be visible in `/models/available`
# GALLERIES=[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}]

## CORS settings
# CORS=true
# CORS_ALLOW_ORIGINS=*

## Default path for models
#
MODELS_PATH=/models

## Enable debug mode
DEBUG=true

## Disables COMPEL (Diffusers)
# COMPEL=0

## Enable/Disable single backend (useful if only one GPU is available)
# SINGLE_ACTIVE_BACKEND=true

## Specify a build type. Available: cublas, openblas, clblas.
## cuBLAS: This is a GPU-accelerated version of the complete standard BLAS (Basic Linear Algebra Subprograms) library. It's provided by Nvidia and is part of their CUDA toolkit.
## OpenBLAS: This is an open-source implementation of the BLAS library that aims to provide highly optimized code for various platforms. It includes support for multi-threading and can be compiled to use hardware-specific features for additional performance. OpenBLAS can run on many kinds of hardware, including CPUs from Intel, AMD, and ARM.
## clBLAS:   This is an open-source implementation of the BLAS library that uses OpenCL, a framework for writing programs that execute across heterogeneous platforms consisting of CPUs, GPUs, and other processors. clBLAS is designed to take advantage of the parallel computing power of GPUs but can also run on any hardware that supports OpenCL. This includes hardware from different vendors like Nvidia, AMD, and Intel.
BUILD_TYPE=cublas

## Uncomment and set to true to enable rebuilding from source
REBUILD=true

## Enable go tags, available: stablediffusion, tts
## stablediffusion: image generation with stablediffusion
## tts: enables text-to-speech with go-piper
## (requires REBUILD=true)
#
# GO_TAGS=stablediffusion

## Path where to store generated images
# IMAGE_PATH=/tmp

## Specify a default upload limit in MB (whisper)
# UPLOAD_LIMIT

## List of external GRPC backends (note on the container image this variable is already set to use extra backends available in extra/)
# EXTERNAL_GRPC_BACKENDS=my-backend:<IP_ADDRESS>:9000,my-backend2:/usr/bin/backend.py

### Advanced settings ###
### Those are not really used by LocalAI, but from components in the stack ###
##
### Preload libraries
#LD_PRELOAD=

#LD_PRELOAD=/lib/aarch64-linux-gnu/libGLdispatch.so.0
LD_LIBRARY_PATH=/usr/local/cuda/lib64/
CUDA_LIBPATH=/usr/local/cuda/lib64/

### Huggingface cache for models
# HUGGINGFACE_HUB_CACHE=/usr/local/huggingface

### Python backends GRPC max workers
### Default number of workers for GRPC Python backends.
### This actually controls wether a backend can process multiple requests or not.
# PYTHON_GRPC_MAX_WORKERS=1

### Define the number of parallel LLAMA.cpp workers (Defaults to 1)
# LLAMACPP_PARALLEL=1

### Enable to run parallel requests
# PARALLEL_REQUESTS=true

### Watchdog settings
###
# Enables watchdog to kill backends that are inactive for too much time
# WATCHDOG_IDLE=true
#
# Enables watchdog to kill backends that are busy for too much time
# WATCHDOG_BUSY=true
#
# Time in duration format (e.g. 1h30m) after which a backend is considered idle
# WATCHDOG_IDLE_TIMEOUT=5m
#
# Time in duration format (e.g. 1h30m) after which a backend is considered busy
# WATCHDOG_BUSY_TIMEOUT=5m

And below are my docker logs when my build fails after starting the container for the first time. Notice it fails when building Whisper:
# github.com/go-skynet/go-llama.cpp
binding.cpp: In function 'void llama_binding_free_model(void*)':
binding.cpp:613:5: warning: possible problem detected in invocation of 'operator delete' [-Wdelete-incomplete]
  613 |     delete ctx->model;
      |     ^~~~~~~~~~~~~~~~~
binding.cpp:613:17: warning: invalid use of incomplete type 'struct llama_model'
  613 |     delete ctx->model;
      |            ~~~~~^~~~~
In file included from sources/go-llama-ggml/llama.cpp/examples/common.h:5,
                 from binding.cpp:1:
sources/go-llama-ggml/llama.cpp/llama.h:70:12: note: forward declaration of 'struct llama_model'
   70 |     struct llama_model;
      |            ^~~~~~~~~~~
binding.cpp:613:5: note: neither the destructor nor the class-specific 'operator delete' will be called, even if they are declared when the class is defined
  613 |     delete ctx->model;
      |     ^~~~~~~~~~~~~~~~~
mkdir -p backend-assets/gpt4all
cp: cannot stat 'sources/gpt4all/gpt4all-bindings/golang/buildllm/*.dylib': No such file or directory
cp: cannot stat 'sources/gpt4all/gpt4all-bindings/golang/buildllm/*.dll': No such file or directory
CGO_LDFLAGS="-lcublas -lcudart -L/usr/local/cuda/lib64/" C_INCLUDE_PATH=/build/sources/gpt4all/gpt4all-bindings/golang/ LIBRARY_PATH=/build/sources/gpt4all/gpt4all-bindings/golang/ \
go build -ldflags "-X "github.com/go-skynet/LocalAI/internal.Version=v2.9.0-39-ge022b59" -X "github.com/go-skynet/LocalAI/internal.Commit=e022b5959ea409586bcead3473bbe8c180b9d2bf"" -tags "" -o backend-assets/grpc/gpt4all ./backend/go/llm/gpt4all/
CGO_LDFLAGS="-lcublas -lcudart -L/usr/local/cuda/lib64/" C_INCLUDE_PATH=/build/sources/go-rwkv LIBRARY_PATH=/build/sources/go-rwkv \
go build -ldflags "-X "github.com/go-skynet/LocalAI/internal.Version=v2.9.0-39-ge022b59" -X "github.com/go-skynet/LocalAI/internal.Commit=e022b5959ea409586bcead3473bbe8c180b9d2bf"" -tags "" -o backend-assets/grpc/rwkv ./backend/go/llm/rwkv
cd sources/whisper.cpp && make libwhisper.a
make[1]: Entering directory '/build/sources/whisper.cpp'
I whisper.cpp build info:
I UNAME_S:  Linux
I UNAME_P:  aarch64
I UNAME_M:  aarch64
I CFLAGS:   -I.              -O3 -DNDEBUG -std=c11   -fPIC -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -pthread -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/aarch64-linux/include
I CXXFLAGS: -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -pthread -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/aarch64-linux/include
I LDFLAGS:  -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/aarch64-linux/lib
I CC:       cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
I CXX:      g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

nvcc --forward-unknown-to-host-compiler -arch=native -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -pthread -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/aarch64-linux/include -Wno-pedantic -c ggml-cuda.cu -o ggml-cuda.o
CFLAGS   += -mcpu=native
make[1]: CFLAGS: No such file or directory
make[1]: *** [Makefile:216: ggml-cuda.o] Error 127
make[1]: Leaving directory '/build/sources/whisper.cpp'
make: *** [Makefile:237: sources/whisper.cpp/libwhisper.a] Error

Full Docker Compose Failure Logs Attached: DockerComposeBuildFail.txt

this looks like there is no nvcc binary found. Do you have the cuda bin in your path?
export PATH=/usr/local/cuda/bin/:$PATH

Hi all, just want to share my appreciation for everyone's efforts! I have done my best to follow and build myself on agx, but I may not be technical proficient to deduct exactly what needs to be done.
Does anyone have any plans to consolidate all the revelations into a step by step guide?
Would be much appreciated!


Using with Laravel Vapor

Is this a possible solution given I cannot access the server configuartion of Vapor...
I love the simplicity of this package but cannot figure out to run on this service.
Is it possible?

Closing as this is answered by issue #226


intellisense is not working with the new preview terminal

Windows Terminal version
v1.19.2682.0
Windows build number
No response
Other Software
No response
Steps to reproduce
use the settings.json and profile file and check the intellisense feature if it works
Expected Behavior
intellisense should work
Actual Behavior
intellisense does not work
profile.txt
settings.txt

Looks like you didn't add "experimental.enableShellCompletionMenu": true, or the { "command": { "action": "sendInput", "input": "\u001b[24~b" }, "keys": "ctrl+space", "name": "Trigger shell completions" } action to your settings.json. Can you try adding those and try again/?

i ahve added it to the profiles list in the settings.json, is that the incorrect place?

i completely misunderstood  the config and did something wrong initially.
it is now working.
Thank you.
closing the ticket.


refactor-example-auth

What does this PR do?
This is a new pull request as a replacement of my old pull request that suffered from merge conflicts.
This is a refactor of the Appwrite's Authentication example using newer technologies like Vite, TypeScript and demonstrating useful use cases like auth-based routing.
Test Plan
I tested this code locally with a Docker image of Appwrite.
Related PRs and Issues
https://github.com/appwrite/demos-for-react/pull/19
Have you read the Contributing Guidelines on issues?
Yes I did read the Contributing Guidelines.

Adding @codercatdev for review, but this should be considered a hacktoberfest PR so maybe Chrisy should merge it


Lineage Tree Annotation Offset

[In ~150 frames-long movie with ~15k segmented objects] Some lineage trees appear outside of the window (branch aligned to the left window wall, see pic below). Also, I noticed that some trees don't have the horizontal line showing, until I find a tree that has it shown, and from that moment on all trees show the line - even those that didn't before.

I think the single branches only get centred in the window after you find a tree that is branching - if you only click on a cell that isn't a part of a wider tree, the centring is off.

Also, sometimes the tree doesn't show at all - but I do not see any pattern here..

.

I've fixed this one in #78 now. The tree not showing at all is related to #68. Working on a fix for that too


Moved shared files to shared folder, in root dir, and symlink

There are numerous files that have the same content but are copied into each example dir. This is because we want all relevant files for a run to be in one directory.
To make things more maintainable we should create a shared folder in which we keep these files and then create symlinks into each example.

Currently the mechnasim of using symlinks has the following problems:

Docker has an unsolved bug that prevents it from using Dockerfiles that are symlinked: https://github.com/docker/compose/issues/7397

We cannot just use a symlinked compose.yml file and have the Dockerfile as a real file in the shared directory, cause this would mean we would have the "context" variable in the compose.yml file to be fixed.
If we ever want to use the shared Dockerfile in a subfolder that is deeper than the first level this directory traversal would fail.
Therefore the solution for now:

Put a Dockerfile and compose.yml that we can reuse in the shared folder
Put a explanaition in the README that the container should be build with the command: docker compose -f ../shared/stress/compose.yml build

We are using build.sh files in the respective folders now


improve map geocoder search usability by zooming to recommended viewport

resolves #436

shoot just merged, @ngottlieb can you follow up with @drwestco suggestions?

copy that, I'll make the changes and submit a new PR @drwestco @nburt


Android Auto/Automotive: Use CarIconSpan in row texts

With the new release 1.3.0-alpha1 of the car app library, it is now possible to add icons within the title and text of a Row. This means we don't need to resort to using emoji anymore for "parking" and "charging" icons, and we could also think about other ways how this can be useful. For example, we could also try using it for implementing #198.

When upgrading the car app library to 1.3.0, remember to add a check for car API level 5 before using setOnContentRefreshListener in the MapScreen


Create waterlevelreader.ino

Create a folder with your project name then add your file inside your project folder for any PR to be merged

Add a specific Readme.md file in your project folder. Use the demo template and fill necessary details.


Feature request - tray checkbox to enable sending

I would like to see a menu item added to the system tray right click menu- a checkbox that when checked enables sending. When not using my computer I'd like to easily be able to disable adrilight to free up cpu cycles for distributed computing projects, and to be able to easily enable whenever I want to watch or play something.

I did try to do this myself... I went so far as to install VS 2017 and figure out how to get a tray menu item added, but I couldn't figure out how to add a checkbox to toggle TransferEnabled. I think it's something to do with the interface or maybe a global variable, but I am just used to c programming for arduino which seems much simpler.
Anyway, I know that was useless information but I figured I'd mention that an option to enable random colors from the tray menu would be useful and also probably easy enough to add if already adding the enable sending menu item.
Thanks,
Ray

I would like a keyboard shortcut for both of the above, or even command line switches.

okay lets think this through as I am actively considering this feature.
tray icon menu: i will do this, wanted to have this myself sometimes
keyboard shortcut: system wide shortcut I guess? which one would you take?
command line switch: how? adrilight is already running. do you want to call it in the command line like adrilight disablesending ? do you expect the second adrilight instance to send this command to the first instance? or how would this use case work?

For the system-wide keyboard shortcut, my preference would be any of these:
^!{c}
^!{d}
^!{o}
^!{p}
Thanks for this, I'll be able to automate turning Adrilight on and off with my projector.

Is this ctrl+alt+c?
I guess you want to turn adrilight of when you turn the projector on?

Yes, sorry ctrl+alt+c. I just cut and paste from my Windows script.
Exactly, my monitor is right below my projector screen.
Thanks

@xaos11 I read your initial request again and was thinking about it. The way adrilight gets the actual pixel data from the desktop is based on changes. it always gets the full screen, but only if something has changes. so as long as your distributed computing apps are not display some kind of graph or something, adrilight will not get new screen content data anyway and will not use precious cpu cycles.

Yeah, when idle I notice negligible cpu/gpu usage. I do however find myself disabling sending for some more resource intensive games that tend not to share resources well, so I do think the feature would be useful otherwise. But it's not that much extra effort so I think I have just grown patient and don't mind the extra clicks.
Honestly at this point I would rather see effort going into something like API connectivity (like Hyperion and f.lux), that could be used to control phillips hue bulbs or in my case, AiLight compatible bulbs. I understand this is a lot of work though, just figured I'd mention it


Instance closed connection

Hello,
I'm having an issue with cloudsql proxy and so far I'm out of ideas on how to solve this. In k8s I'm running two pods in the same namespace and each pod has a cloudsql-proxy container running as a sidecar.
I have a single secret with the service account credentials and those credentials are passed to the cloudsql-proxy.
In one pod the cloudsql-proxy connects successfully:
2018/07/08 18:48:13 using credential file for authentication<EMAIL_ADDRESS>2018/07/08 18:48:13 Listening on <IP_ADDRESS>:5432 for project-id:europe-west1:prod-db
2018/07/08 18:48:13 Ready for new connections
2018/07/08 18:48:16 New connection for "project-id:europe-west1:prod-db"

But in the other pod:
2018/08/06 08:53:01 using credential file for authentication<EMAIL_ADDRESS>2018/08/06 09:25:01 Listening on <IP_ADDRESS>:5432 for project-id:europe-west1:prod-db
2018/08/06 09:25:01 Ready for new connections
2018/08/06 09:27:14 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:27:14 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:27:18 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:27:18 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:27:20 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:27:20 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:27:23 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:27:23 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:27:27 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:27:27 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:27:33 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:27:33 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:27:45 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:27:45 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:27:48 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:27:48 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:27:51 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:27:51 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:27:54 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:27:54 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:27:58 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:27:58 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:28:03 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:28:03 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:28:15 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:28:15 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:28:18 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:28:18 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:28:22 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:28:22 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:28:25 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:28:25 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:28:29 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:28:29 Instance project-id:europe-west1:prod-db closed connection
2018/08/06 09:28:33 New connection for "project-id:europe-west1:prod-db"
2018/08/06 09:28:33 Instance project-id:europe-west1:prod-db closed connection

Any hint?
Edit:
Just to add some more information. The pod that is able to establish a connection has two containers running, the cloudsql-proxy and another one that connects to it. The pod that has the issue has 4 containers, 1 running the cloudsql-proxy and 3 others that connect to it.

Looks like what would happen if you had the password wrong. What errors
occur in the application which is trying to access the database?

Closing - over 30 days without response. If this is still an issue please free free to reopen.

Hello,
I think this is linked: my container connects perfectly through the proxy, but I'm using an ORM which close and open connections on the fly (which may be a bad behaviour, I'm not sure).

Should this really be an "ERROR" level message ?


Implement conversion of fit parameter results to JSON compatible dictionary.

Added neurom.stats.FitResults namedtuple
Simplified examples.

Current coverage is 100.00%

Merging #121 into master will not affect coverage as of 2ca152b

@@            master    #121   diff @@
======================================
  Files           30      30
  Stmts          973     983    +10
  Branches         0       0
  Methods
======================================
+ Hit            973     983    +10
  Partial          0       0
  Missed           0       0

Review entire Coverage Diff as of 2ca152b

Powered by Codecov. Updated on successful CI builds.


[Proposal] Move the sample applications used in performance test from Perf repo to MVC repo.

So as to keep the application up-to-date, adding functional tests and prevent build break.

/cc @rynowak
Change of plan. I'll retire this bug. More samples will be move to Performance repo instead.


about batch size

I'm reproducing the performance using this repo but I didn't reproduce the reported resnet50 51.3 Box AP. I guess my batch size setting is wrong. I use LR 0.02, 1 image per gpu and 32 GPUs for resnet50, and I got 50.4 Box AP.
May I ask the image per GPU and GPU number for resnet50 and resnext101-32x4d?
thank you.

Hi, please take a look at the learning schedule/batch size here
https://github.com/joe-siyuan-qiao/DetectoRS/tree/master/configs/htc
In your case, you might want to change the base learning rate to 0.04 to use 32 images/batch. But I'm not sure how that will affect APs.


[main_0.7 cherry pick] Fix scope of window check in Segmented Control  (#1423)

Platforms Impacted

[x] iOS
[ ] macOS

Description of changes
Mostly #1423 with a touch of #1216 to remove usage of screen.roundToDevicePixels
Pull request checklist
This PR has considered:

[ ] Light and Dark appearances
[ ] iOS supported versions (all major versions greater than or equal current target deployment version)
[ ] VoiceOver and Keyboard Accessibility
[ ] Internationalization and Right to Left layouts
[ ] Different resolutions (1x, 2x, 3x)
[ ] Size classes and window sizes (iPhone vs iPad, notched devices, multitasking, different window sizes, etc)
[ ] iPad Pointer interaction
[ ] SwiftUI consumption (validation or new demo scenarios needed)
[ ] Objective-C exposure (provide it only if needed)

Microsoft Reviewers: Open in CodeFlow

before after screenshots? especially because it isn't the same cherry pick like getting rid of all the  screen.roundToDevicePixels usage


Add option to disable release_target_info

Describe the Feature
The release_target_info method attempts to reduce the peak memory footprint of the build, but at the cost of re-invoking the changed method. This isn't free. An option to disable this behavior could make for faster builds when developers are working on machines with plenty of memory.
Required information
Discussed directly with Bill.

Version of SCons
3.1.1

Version of Python
Python 3.6

Which python distribution if applicable (python.org, cygwin, anaconda, macports, brew,etc)
N/A

How you installed SCons
Vendored into project

What Platform are you on? (Linux/Windows and which version)
Linux Ubuntu 18.04

How to reproduce your issue?  Please include a small self contained reproducer. Likely a SConstruct should do for most issues.
Do a no-op build of mongodb and time it. Then make release_target_info into a no-op and retime. You should see that about 30 seconds of wall clock time is saved.

How you invoke scons (The command line you're using "scons --flags some_arguments")

python3 src/third_party/scons-3.1.1/scons.py --implicit-cache --build-fast-and-loose=on --dbg=on --opt=on --variables-files="/home/andrew/.scons/site_scons/mongo_custom_variables.py" --link-model=static --install-mode=hygienic -j24 CC=/usr/bin/clang-7 CXX=/usr/bin/clang++-7 install-all-meta CCFLAGS=-gsplit-dwarf --cache --debug=time
Additional context
Ideally, this would be controlled by a command line switch, and allow SetOption to configure the value so it can be programmatically changed from within a running SConstruct.
When running with doing the release:
142.63user 2.06system 2:24.66elapsed 100%CPU (0avgtext+0avgdata 824140maxresident)k
Virtual memory: ~12 GB
Peak RSS: ~805 MB
When running without doing the release:
128.64user 2.53system 2:11.16elapsed 100%CPU (0avgtext+0avgdata 843256maxresident)k
Virtual memory: ~12 GB
Peak RSS: 824 MB
It seems to save 20MB of memory but cost 15 seconds (I've seen as much as 30). That doesn't seem like a good trade-off on modern systems. I'm not sure it even does anything in our build.

Having thought about this a little more, if the memory savings for a MongoDB build (which is a fairly large project I think?) are all of 25 MB, I think I'd propose simply removing the release_target_info subsystem entirely. Memory is cheap.

we can't do that without a deprecation cycle.
Best way to do that would be:

Add a flag to control it
Change default to not free
Assuming no one has an issue, remove the option and remove the logic.

Well, release_target_info isn't really user facing is it? So I don't see why it couldn't just be removed. If there is worry that maybe some people have accessed the symbol despite it being internal (to monkey patch it or similar) then If it just became a no-op for a while that would still work too. I guess the keep_target_info attribute could be considered user facing, but in that case you could just deprecate that attribute and still just make the default be to not release memory. It seems like an implementation detail.

It's possible we've still got users stuck in 32bit world... where it could matter.  When this was implemented for some sample build I think it was saving 100mb's of peak size.. which kept it from swapping.
I'd rather (minimall) add the option and default off..
before removing the logic.. less painful to unravel

True. I sort of forgot that 32-bit exists but of course it does. You could add it as an option but default it differently depending on 32-bit vs 64-bit python.

We'd have to default to current behavior, and then deprecate to off by default or off on 64 bit only.

@bdbaddog - So what would the right sequence of activities look like for this ticket in order to achieve a deprecation cycle? Ideally it would look like something where initially (4.0?) there was a flag --release-target-info that defaulted to True. And then eventually, later, defaulted to False? Where and when does the deprecation happen?

We've got a wiki page on that :)
https://github.com/SCons/scons/wiki/DeprecationCycle
But yes as you said, add feature where default is current behavior. announce it's going to change in next release (4.1.0 probably, might 5.0.0 but I don't think so), then in that release flip default.
I'd guess at this point that few would note and complain about flipping the default, but I've been surprised before.

Hmmm, this feature is actually fairly recent... RELEASE 2.3.1. Although I guess that's a decade ago, so not so new.
Should be easy to put this in, the code already checks for self.attributes.keep_targetinfo so presumably (?) it's just a question of being able to set that.

Hmmm, this feature is actually fairly recent... RELEASE 2.3.1. Although I guess that's a decade ago, so not so new.
Should be easy to put this in, the code already checks for self.attributes.keep_targetinfo so presumably (?) it's just a question of being able to set that.

I think that's a per Node setting, the idea would be to totally skip releasing the info.
The logic was added a while back to reduce the maximum RAM footprint as machines were running out of memory (pre 64 bit, > 4G RAM common in a build machine days)

Darn, you're right (that it's per-node)

Also, just bikeshedding here, but the flag could also be something like --reduce-memory-usage or --optimize-for=[performance|memory] since there may be other places in the future where a performance vs. memory trade-off might make sense.

I"m not sure we'd need a deprecation cycle for this?
It's a undocumented internal function.
That said deleting it wholesale could impact users where memory is tight, not sure how many/if any still are in this situation.


Feature/kak/upgrade to xenial

Overview
Upgrade to support Xenial, which means switching the service to use systemd.
Closes #34.

Thanks for the review, @hectcastro. I've updated the PR.


Error after pressing button in menu

After pressing the "screw" button this error message appears

Also after pressing the button when body selected Fusion crashes

Got the same problem :(

Hello @FenixC4 @RZFX
I added the current verion b4e784d as v1.3.0 to the Autodesk store, it should be published soon. Could you check if the error still persists. I had it also in a previous version but now it is gone.
Thank you very much!

After your update everything is working, i think.


Bug: getsto_MP / omni_getsto   * doesn't work

Calling getsto_MP/omni_getsto  works for the individual hash/txid  but when trying to get the list of all outputs omnicore is not able to recognize the *  for all addresses:
getsto_MP 0937f162eda99e5aeeab550b26ea7cdd1322a3281fe1721b6ae9c8eb0eab374d
{  "error": null,  "id": null,  "result": {   "amount": "1.00000000", ...... } }
getsto_MP 0937f162eda99e5aeeab550b26ea7cdd1322a3281fe1721b6ae9c8eb0eab374d *
Error in getsto_MP RPC call: {u'message': u'Invalid address', u'code': -5}
getsto_MP 0937f162eda99e5aeeab550b26ea7cdd1322a3281fe1721b6ae9c8eb0eab374d "*"
Error in getsto_MP RPC call: {u'message': u'Invalid address', u'code': -5}
omni_getsto 0937f162eda99e5aeeab550b26ea7cdd1322a3281fe1721b6ae9c8eb0eab374d *
Error in omni_getsto RPC call: {u'message': u'Invalid address', u'code': -5}
This is used by omniEngine when parsing/updating its database balances.
@zathras-crypto or @dexX7 i'm hoping this is a simple bug to patch?
This is running compiled tip to 0b6ff8818983a06146bc75d620755ba831ee04d3

Thanks @achamely!
I can confirm this is an issue. I won't be able to push the fix immediately, but it's probably resolved within the next 24 hours.
@zathras-crypto: it likely affects all RPCs with "address or wildcard/empty" input, which already use the ParseX() functions, and we could add something like ParseAddressOrWildcard():

src/omnicore/rpcvalues.cpp#L14-L21
src/omnicore/rpc.cpp#L1496

We might also add ParseAddressOrEmpty() to simplify:

src/omnicore/rpc.cpp#L350-L351

std::string ParseAddressOrEmpty(const json_spirit::Value& value)
{
    if (value.get_str().empty()) {
        return "";
    }
    return ParseAddress(value);
}

std::string ParseAddressOrWildcard(const json_spirit::Value& value)
{
    if (value.get_str() == "*") {
        return "*";
    }
    return ParseAddress(value);
}

Confirmed behaviour also, nice catch!!!
@dexX7 yep that sounds like a plan, it's not feasible to just allow wildcards in ParseAddress since there are many places where a wildcard is not supported (such as a from address), so your approach sounds like the simplest way to handle it as we can be granular about whether we support a wildcard in the field :)

Fix pending via #213.
@achamely: after #213 I'd like to merge #207, which triggers a reparse (thanks to @zathras-crypto .. :). This should take about 10-25 minutes, but I just wanted to give you a heads up.

Sweet, thanks guys. will check tomorrow after 213 gets merged (and noted on the 207 reparse)

@achamely: fix merged. :)

thanks guys, i'm pulling / compiling now. will report back

Confirmed fixed and functional.  Thanks guys.
@dexX7 the reparse took a bit longer than 25 min fyi (closer to 2 hours) but that's probably due to the machine i'm working on

2 hours? Wow, that's a lot. Did you reparse recently, but before we added the block skipping and checkpointing? For me this cut the time in half, so I don't even want to imagine, hehe.. :)

(closer to 2 hours)

Wow, even with slow hardware the seed blocks should make it quicker than that - do you perhaps run with --omnidebug=all?

yes there had a been a fresh reparse a about a week ago and no, it wasn't running with omnidebug
at the time

For comparison on fairly underwhelming hardware a fresh parse for me is somewhere between 5 and 8 minutes depending on OS caching.  2 hours sounds like either blisteringly slow hardware or something is wrong :(


Restore service latency test & fix bugs

Ref #9555
Ref #9354

@k8s-bot test this please
I believe the test failure was a flake caused by overloading GitHub while starting up the Jenkins builder.

Comments addressed, PTAL.

Changed to not make 600MB log files.

Not a real e2e failure - this hit jenkinsci/ghprb-plugin#104.

shippable hit two different flakes... we really need to clean that up.

@quinton-hoole I can haz OK? this is blocking my other PR...

Oh yes.  My apologies!  LGTM!

Merging because it's purely test code, and a very useful signal.

Very easy to roll back if it fails.  I'll keep an eye on it this weekend.

@k8s-bot test this please

I'm demoting this to flaky, as it's failing ~70% of the runs on our Jenkins continuous integration with the following error:
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service_latency.go:99
Tail latency is > 4x median latency

I have reason to believe that this is a GCE issue rather than K8s.  I'll chat to you on Monday - pretty sure that we can sort it out.

My theory was rubbish.  Jenkins was misreporting this test as stable in some environments, whereas in actual fact it is flaky in all of them.  Back to you @lavalamp

Interesting, I will take a look. Why didn't I see any failures when I looked yesterday? Can you link to a failure?

It's in the flaky bucket.  I've sent you a link via email - it's a Google-internal link.

@lavalamp This is still failing ~70% of the time.  I'll open a separate issue to track fixing this.
Jenkins link:
job/kubernetes-e2e-gce-flaky/1162/testReport/(root)/Kubernetes%20e2e%20suite/Service_endpoints_latency_should_not_be_very_high/history/

Yes, I know how to fix it and have a ton of code locally, but I'm busy fixing the proxy.

No worries.  Just want to make sure that we don't lose track of it.


Add MarshalJSON methods to *Document / *Array

In some cases, we need JSON representation of Document/Array, not JSON-like like we currently have. For example, extension functions accept BSON or JSON, and logging their parameters in JSON would make it easier to copy and paste them from logs to the psql console.
Let's implement encoding/json.Marshaler on *Document and *Array.
Let's do the basic JSON for now, but document the fact that the exact JSON representation may change. For example, we might use Extended JSON v2 in the future.

if no one is working on it, I can pick this up!!

The only thing that is left is to update test cases: https://github.com/FerretDB/wire/blob/e3aefb92d1578600de5a3fee1b9e72eff1faf788/wirebson/bson_test.go#L948-L952