Make torch xla available on GPU

What does this PR do?
Make torch xla available on GPU. Users could run both native Torch training jobs and TorchXLA jobs within the same environment without needing to uninstall or reinstall TorchXLA.
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you read the contributor guideline,
Pull Request section?
[x] Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case. https://discuss.huggingface.co/t/should-we-optimize-the-logic-for-enabling-torchxla-in-a-gpu-environment/60008
[ ] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
[ ] Did you write any new necessary tests?

Who can review?
@muellerzr

The main changes:

Change is_tpu_available to is_torch_xla_available;
Add USE_TORCH_XLA to enable or disable torch_xla;
Use torch native autocast and torch_xla GradScaler for AMP on GPU;
Use xm.all_reduce instead of  accelerator.reduce to perform an in-place all-reduce for Data Parallel;
Avoid all-reduce twice by _set_sync_gradients(True);

hi anyone can help take a look? thanks.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Hi, @muellerzr . Sorry for the late reply. I spent some time to resolving the UT issue to ensure that all UTs passed.
I ran the UTs (using make test with 2 GPUs) in three different environments:

RUN_SLOW=1 make test: CUDA UT without torchxla installed.
USE_TORCH_XLA=0 RUN_SLOW=1 make test: CUDA UT with torchxla installed, while setting USE_TORCH_XLA to 0 (to disable torchxla).
RUN_SLOW=1 make test: TorchXLA UT with torchxla installed and USE_TORCH_XLA set to 1 (default value).

(I don't have a TPU at the moment. I'll run the TPU UTs as soon as I acquire the TPU environment.)
All the three UTs above were executed with RUN_SLOW=1, and all passed except for one: tests/test_cli.py::AccelerateLauncherTester::test_notebook_launcher.
It appears that this test has also failed in the main branch. Before merging this CI, this test is fine.
I'm currently investigating the reason for its failure.
Here is the failure error:
RUN_SLOW=1 USE_TORCH_XLA=0 python -u -m pytest -s -v tests/test_cli.py::AccelerateLauncherTester::test_notebook_launcher

E           accelerate.test_utils.testing.SubprocessCallException: Command `python /root/wangang.wa/code/anw_accelerate/accelerate/src/accelerate/test_utils/scripts/test_notebook.py` failed with the following error:
E
E           Test basic notebook can be ran
E           Launching training on 2 GPUs.
E           Traceback (most recent call last):
E             File "/root/wangang.wa/code/anw_accelerate/accelerate/src/accelerate/launchers.py", line 200, in notebook_launcher
E               start_processes(launcher, args=args, nprocs=num_processes, start_method="fork")
E             File "/root/wangang.wa/code/pytorch/torch/multiprocessing/spawn.py", line 202, in start_processes
E               while not context.join():
E             File "/root/wangang.wa/code/pytorch/torch/multiprocessing/spawn.py", line 163, in join
E               raise ProcessRaisedException(msg, error_index, failed_process.pid)
E           torch.multiprocessing.spawn.ProcessRaisedException:
E
E           -- Process 1 terminated with the following error:
E           Traceback (most recent call last):
E             File "/root/wangang.wa/code/pytorch/torch/multiprocessing/spawn.py", line 74, in _wrap
E               fn(i, *args)
E             File "/root/wangang.wa/code/anw_accelerate/accelerate/src/accelerate/utils/launch.py", line 563, in __call__
E               self.launcher(*args)
E             File "/root/wangang.wa/code/anw_accelerate/accelerate/src/accelerate/test_utils/scripts/test_notebook.py", line 13, in basic_function
E               print(f"PartialState:\n{PartialState()}")
E             File "/root/wangang.wa/code/anw_accelerate/accelerate/src/accelerate/state.py", line 233, in __init__
E               torch.cuda.set_device(self.device)
E             File "/root/wangang.wa/code/pytorch/torch/cuda/__init__.py", line 404, in set_device
E               torch._C._cuda_setDevice(device)
E             File "/root/wangang.wa/code/pytorch/torch/cuda/__init__.py", line 284, in _lazy_init
E               raise RuntimeError(
E           RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

Running that test standalone should not (and does not fail). Are you initializing CUDA somewhere now suddenly?
To verify this, in Jupyter on a GPU-enabled system do the following:
import torch
from accelerate import Accelerator, notebook_launcher

torch.cuda.is_initialized()

This should be False

Thanks for your reply, muellerzr.  Following your suggestion, I tried running your code, and torch.cuda.is_initialized() returned false in Jupyter. And I switched to another worker, but encountered the same issue. Perhaps I need to delve deeper into this problem.

Mostly likely yes. My best recommendation for doing so is running the contents of that notebook test inside Jupyter to see if that works. I’ll rerun it here locally in a moment to verify but the test should work.
What is your output of pip freeze?

 If you uninstall it does the test pass?

No, even after uninstalling bitsandbytes, the issue persists.

Thanks, zach. I’m currently experimenting with the accelerate main branch in pytorch official 2.1 image(pytorch/pytorch:2.1.0-cuda11.8-cudnn8-runtime), but the issue persists.
Could you please take a look at this script and see if there are any issues?
docker run --gpus all --net host --ipc host --shm-size 10G  -it --rm --cap-add=SYS_PTRACE pytorch/pytorch:2.1.0-cuda11.8-cudnn8-runtime bash -c 'apt-get update; \
apt-get install git -y ; \
git clone http://github.com/huggingface/accelerate; \
cd accelerate; \
pip install pytest; \
pip install -e .;\
RUN_SLOW=1 python -u -m pytest -s -v tests/test_cli.py::AccelerateLauncherTester::test_notebook_launcher;'

Following the incorrect prompt (To use CUDA with multiprocessing, you must use the 'spawn' start method), I changed the start_method from fork to spawn in the notebook_launcher function, and the test passed.

We can not do fork, it must be spawn :)
but yes I’ll take a look, it passed yesterday on main for me though.

I ran pytest -sv tests/test_cli.py::AccelerateLauncherTester

@anw90 btw I think I know what may be wrong and why it's failing. We can merge this safely after resolving the conflicts. Thanks for your patience and working with us. (Probably stemed from the needed solution in here: https://github.com/huggingface/accelerate/pull/2272)

Thanks for your reviewing. All code conflicts have been resolved, and all unit tests have passed except for the 'test_notebook_launcher'.
BTW, after following this blog, I managed to get a worker with 8 TPU-v2s and ran the unit tests on it. But the UT fails a lot with the accelerate code cloned from http://github.com/huggingface/accelerate main branch.
The commands are as follows:
# set up a fresh worker with TPUv2-8.
gcloud compute tpus tpu-vm create my-tpu \
--accelerator-type=v2-8 \
--version=tpu-ubuntu2204-base \
--zone=us-central1-b \
--project=PRJECT_ID

# ssh into the worker.
gcloud compute tpus tpu-vm ssh my-tpu --zone=us-central1-b

# Install torch, torchx-xla, and torchvision
pip install torch~=2.1.0 torch_xla[tpu]~=2.1.0 torchvision -f https://storage.googleapis.com/libtpu-releases/index.html
export PJRT_DEVICE=TPU

# check tpu devices
python -c 'import torch_xla; print(torch_xla._XLAC._xla_get_all_devices())'  # ['TPU:0', 'TPU:1', 'TPU:2', 'TPU:3', 'TPU:4', 'TPU:5', 'TPU:6', 'TPU:7']

# run tests
git clone http://github.com/huggingface/accelerate
cd accelerate/
pip install pytest transformers parameterized
pip install -e .
make test

The failed UTs are as follows:
=========================================================================================== short test summary info ============================================================================================
FAILED tests/test_accelerator.py::AcceleratorTester::test_env_var_device - AssertionError: 'xla:0' != 'cuda:64'
FAILED tests/test_accelerator.py::AcceleratorTester::test_save_load_model_use_pytorch - AttributeError: 'MpDeviceLoaderWrapper' object has no attribute 'dataset'
FAILED tests/test_accelerator.py::AcceleratorTester::test_save_load_model_use_safetensors - RuntimeError: Attempted to access the data pointer on an invalid python storage.
FAILED tests/test_accelerator.py::AcceleratorTester::test_save_load_model_with_hooks_use_pytorch - AttributeError: 'MpDeviceLoaderWrapper' object has no attribute 'dataset'
FAILED tests/test_accelerator.py::AcceleratorTester::test_save_load_model_with_hooks_use_safetensors - RuntimeError: Attempted to access the data pointer on an invalid python storage.
FAILED tests/test_accelerator.py::AcceleratorTester::test_save_model_offload_use_safetensors - safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge
FAILED tests/test_accelerator.py::AcceleratorTester::test_save_model_use_safetensors - safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge
FAILED tests/test_cli.py::AccelerateLauncherTester::test_accelerate_test - RuntimeError: 'accelerate test' failed with returncode 1
FAILED tests/test_grad_sync.py::SyncScheduler::test_gradient_sync_cpu_multi - torch.multiprocessing.spawn.ProcessRaisedException:
FAILED tests/test_grad_sync.py::SyncScheduler::test_gradient_sync_cpu_noop - torch.multiprocessing.spawn.ProcessRaisedException:
FAILED tests/test_optimizer.py::OptimizerTester::test_accelerated_optimizer_pickling - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_scheduler.py::SchedulerTester::test_lambda_scheduler_not_step_with_optimizer_single_process - torch.multiprocessing.spawn.ProcessRaisedException:
FAILED tests/test_scheduler.py::SchedulerTester::test_lambda_scheduler_steps_with_optimizer_multiprocess - torch.multiprocessing.spawn.ProcessRaisedException:
FAILED tests/test_scheduler.py::SchedulerTester::test_lambda_scheduler_steps_with_optimizer_single_process - torch.multiprocessing.spawn.ProcessRaisedException:
FAILED tests/test_scheduler.py::SchedulerTester::test_one_cycle_scheduler_not_step_with_optimizer_single_process - torch.multiprocessing.spawn.ProcessRaisedException:
FAILED tests/test_scheduler.py::SchedulerTester::test_one_cycle_scheduler_steps_with_optimizer_multiprocess - torch.multiprocessing.spawn.ProcessRaisedException:
FAILED tests/test_scheduler.py::SchedulerTester::test_one_cycle_scheduler_steps_with_optimizer_single_process - torch.multiprocessing.spawn.ProcessRaisedException:
FAILED tests/test_state_checkpointing.py::CheckpointTest_use_safetensors::test_automatic_loading - RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx
FAILED tests/test_state_checkpointing.py::CheckpointTest_use_safetensors::test_can_resume_training - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_state_checkpointing.py::CheckpointTest_use_safetensors::test_can_resume_training_checkpoints_relative_path - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_state_checkpointing.py::CheckpointTest_use_safetensors::test_can_resume_training_with_folder - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_state_checkpointing.py::CheckpointTest_use_safetensors::test_checkpoint_deletion - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_state_checkpointing.py::CheckpointTest_use_safetensors::test_invalid_registration - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_state_checkpointing.py::CheckpointTest_use_safetensors::test_with_save_limit - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_state_checkpointing.py::CheckpointTest_use_safetensors::test_with_scheduler - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_state_checkpointing.py::CheckpointTest_use_pytorch::test_automatic_loading - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_state_checkpointing.py::CheckpointTest_use_pytorch::test_can_resume_training - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_state_checkpointing.py::CheckpointTest_use_pytorch::test_can_resume_training_checkpoints_relative_path - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_state_checkpointing.py::CheckpointTest_use_pytorch::test_can_resume_training_with_folder - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_state_checkpointing.py::CheckpointTest_use_pytorch::test_checkpoint_deletion - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_state_checkpointing.py::CheckpointTest_use_pytorch::test_invalid_registration - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_state_checkpointing.py::CheckpointTest_use_pytorch::test_with_save_limit - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_state_checkpointing.py::CheckpointTest_use_pytorch::test_with_scheduler - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_tpu.py::MultiTPUTester::test_tpu - RuntimeError: '/usr/bin/python /root/accelerate/tests/xla_spawn.py --num_cores 8 /root/accelerate/src/accelerate/test_utils/scripts/test_script.py' failed with returncode 1
FAILED tests/test_tracking.py::CustomTrackerTestCase::test_init_trackers - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
FAILED tests/test_tracking.py::CustomTrackerTestCase::test_log - ValueError: AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass `cpu=True` to `Accelerator()`.
===================================================================== 36 failed, 98 passed, 138 skipped, 13 warnings in 142.35s (0:02:22) ======================================================================
make: *** [Makefile:27: test] Error 1

Perhaps we could address this in a separate PR.

Do you think we should make similar adjustments to the transformers library to enable TorchXLA on GPU. If it's necessary, I could submit a new PR for that later on.

Let's address this in a separate PR please.

The code conflict has been resolved and all UTs, including the test_notebook_launcher have passed.

@anw90 can you rebase from main one more time (it's the failing test currently) and then I'll run this on our runners to make sure everything looks solid before final reviews from myself and a few others (considering how big this PR is)

@anw90 can you rebase from main one more time (it's the failing test currently) and then I'll run this on our runners to make sure everything looks solid before final reviews from myself and a few others (considering how big this PR is)

Ok, done.

Hi @muellerzr , is there any new progress

Hi, I was on holiday. Will be taking a look at this soon (next few days)

@anw90 can you go ahead and fix the merge conflicts? Then we can go ahead and merge this in I think by now :)

The other failures are okay, can you run make style; make quality one last time and this will get merged today!

Sorry for the late reply due to the holidays.  The code is formatted, and all unit tests have passed.


Parser errors when using the sample docker compose file listed at https://opendistro.github.io/for-elasticsearch-docs/docs/install/docker/

Hi all am a newbie with docker, so when I attempted to create a docker-compose.yml file using the Sample Docker Compose File located at https://opendistro.github.io/for-elasticsearch-docs/docs/install/docker/ I got the following error:
support@guacamole-docker:~/open-distro-elasticsearch$ docker-compose up
ERROR: yaml.parser.ParserError: while parsing a block mapping
  in "./docker-compose.yml", line 1, column 1
expected <block end>, but found '<scalar>'
  in "./docker-compose.yml", line 1, column 14

The docker compose file I created is just a copy paste from https://opendistro.github.io/for-elasticsearch-docs/docs/install/docker/ (making sure the EOL was for Linux, which is our host)
support@guacamole-docker:~/open-distro-elasticsearch$ docker --version
Docker version 19.03.5, build 633a0ea838

support@guacamole-docker:~/open-distro-elasticsearch$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
support@guacamole-docker:~/open-distro-elasticsearch$

What I copied:
version: '3'
services:
  odfe-node1:
    image: amazon/opendistro-for-elasticsearch:1.2.0
    container_name: odfe-node1
    environment:
      - cluster.name=odfe-cluster
      - node.name=odfe-node1
      - discovery.seed_hosts=odfe-node1,odfe-node2
      - cluster.initial_master_nodes=odfe-node1,odfe-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - odfe-data1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - odfe-net
  odfe-node2:
    image: amazon/opendistro-for-elasticsearch:1.2.0
    container_name: odfe-node2
    environment:
      - cluster.name=odfe-cluster
      - node.name=odfe-node2
      - discovery.seed_hosts=odfe-node1,odfe-node2
      - cluster.initial_master_nodes=odfe-node1,odfe-node2
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - odfe-data2:/usr/share/elasticsearch/data
    networks:
      - odfe-net
  kibana:
    image: amazon/opendistro-for-elasticsearch-kibana:1.2.0
    container_name: odfe-kibana
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      ELASTICSEARCH_URL: https://odfe-node1:9200
      ELASTICSEARCH_HOSTS: https://odfe-node1:9200
    networks:
      - odfe-net

volumes:
  odfe-data1:
  odfe-data2:

networks:
  odfe-net:

Opened in the wrong repo I think,closing


.set('Authorization', 'Bearer perm: *********') doesn't work

I am managing YouTrack via http-client based on<EMAIL_ADDRESS>Following YouTrack guide:
const agent = require('superagent');
return agent
  .get('yt.*****.ru/rest/admin/project')
  .set('Authorization', 'Bearer perm:********************')
  .then(response => {
    console.log(response.status);
  })
  .catch(response => console.log(`Error ${response}`));

Expected: 200 -- Success
Actual: 401 -- Unauthorized.
It works correctly on 3.5.0 version

Does it work in 3.7.0?

Yes, It works in 3.8.0 & older.

So the problem is caused redirects to another domain. (you can use .on('redirect', cb) to inspect them).
Change in 3.8.1 is considered a security issue, because previously the Authorization would have been sent to every domain redirected to.
Superagent doesn't have a way to know which domains are supposed to get Authorization, so it plays safe and allows it only on the original request.


Use pkg/controller helpers

This is an experimental, ongoing, non-invasive rewrite of the image-reflector controller to use the controller helpers from https://github.com/fluxcd/pkg/pull/56.

Closing as abandoned; may be reattempted at some point.


thing about autohotkeys

Hi, idk if this is a problem, so i resolved to post.
When Tibialyzer are maximized and i press my hotkey to "Suspend", the hud show on Tibiaclient and the autohotkey stop.
But if the tibialyzer are minimized in hidden icons and i press my "Suspend" hotkey, doesnt show the Suspended Hud on tibia client.
EDIT### - i found another thing, if tibialyzer are minimized on hidden icons, my hotkey of (summary@, reset@) when i press doesnt work too.
thxx

Minimizing to tray has been replaced with regular minimizing, so this should be "fixed" because minimizing to tray is no longer possible.


Updated the ansible-lint github workflow

changed version: 3.6 to 3.7.13.

Closing pull request as project is being archived.


Show application icon when using sway window module

Demo:
https://user-images.githubusercontent.com/19961516/137642147-1f71a58e-a91c-41d0-8124-e40e895153e0.mp4
Signed-off-by: Felix Weilbach<EMAIL_ADDRESS>

@Alexays Thanks for this nice project. Could you please give me some feedback if that is a feature that might get accepted?

LGTM, can you just update man? :)

LGTM, can you just update man? :)

Updated;)

This is not documented in man? With "height": 20, in ~/.config/waybar/config, icon is not scaled down to fit this so the bar height changes between 20 when there is no icon to something bigger, but I want 20 to be the highest height. Lastly, is there an option to disable the icon?

Oh, interesting feature. Just updated waybar and suddenly the icons showed up, interesting.
I noticed that the icon for XWayland apps don't show up. Is this a known issue?

LGTM, can you just update man? :)

Updated;)

Doesn't look like that actually ever happened...

LGTM, can you just update man? :)

Updated;)

Doesn't look like that actually ever happened... Edit: ah, fixed by #1459.

Actually I misunderstood it and thought that the request was just to update (rebase) the pull request. Sorry for that.


Syntax errors that only occur on love.js

The affected code is:
...
                elseif side == 1 then
                    if id == 16 then lastcell.rot = (lastcell.rot + 1)%4 end
                    dir = (dir + 1)%4
HERE:           else goto stop end
            elseif id == 31 or id == 92 then
                if side == 0 or side == 2 then
                    if id == 31 then lastcell.rot = (lastcell.rot - 1)%4 end
                    dir = (dir - 1)%4
...

On the normal windows 64-bit love.exe from https://love2d.org the game loads up and looks like this:

The full love file can be found here
I am using compatibility mode with love.js if that makes any difference

I have fixed the issue in my particular game by replacing all the usages of goto with an alternative.

I don't think the issue should be closed however, as goto is an important part of lua syntax and should work.

Goto was only introduced in Lua 5.2. Luajit adopted the feature. Since there is no wasm support for luajit, the web target relies on Lua 5.1. There is no fix, short of getting luajit to target wasm. When using the web port make sure to stick to Lua 5.1 syntax.


Error when dropping column in SQL Server 2014

I have an error when dropping a column that has a foreign key in SQL Server 2014. It seems to be looking for the foreign key to drop it in sys.default_constraints but the key is in sys.foreign_keys. I'm not 100% sure this is a change in SQL Server 2014, but it's my current setup.
I Can work around the issue by executing the SQL to drop the constraint since I named it myself, but I think this is a bug.
I'll see about looking at the code to change it myself.
The code:
Delete.Column("FormId")
    .FromTable("PaperContract")
    .InSchema("PaperContract");

Migrate.exe : !!! An error occured executing the following sql:
At line:1 char:1
+ & "C:\dev\vooban\gppp\packages\FluentMigrator.<IP_ADDRESS>\tools\Migrate.e ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (!!! An error oc... following sql::String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

DECLARE @default sysname, @sql nvarchar(max);

-- get name of default constraint
SELECT @default = name
FROM sys.default_constraints
WHERE parent_object_id = object_id('[PaperContract].[PaperContract]
')
AND type = 'D'
AND parent_column_id = (
SELECT column_id
FROM sys.columns
WHERE object_id = object_id('[PaperContract].[PaperContract]')
AND name = 'FormId'
);

-- create alter table command to drop constraint as string and run it
SET @sql = N'ALTER TABLE [PaperContract].[PaperContract] DROP CONSTRAINT ' + @default;
EXEC sp_executesql @sql;

-- now we can finally drop column
ALTER TABLE [PaperContract].[PaperContract] DROP COLUMN [FormId];

The error was The object 'FK_PaperContract_FormId' is dependent on column 'FormId'.
ALTER TABLE DROP COLUMN FormId failed because one or more objects access this column.

!!! The object 'FK_PaperContract_FormId' is dependent on column 'FormId'.
ALTER TABLE DROP COLUMN FormId failed because one or more objects access this column.

I just figured out that I'm supposed to drop the foreign key with Delete.ForeignKey first...


License is missing

I want to package this project into Conda, Galaxy and Docker, we need to have a License to redistribute this software.
Thanks!

Oh it's MIT, correct? Can we put it into a LICENSE file into the root?

Added LICENSE.txt to root.

Awesome, thanks!


Find all confounding factors in case-control results

This script will use the 'find-ancestors' function

@mbernste And can we close this one?


Portfolio form section branch

Name field accepts up to 30 chars.
Email field accepts only emails.
Message field accepts up to 500 chars.
All fields have placeholder text.
All fields have the correct type.
A user cannot submit the form if one or more of the fields is empty.
I Collect data by using the Formspree service.
I used Flexbox to lay out elements on the page.
I followed the design as much as possible, using the templates in Figma.
I Implemented the button interactions.

Name field accepts up to 30 chars.
Email field accepts only emails.
Message field accepts up to 500 chars.
All fields have placeholder text.
All fields have the correct type.
A user cannot submit the form if one or more of the fields is empty.
I Collect data by using the Formspree service.
I used Flexbox to lay out elements on the page.
I followed the design as much as possible, using the templates in Figma.
I Implemented the button interactions.
I removed the live demo from the README file

I have added the "pressed state" for the buttons using :active. I have done this following the Figma template.

Thank you.


Assume a robot without a parent joint on its base is on the worldXY origin

Rather simplistic fix for robots that are not attached to any fixed joint, like world or similar.
What type of change is this?

[x] Bug fix in a backwards-compatible manner.
[ ] New feature in a backwards-compatible manner.
[ ] Breaking change: bug fix or new feature that involve incompatible API changes.

Checklist

[x] Add the change to the CHANGELOG.rst file in the Unreleased section under the most fitting heading: Added, Changed, Removed.
[x] Run all tests on your computer (i.e. invoke test).
[ ] If you add new functions/classes, check that:

[ ] Are available on a second-level import, e.g. compas_fab.robots.Configuration.
[ ] Add unit tests (especially important for algorithm implementations).

LGTM! ;)


Bump rootless install script up to 20.10.6

Signed-off-by: Stefan Scherer<EMAIL_ADDRESS>

@StefanScherer can you run the job to update it? (I don't know where/how to run that 😂)

Jenkins job https://ci-next.docker.com/teams-core/job/get.docker.com/job/upload-rootless-install/25/ got triggered and the script is uploaded automatically.


Hash sum mismatch

Cannot install dotnet in 18.04 . erro is Hash sum mismatch
Failed to fetch https://packages.microsoft.com/ubuntu/18.04/prod/pool/main/d/dotnet-runtime-3.1/dotnet-runtime-3.1.3-x64.deb  Hash Sum mismatch
General
cannot install dotnet

@dagood @leecow

@jlgpjong, can you please provide the output of all the commands you ran? I don't hit this issue on my machine, and there isn't enough info in this issue to figure it out.
/cc @NikolaMilosavljevic

got it fix by restarting the pc and redo installation dotnet3.1


ollama model not support tool calling

I use ollama model in langgraph multi-agent SupervisorAgent framework,
when I use API llm, that is give actual key and url, it can run successfully, but after changing to ollama server, can't call tools.
my code:
def get_qwen7b():
model = ChatOpenAI(
model_name="qwen2:7b",
openai_api_base="http://localhost:11434/v1",
openai_api_key="none",
streaming=True,
temperature=0.01
)
return model
llm = get_qwen7b()
class SupervisorAgent(BaseAgent):
def init(self, members, llm,  **kwargs):
system_prompt = (
"""You are a supervisor tasked with managing a conversation between the
following workers:  {members}. Given the following user request,
respond with the worker to act next. Each worker will perform a
task and respond with their results and status. When finished,
respond with FINISH."""
)
# supervisor is an LLM node. It just picks the next agent to process
options = ["FINISH"] + members
    function_def = {
        "name": "route",
        "description": "Select the next role.",
        "parameters": {
            "title": "routeSchema",
            "type": "object",
            "properties": {
                "next": {
                    "title": "Next",
                    "anyOf": [
                        {"enum": options},
                    ],
                }
            },
            "required": ["next"],
        },
    }

    prompt = ChatPromptTemplate.from_messages(
        [
            ("system", system_prompt),
            MessagesPlaceholder(variable_name="messages"),
            ("system",
             "Given the messages above, who should act next? Or should we FINISH? If get the final result, we should FINISH. Select one of: {options}"),
        ]
    ).partial(options=str(options), members=", ".join(members))
    tools = [convert_to_openai_tool(f) for f in [function_def]]
    model = llm.bind(tools=tools)
    self.agent = prompt | model | OutputParser()

that is, my agent framework is https://github.com/langchain-ai/langgraph/blob/main/examples/multi_agent/agent_supervisor.ipynb
I want to replace the llm from API to local ollama server. but failed in tool calling.

You need to use a model that supports tools.  Alternatively, if you want to use qwen2,modify the template to handle tool calls.

You need to use a model that supports tools. Alternatively, if you want to use qwen2,modify the template to handle tool calls.

if I use models who support tools, it also doesn't work but with other error, my framework is langgraph.
LLM.invoke(input_messages).content
^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'invoke'

This would seem to be a langgraph issue.  If you could provide a simple, standalone script that demonstrates your problem, it might be easier to diagnose the issue you are having.

Hi @rick-github, I would like to use my own fine-tuned model for tool calling in Ollama.
Do you happen to know if updating the template is enough for it to support tool calls? I've been having trouble finding information on how to make this possible with a custom model so I am kind of lost.
Is having a Jinja template like this enough for the Ollama chat completion to return a valid response_message.tool_calls? Or does the model need to generate a special format?
Thank you in advance

If you are fine-tuning using a base model that supports tool use, then using the template from the original model should work.
If the base model wasn't trained for tool use it's a bit trickier.  Before ollama supported it, tool use was done by explicitly injecting a system message, see https://github.com/ollama/ollama/issues/5793#issuecomment-2238938100.  With tool support, tool calls are a bit simpler because you can use the template to construct the message sent to the model as you point out.  Note that the template will need to be modified to account for tokens specific to a model, message layout, etc.  The new problem is trying to convert the result from the model into a tool_calls response.  If the response is well-formed JSON, then there's a good chance that it will be converted into a tool_calls structure.  So this really relies on two things: a very clear system message with examples to allow the model to properly format the response, and the ability of the model to follow instructions without adding extra verbiage.

Note that the templating language is not Jinja, it's Go templating

Thank you so much for your help and clarifying the templating language, I now understand  everything better.
Indeed, my model is based on Gemma so it doesn't support tool calling on its own. My main concern is with the model response: if it's a valid JSON, Ollama will handle the conversion to a tool_calls structure, correct? Is there anything else I need to implement?

This would seem to be a langgraph issue. If you could provide a simple, standalone script that demonstrates your problem, it might be easier to diagnose the issue you are having.

Thanks so much! it is indeed what you said!


video_gpu_accel_demo.py - TypeError: "tasks" must be a tuple object or a sequence object

Bug Description
When I try inferencing a video by running video_gpuaccel_demo.py, I am getting type errors. I wanted to see if the inference time for the video would improve
Reproduction

What command or script did you run?

python demo/video_gpuaccel_demo.py demo/demo.mp4 configs/rtmdet/rtmdet_l_8xb32-300e_coco.py checkpoints/rtmdet_l_8xb32-300e_coco_20220719_112030-5a0be7c4.pth --nvdecode --out result.mp4

Did you make any modifications on the code or config? Did you understand what you have modified?

 I have not done any modifications to the video_gpu_accel_demo.py file

What dataset did you use?

I used the demo video within the demo folder

Environment
sys.platform: win32
Python: 3.8.19 (default, Mar 20 2024, 19:55:45) [MSC v.1916 64 bit (AMD64)]
CUDA available: True
MUSA available: False
numpy_random_seed:<PHONE_NUMBER>
GPU 0: NVIDIA RTX A500 Laptop GPU
CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4
NVCC: Cuda compilation tools, release 12.4, V12.4.99
MSVC: Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30154 for x64
GCC: n/a
PyTorch: 2.2.2
PyTorch compiling details: PyTorch built with:
  - C++ Version: 201703
  - MSVC 192930151
  - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.3.2 (Git Hash 2dc95a2ad0841e29db8b22fbccaf3e5da7992b01)
  - OpenMP 2019
  - LAPACK is enabled (usually provided by MKL)
  - CPU capability usage: AVX2
  - CUDA Runtime 12.1
  - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
  - CuDNN 8.8.1  (built against CUDA 12.0)
  - Magma 2.5.4
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.8.1, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /Zc:__cplusplus /bigobj /FS /utf-8 -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE /wd4624 /wd4068 /wd4067 /wd4267 /wd4661 /wd4717 /wd4244 /wd4804 /wd4273, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.2.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

TorchVision: 0.17.2
OpenCV: 4.7.0
MMEngine: 0.10.3
MMDetection: 3.3.0+0dede7a

Error traceback
Traceback (most recent call last):
  File "demo/video_gpuaccel_demo.py", line 144, in <module>
    main()
  File "demo/video_gpuaccel_demo.py", line 114, in main
    for i, (frame_resize, frame_origin) in enumerate(
  File "C:\Users\bbdnet2520\.conda\envs\openmmlab\lib\site-packages\mmengine\utils\progressbar.py", line 240, in track_iter_progress
    raise TypeError(
TypeError: "tasks" must be a tuple object or a sequence object, but got <class 'ffmpegcv.ffmpeg_reader.FFmpegReaderNV'>

I encountered the same problem as you

Was this solved?

You have to change line 115 to:
zip(track_iter_progress((video_resize, len(video_resize))), video_origin)):

track_iter_progress expects either a sequence of type Sequence or a tuple of an iterable with its length(Iterable, int) and video_resize is an Iterable but not a Sequence.


configd: capture scripts outputs

Hi!
For now if the configd action fails, the output of the command can only be seen in the shell.
pr tries to help with this with a migration from subrocess.call to subprocess.run so that we can grab the commands output.
This can help, for example, with services startup errors debugging (configuration errors, socket conflicts, etc.) when such errors do not get into the plugin log.

(as a bonus, get rid of pgrep false-errors like https://forum.opnsense.org/index.php?topic=31079.0).
catch 'parameter mismatch' for inline actions also (ref. https://forum.opnsense.org/index.php?topic=30908.msg149703#msg149703)
Thanks!
ps. it seems that it was even in the plans, but for some reason did not get around to it? ) (https://github.com/opnsense/core/issues/3574#issuecomment-539113436)

not relevant to the code in master branch anymore obviously )
will try to play with updated configd first

@kulikov-a I've only moved it around to make room, error logging could still be improved

Hi, Ad!
yes, I'm just afraid to do it without testing. Will make a new pr when I test it and if it still be relevant, if you don't mind )

Hi Alex,
If you want to take a stab at it, go ahead, I think we can simplify this even more by merging some functionality, but there's no rush, I do agree logging could be better here (shouldn't be too difficult).
Best regards,
Ad


Query on fajr start time with 18 deg angle and Magrib time

Is there an offset of -3 minutes to true twilight start with 18 deg methods. I am getting for my location a time, which is 3 minutes less than reported in the link here-
My Lat : Long - 52.0620° N, 1.2705° E
https://www.thetimeandplace.info/uk/martlesham-heath-suffolk#.Xy_WMYhKjIU
Second question I have is, what should be end time for Maghrib- is it until twilight totally disappears- and exactly when Isha starts? (I am aware of Ahadith where it's preferred to pray in the beginning of time- but just checking what's your view?)

If you're using the Moonsighting Committee Method, that method specifies a 3 minute buffer after the start of sunset for the sun to be fully set before it is considered time for Maghrib.
In terms of the FINAL end time for Maghrib, my understanding is that would be up until it is time for Isha.


New Relic Reporting : Monospice\LaravelRedisSentinel\Horizon\HorizonServiceBindings::getIterator() should either be compatible with IteratorAggregate::getIterator()

Hello there,
Since upgrading to PHP8.1 New Relic is alive with the below error.
I can't actually see these logs in Laravel.log or the Horizon.log - I also can't confirm that it's even real since New Relic has been unreliable since 8.1 but I'm wondering if anyone else is experiencing this issue at all ?

Any assistance / thoughts would be great.
Return type of Monospice\LaravelRedisSentinel\Horizon\HorizonServiceBindings::getIterator() should either be compatible with IteratorAggregate::getIterator(): Traversable, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice
                in Composer\Autoload\includeFile called at /var/www/hidden/vendor/composer/ClassLoader.php (428)
                                                  /in Composer\Autoload\ClassLoader::loadClass called at ? (?)
              in is_a called at /var/www/hidden/vendor/laravel/framework/src/Illuminate/Log/LogManager.php (371)
…:createMonologDriver called at /var/www/hidden/vendor/laravel/framework/src/Illuminate/Log/LogManager.php (210)
…\LogManager::resolve called at /var/www/hidden/vendor/laravel/framework/src/Illuminate/Log/LogManager.php (125)
…\Log\LogManager::get called at /var/www/hidden/vendor/laravel/framework/src/Illuminate/Log/LogManager.php (112)
…g\LogManager::driver called at /var/www/hidden/vendor/laravel/framework/src/Illuminate/Log/LogManager.php (101)
…lled at /var/www/hidden/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php (101)
…lled at /var/www/hidden/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php (72)

By the way, i have fixed this in my fork as the error suggests : adding #[\ReturnTypeWillChange] to HorizonServiceBindings::getIterator() like so :
/**
     * Get an iterator for the service bindings so we can iterate over an
     * instance of this class.
     *
     * @return ArrayIterator
     */
    #[\ReturnTypeWillChange]
    public function getIterator()
    {
        return new ArrayIterator($this->serviceBindings);
    }


More consistent api endpoints for adding users to teams and orgs

At the moment we do:
POST /teams/23/users/
{"user_id": 21}

It would be more consistent with the rest of the api to be able to do:
PUT /teams/23/users/21

@rudigiesler ready for review

Just the one comment from me, looks good otherwise

@rudigiesler Ready for re-review.

:+1:


Remove Unstable Rust Packaging?

I'm opening an issue here to ask whether this would be a good idea.
Those who do want unstable Rust packages have another way of doing so. I'll let @eddyb explain how if he wants to.
But the current dev version is pre-1.0.0, and I don't see anybody wanting to update it. To keep the package repository slim, I'd like to remove them.
Does anybody have a problem with this? Those who are core maintainers, can you say that you don't have a problem with this?

@Ericson2314 and @Tobba have a working solution that uses the upstream nightlies and is parametrized by a nightly date and a hash of the downloaded archive.

Yeah IMO nixpkgs need only have stable, or maybe beta. No need for nightly/HEAD source builds unless we get a critical mass rustc developers using nixos (I'm sure @eddy can fend for himself :D).
I'm not sure our script belongs in it as downloading binaries is gross, but to compensate I made https://nixos.org/wiki/Rust so people have a way to discover it. I suppose whatever you do, good to document it too on that that page?

:+1: to left only stable version.

I am opposed to removing the unstable Rust packaging.
I use unstable Rust frequently because it's the only way to use certain useful features such as compiler plugins (e.g. all of the lints in rust-clippy require unstable Rust).
I don't find it to be a big burden to keep the unstable Rust compiler updated, since much of the Nix packaging is already shared between the stable and the unstable Rust compilers.
In fact, I already have a patch ready to update unstable Rust to a more recently nightly, but I'm waiting for #8668 to be merged first. I also have a python script that can update unstable Rust to the latest nightly (in fact, to exactly the same commit that the latest nightly has been built from).
I much prefer using this script and compiling from source than downloading and running random binaries from the Internet. Each binary that you download and run increases the chance that your machine is trojaned due to the downloading server having been compromised. It is much harder to trojan a git repository without it being quickly noticed by someone.

@wizeman if you want to maintain Nightly, I see no problem with that.

@Ericson2314 No problem, I'll be happy to do it!

Alright, you have a script that builds from source instead of downloading a binary. This still has to be updated every day to maintain freshness. Given that there's at least a week before the unstable branch updates (and lately it's been more like a month), and the fact that stable releases of nixpkgs are once every couple months, is there a place for daily changing packages in nixpkgs? Is there a reason it has to be in nixpkgs and not as part of some other nix repo?
But this issue is closed by nature of @wizeman being willing to maintain the nightly packages. My questions are more questions of policy.

Who updates GHC head?


[Catalogs] [Delta Lake] Add support for reading tables with deletion vectors

Delta Lake can improve the efficiency of row deletions with deletion vectors, which is an "has this row been deleted" bitmap that elides rewriting Parquet files whenever a row is deleted. Our Delta Lake reader should support reading Delta Lake tables that contain deletion vectors, which would involve carrying along the bitmap into the Parquet scan as a mask and pruning the relevant rows from the scan before they are decoded and materialized.
Resources

https://docs.delta.io/latest/delta-deletion-vectors.html
https://github.com/delta-io/delta/blob/master/PROTOCOL.md#deletion-vectors
https://docs.google.com/document/d/1lv35ZPfioopBbzQ7zT82LOev7qV7x4YNLkMr2-L5E_M/edit#heading=h.cb81q2y0eh35
https://docs.google.com/document/d/1lv35ZPfioopBbzQ7zT82LOev7qV7x4YNLkMr2-L5E_M/edit#heading=h.zh5i3rlz2brp
https://docs.google.com/document/d/1lv35ZPfioopBbzQ7zT82LOev7qV7x4YNLkMr2-L5E_M/edit#heading=h.yvlcj9hgj4tr

This functionality is not currently supported in delta-rs, but it is in delta-kernel-rs. Our plan is currently to develop a new, all-rust code path for read_deltalake that uses delta-kernel-rs instead, which will be disabled by default initially, but can be enabled with a parameter and will support deletion vector reading

Do you guys have an update on the delta-kernel-rs implementation? :) I think the kernel is the most sustainable approach. We are currently, evaluating daft. It would be an important requirement because most of our data is saved in delta tables with deletion vectors activated.

@kevinzwang Do you guys have an update on the delta-kernel-rs implementation? :) I think the kernel is the most sustainable approach. We are currently, evaluating daft. It would be an important requirement because most of our data is saved in delta tables with deletion vectors activated.

We need delta-kernel-rs to implement writes! Otherwise we can't migrate :(

@jaychia Thanks for the quick response. Writes are for me not important for now. But rather reading Delta tables with existing deletion vectors. Based on above it seems already available in Delta Kernel :)

@kevinzwang any thoughts on a partial migration to delta-kernel-rs for just the reads?

From what I've seen, delta-rs has not made much progress on deletion vector support so far. I'll explore using delta-kernel-rs for reads

I think we are waiting for delta-kernel-rs to reach a more mature state before we make the engineering effort to switch to them. As a result, delta reads with deletion vectors won't be in our near term plans, but I'll check in on this issue again in the future.
In the meantime @datanikkthegreek (or others) if you would be interested in contributing this functionality, I'd be happy to help!

@kevinzwang Thanks :)
I am not really familiar in rust. So unfort. I can't support. I am happy to support testing the feature once in pre-release though and give feedback if needed.

No worries @datanikkthegreek. We'll let you know once we support this!


No indication of where other party has read to

Official clients indicates upto where the other parties in the conversation has read to. YakYak doesn't seem to have any such indicator as far as I've seen.

:+1:

That'd be nice to have.

:+1: This is the main thing missing.

#55

Poked around a bit.  Looks like the information is definitely available.  So would just be a matter of displaying it.  Which admittedly looks like the hardest part :grin:

where did you find it available?

Partial implementation of this in 1.4.0-beta3
see #451 and activate option via view -> experimental
The library that yakyak uses does not support (yet) having this information when opening a conversation, only after first interaction

can anyone test a build with this fixed?
@silveran
reply with operating system

It works like charm for me on MacOS Sierra

the implementation is not complete, you only get the seen after the other user interacts or opens the conversation.
doesn't it?

Just tried it, I confirm what you said.
After launch the app, no read notification before the other open the discussion on his app.

@silveran had problems trying to only show the last conversation seen (instead of all others)
here it is: https://github.com/averissimo/yakyak/releases/tag/sandbox

I got an issue with the zip, unable to unzip, it gives me an error 20 - Not a directory

sorry, tried to be a smart ass and install the new library manually -_-
uploading now an extractable version

Seems to work well now

So cool! thanks for testing.

this will be available in 1.4.1 (you can test it in the pre-release, or just wait a little bit)

Running 1.4.4, this seems to be missing. Has something changed in the API, or has the feature been removed again? Cannot find a way to activate it.


Default-bundled css + js should use only from reveal.js core.

Currently
Anytime include files are:

revealjs/css/reveal.css (from builder class)
revealjs/lib/css/zenburn.css (from builder class)
revealjs/js/reveal.js (extension config)

Expected
revealjs/lib/css/zenburn.css is theme of highlight.js.
It is not minimum component of reveal.js, it should be removed.
Refs

Improved from https://github.com/attakei/sphinx-revealjs/issues/27#issuecomment-631970588

Fixing on https://github.com/attakei/sphinx-revealjs/commits/feature/extra-css

Fix by 0.12.0


Provide access to Row::getColumnName from Row Description.

This is an enhancement for #135

Hi @pratikpparikh,
I think we need test for Row in RowTest

@EmadAlblueshi we should be covered not with test cases and appropriate code check-in for this request.

I'll merge it

thanks for the contribution!


xinfo responds with an empty map

XINFO consumers should respond with an array, whether it's RESP2 or RESP3 (https://redis.io/commands/xinfo-consumers/), but when I call it I get a map back. If there is only 1 consumer, the map is empty ({}). If there are 2 consumers, then the key is the first consumer & the value is the 2nd consumer.
XINFO groups also behaves like this.
When I issue the command directly to redis-cli it works as expected. Is there perhaps a setting that I'm missing?
commander v0.8.1
redis v7

Sorry, cannot reproduce.
But the data are returned directly as received from ioredis connection. So its probably a problem there handling the data parsing....
https://github.com/redis/ioredis/
ioredis driver is already at the latest version here.


Why feature sources do not contain loadstart and loadend events?

Tile and Image soruces contain "tileloadstart" and "tileloadend" events. But, feature sources do not containd any event to get "loadstart" and "loadend", so we need to customize the feature sources using ajax calls. Do you have a plan to add load events in feature sources?

Hi,
I had/have (?) the almost same question. For me it is more about: getting an event after ALL Features are loaded.
I dived into it a little bit. The tileloadstart|end are triggered for EVERY tile/image before/after loading.
This event is triggered as a kind of an extend on the change-event. So I guess you can use the change-event to get almost the same on the other sources.
Does this help?
Greetz, flexjoly

Change event fires after a feature add/remove on a source, am I right? But I need get an event before/after/error  ajax request to server.

We would accept a pull request here.

@flexjoly, Change event fires after a feature add/remove on a source, am I right? But I need get an event before/after/error ajax request to server.

I would like this event also (but it is nice-to-have for us). If I have time this or next week I will try to make something.
@bayramucuncu Have you noticed that those are quite different Events then the tileloadstart and tileloadend ?
As far as I can see, this can be done in ol/loadFeaturesXhr

@flexjoly, Change event fires after a feature add/remove on a source, am I right? But I need get an event before/after/error ajax request to server.

I would like this event also (but it is nice-to-have for us). If I have time this or next week I will try to make something.
@bayramucuncu Have you noticed that those are quite different Events then the tileloadstart and tileloadend ?
As far as I can see, this can be done in ol/loadFeaturesXhr

Yes this can be done in ol/loadFeaturesXhr, and may be extra const like FEATURELAOADSRAT, FEATURELOADEND, FEATURELOADERROR.
I will try it as well.

A possibility would be to expand the loader with 2 optional arguments: success and error. The firing of the events could be done inside the source itself
https://github.com/openlayers/openlayers/blob/7eb60fae239db9fe44f54900bc40bba87c0ab230/src/ol/source/Vector.js#L888
if the callbacks are not provided only loadstart would be fired.

A possibility would be to expand the loader with 2 arguments: success and error. The firing of the events could be done inside the source itself
https://github.com/openlayers/openlayers/blob/7eb60fae239db9fe44f54900bc40bba87c0ab230/src/ol/source/Vector.js#L888
if the callback arguments are not executed only loadstart would be fired.

I think we do not need to expand loadFeatures(extent, resolution, projection) . Because it does not include the xhr process and events. I think the best place is loadFeaturesXhr. And we sholut fire a new event type in here.

I would not want to extend loadFeatures but fire the events in there. I would extend the FeatureLoader type to accept callbacks.
https://github.com/openlayers/openlayers/blob/14b931371dfc0c2538db475d551eb1f2c6473749/src/ol/featureloader.js#L26
This for me has 2 advantages:

The same callbacks can be called in custom loaders and the VectorSource would automatically fire the correct events.
The events are dispatched inside the class itself. This makes it easier to understand what happens in the class.

Alternatively you could of course specify that the loaders should fire the events on the source, but this seems a unclean to me.

the xhr function in featureloader.js implents the FeatureLoader type and calls loadFeaturesXhr.

I would not want to extend loadFeatures but fire the events in there. I would extend the FeatureLoader type to accept callbacks.
https://github.com/openlayers/openlayers/blob/14b931371dfc0c2538db475d551eb1f2c6473749/src/ol/featureloader.js#L26
This for me has 2 advantages:

The same callbacks can be called in custom loaders and the VectorSource would automatically fire the correct events.
The events are dispatched inside the class itself. This makes it easier to understand what happens in the class.

Alternatively you could of course specify that the loaders should fire the events on the source, but this seems unclean to me.

Okay, it make sense. I have planned to create a new event type named VectorSourceLoadEvent and fire it in loadFeatures  function like following

this.dispatchEvent(new VectorSourceLoadEvent(VectorEventType.FEATURELOADSTART));
this.dispatchEvent(new VectorSourceLoadEvent(VectorEventType.FEATURELOADEND));
this.dispatchEvent(new VectorSourceLoadEvent(VectorEventType.FEATURELOADERROR));

You can do it similar to this way.

Keep in mind that users can provide their own loaders by configuring a tileLoadFunction. If a listener was fired in the default ol/featureloader/loadFeatures function, it would only work if no tileLoadFunction was configured, or if it uses loadFeaturesXhr, which is the default loadFeatuers function, or if they configure their custom loader with the correct callbacks.
So a good place to implement this would be ol/featureloader/xhr, which would take success and error callbacks as additional arguments. The events would then be fired by those callbacks, whild would be given to the xhr function in ol/source/Vector#loadFeatures.

As I have some time on my hand, I will implement this.

@ahocevar I have some thoughts and questions about this. Do you have a short moment to answer these?
To support the success and error calls the FeatureLoader type would be extend to:
 * @typedef {function(this:(import("./source/Vector").default|import("./VectorTile.js").default),
 *           import("./extent.js").Extent,
 *           number,
 *           import("./proj/Projection.js").default,
 *           function(): void=,
 *           function(): void=): void} FeatureLoader

I was wondering if the this binding is really necessary as the target should be known when assigning the featureloader. It would make the type a little easier. Alternatively the target could be passed in. Both would be breaking changes and are not really necessary.
With individual success and failure callbacks for every request the loadFeaturesXhr function in the current from does not make this much sense anymore. I would refactor it a little.
The xhr function also does not make very much sense as it was intended to support both vector sources and vector tiles but is now only used by vector sources. I would move it into source/Vector.js as createDefaultLoadFunction.
To match things up I would also refactor the defaultLoadFunction in source/VectorTile.js to createDefaultLoadFunction.
Alternatively both default function creators could live in featureloader.js.

The LoadFunction type of ol/Tile has the tile passed in. To unify these loader function a change to pass the vector source / vector tile into the FeatureLoader would make sense. As this is a breaking change I would leave it for now and if desired I could create a second PR with breaking changes.


ValueError: Exception encountered when calling layer "attention" (type Attention)

Hi, I tried to copy all modules onto a google colab notebook and also added the KG Data for Small MIND dataset. However, when I run the notebook the following error pops up:
ValueError                                Traceback (most recent call last)
<ipython-input-26-0a1ef422da03> in <module>()
----> 1 model,news_encoder,user_encoder,rews = create_model(category_dict,subcategory_dict,title_word_embedding_matrix,entity_emb_matrix)
      2 Res.append({'AUC':[],'MRR':[],'nDCG5':[],'nDCG10':[]})
      3 train_generator = get_hir_train_generator(0.9999,news_info,index2nid,news_vert,news_subvert,news_entity_np,news_entity,train_user['click'],train_user_id,train_sess,train_label,16)
      4 model.fit_generator(train_generator,epochs=4,verbose=3)
      5

3 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    697       except Exception as e:  # pylint:disable=broad-except
    698         if hasattr(e, 'ag_error_metadata'):
--> 699           raise e.ag_error_metadata.to_exception(e)
    700         else:
    701           raise

ValueError: Exception encountered when calling layer "attention" (type Attention).

in user code:

    File "<ipython-input-10-c62a5f68bc7e>", line 55, in call  *
        A = K.permute_dimensions(A, (0,3,2,1))
    File "/usr/local/lib/python3.7/dist-packages/keras/backend.py", line 3288, in permute_dimensions
        return tf.compat.v1.transpose(x, perm=pattern)

    ValueError: Dimension must be 5 but is 4 for '{{node attention/transpose_7}} = Transpose[T=DT_FLOAT, Tperm=DT_INT32](attention/truediv, attention/transpose_7/perm)' with input shapes: [?,20,30,20,30], [4].

Call arguments received:
  • x=['tf.Tensor(shape=(None, 30, 300), dtype=float32)', 'tf.Tensor(shape=(None, 30, 300), dtype=float32)', 'tf.Tensor(shape=(None, 30, 300), dtype=float32)']

Steps to reproduce the error: connect to the following notebook  https://colab.research.google.com/drive/1h88uAM3MIi0htHJoJiY9cRMJ3jo0lrLu?usp=sharing and click Run All.
I did modify the tensorflow import (use tf.compat.v1. instead of tf. for some imports) since Colab uses Tensorflow v2 by default. I realized that either tensorflow v1 and v2 both lead to the abovementioned error. If possible, could you check if the original code still works ? I tried to debug the program but couldn't find the cause.

This is caused by the inconsistent version of tf and keras. The code can be executed under  (1) tensorflow-gpu == 1.13.0 (2) keras == 2.2.4


TabNine version keeps flipping (Pro -> Starter -> Pro)

please complete the following information:

OS version: Win 11
Editor: intelij
Editor version:	Ultimate 2023.1.3
Programming language: Java and Angular
TabNine extension version: 1.0.16
Engine version (could be located within Tabnine Hub): 4.5.9-41

Issue Details:
I've purchased the Pro version and the TabNine version keep flipping between Starter and Pro version. Does not matter the language I'm using.
https://github.com/codota/TabNine/assets/20236895/a4b52755-f675-456d-bae2-6fb95ebc3529
gz#24407(related to Zendesk ticket #24407)

Hi @absantanna2412
Are you behind Proxy/VPN/Firewall/Antivirus? Please follow the steps here.

Also, please send us the log files. See here how to do that.


Enable provider autodiscovery on Laravel 5.5+

This change supports package autodiscovery in Laravel 5.5+ making installation just that little bit easier.

+1. This is important.

This can be merged without any risk


Simplify case before trying to detect sign/width

This PR changes order of detecting sign/width and simplify in case.
This fixes the case where case uses items that requires simplify before trying to detect sign/width, e.g. $bits call.

This has been superseded by #3116.


Error messages are now printed using logger.error instead of System.error

Using logger for printing messages make it consistent. And it also makes sure all logs go through same logging configuration (logback.xml) which has the s3 prefix in it. Using println will bypass it.

@andrewgaul sorry I couldn't fully understand it. Can you explain a bit more about it?

@andrewgaul Is this change ok? You can configure it through properties

@andrewgaul any update on this pull request

@kishore25kumar Please respond to my original review comment.

@andrewgaul I didn't find anything similar to something you have specified. The best I could found is set the System.err to a different function and write it using a logger in the function. What to set for System.err is decided by a config flag. That is the change I have made in the pull request.

@andrewgaul any suggestions on the pull request? I tried answering your question in previous pull request

@andrewgaul any suggestions on this pull request?

You did not respond to my original review comments.  I suggested looking at an isatty alternative like System.console.  Let me Google that for you:
https://stackoverflow.com/questions/1403772/how-can-i-check-if-a-java-programs-input-output-streams-are-connected-to-a-term
This would allow the common case of an interactive user getting help text formatted as all other command line programs do while logging when System.err is not connected to a terminal.  This is similar to how ls decides whether to color its output.  For what it's worth, this is how mysqld and other
Your approach, adding a flag to satisfy a single niche user, is not scalable from a complexity point of view.  Have you instead considered redirecting output from the s3proxy executable, e.g., sed 's/^/[s3proxy] /?
Why do you have such urgency for this logging-only patch?  Just put it in your private branch if you are so committed to this.

@andrewgaul I have updated pull request based on system.console exists or not. Sorry for the confusion as I don't have an idea about attay. Thaks for clarifying it helped.

I guess this is an improvement, but the first thing I tried does not log output correctly, with part of usage going to logger and the rest going to stderr:
$ target/s3proxy 2>&1 | cat
[s3proxy] E 04-18 19:25:46.696 main org.gaul.s3proxy.Main:251 |::] Usage: s3proxy [options...]

 --properties FILE : S3Proxy configuration (required)
 --version         : display version (default: false)

What are you even trying to accomplish with this pull request?  Why do you want usage to go to logger?

@andrewgaul

In a production environment, the service is running with multiple log appenders. Logging all the messages using logger will write it to attached log appenders. Example Splunk log appender.
All the messages that are logged from s3proxy will have the same format as they will follow logback.xml format.
In a production environment, deployed instances are not accessible for debugging issues. Log appenders are the solution here.

I figured out the issue. command line parser is not using the printstream directly instead it is using outputstream and wrapping this using a writer. So finally the ouputstream write function is called instead of printstream print function. I am looking into it.

@andrewgaul Fixed the issue


Request attributes exposed by circuit breakers are preserved permanently when called asynchronously

As shown in the code below, the request attributes are passed to the thread calling the execute method through the RequestContextHolder (ThreadLocal)
https://github.com/spring-cloud/spring-cloud-openfeign/blob/8493de9aa6038024acab81142c5cb714d1b9edfc/spring-cloud-openfeign-core/src/main/java/org/springframework/cloud/openfeign/FeignCircuitBreakerInvocationHandler.java#L124-L136
This code is from: https://github.com/spring-cloud/spring-cloud-openfeign/issues/193
And updated: https://github.com/spring-cloud/spring-cloud-openfeign/issues/572
This creates a conflict. The idea here should be to use ThreadLocal to pass request attributes in async calls and do nothing in sync calls. And now ThreadLocal is not cleaned up after using it in async call.
The easiest way is to compare whether the calling thread and the executing thread are consistent before the operation:
final RequestAttributes requestAttributes = RequestContextHolder.getRequestAttributes();
final Thread caller = Thread.currentThread();
return () -> {
                boolean isAsync = caller != Thread.currentThread();
                try {
                    if (isAsync) {
                        RequestContextHolder.setRequestAttributes(requestAttributes);
                    }
                    return dispatch.get(method).invoke(args);
                }
                catch (RuntimeException throwable) {
                    throw throwable;
                }
                catch (Throwable throwable) {
                    throw new RuntimeException(throwable);
                } finally {
                    if (isAsync) {
                        RequestContextHolder.resetRequestAttributes();
                    }
                }
 };

Hello, @vicasong. Thanks for reporting this. Makes sense. Would you like to create a PR with your proposed fix?

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

PR created


make intialize only available in OapFileFormat

What changes were proposed in this pull request?
Fixes #647
initalize() can be placed only in OapFileFormat, so that we don't need FileFormat.scala and other dependent source files in our repo to override upstream.
How was this patch tested?
existed ut.

For reviewers: if you want to run unit test locally, you need to explicitly run mvn clean to get rid of classes corresponding to deleted source files.

We can change this now because inferSchema can be called with parameters?

I think this is a good idea. Later if we want to support more formats, i.e. add index to them, we can add OapFileFormat wrapper just like so far we have done for parquet, rather than copy/hack so many fileformats from Spark upstream.
Just to confirm. Does parquet's OapFormat wrapper utilize this inferscheme?

@yueawang when creating index on parquet, we are using ReadOnlyParquetFileFormat, just for reading data. When querying parquet, we are just using OapFileFormat.

Good Job!!!!! @adrian-wang


[Feature Request] Support Unicode / Emoji search

Any plan to support search of unicode languages like Bangla (বাংলা) , characters like ö or emoji 🌲 ?

I guess it's hard to have. there are two componets of the searching function.

When use zola to build from converting markdown files for your ob notes, zola will passthough the contents and make a indexing .js file for searching.
use elasticlunr to comparing what you typing in, comparing and return the results.

the elasticlunr did'nt update in past 3 years. but there are some multi language comparing plugin repo right now. if you are farmilar with js, it will take no long time to modified the search.js to support the multilingual comparing.
But again you should also know something about rust, since the indexinig file generate from zola will just provided for specific language. Without the indexing file, your comparing engine will just find nothing to compare


Provide mechanism for read lock on the cluster

As an administrator in need of performing atomic operations on the state of a cluster it would be useful to perform an atomic lock on the cluster.  This would then allow the use of external utilities like LVM/BTRFS snapshotting to assist with the backup recovery process which would be nice for extremely large data sets.
This is a common operation with relational databases.  From a historic perspective, this was the only way to back up some pieces of software without proprietary tools like InnoDB.

To some extent etcd has similar functionality with quota alarms. The lock feature could probably be merged into the alarm system (e.g., a user-triggered "maintenance" alarm that locks writes). I'm assuming the expectation is writes would block server-side until the lock is lifted? Or is it OK to fail them?

I'm assuming the expectation is writes would block server-side until the lock is lifted? Or is it OK to fail them?

I would follow what mysql does. I can do some research on it.

/cc @brianredbeard might provide more insights.

I could go a few different ways here.  I think that a timeout is actually
fine for this.  I don't know the level of code complexity, but being able
to say "while (client has not released lock) or (client connection timeout)
hold lock" would be fine. Let's walk through two (over engineered) use
cases:

"etcd large backup operator" -
In situations where gigabytes of data are stored in etcd (despite pleas not
to do this) an operator could write a snap request to
/_etcd/snapshot/request. Provocateurs/operator agents could then respond
with a nonce to /_etcd/snapshot/member_name which they will send via TCP to
the operator on snapshot completion.

Once all provocateurs have written their nonce, the operator locks. Members
take their snapshot and respond. When all participants complete, the
operator releases the lock.

the operator locks for snapshot and holds the lock until it has
confirmation that the operation completes.  If the client connection breaks
the server kicks off a hold off period after which it automatically
releases the lock, in effect creating a dead man's switch.

These are just two off the cuff ideas. Curious as to your thoughts.
On Mon, Jun 19, 2017, 18:21 Xiang Li<EMAIL_ADDRESS>wrote:

/cc @brianredbeard https://github.com/brianredbeard might provide more
insights.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/coreos/etcd/issues/8134#issuecomment-309617192, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AADgxulVIVxIWq1bqOQPP1ONZcuZqwz4ks5sFx6YgaJpZM4N-nHc
.


Add test phases graph and a TODO.

This change is

/lgtm

Holding.. Because there seems to be no options in the 'phases' dropdown next to the one where we're selecting the job_summary.

/lgtm


[unixODBC][Driver Manager]Can't open lib '/opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.8.so.1.1' : file not found

Please forgive me for this long post, I have been searching in vain for an answer to my problem so I have tried to provide as much information as possible in this post, in the hope that someone will be able to help solve my problem.
I am not new to Linux, nor am I very proficiant so please give me some details on what you would like me to do if you need more information.
SQL Server Version
Microsoft SQL Server 2019 (RTM-GDR) (KB4583458) - 15.0.2080.9 (X64)   Nov  6 2020 16:50:01   Copyright (C) 2019 Microsoft Corporation  Enterprise Edition (64-bit) on Windows 10 Pro 10.0 <X64> (Build 19042: )

Client Version
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:        10
Codename:       buster

PHP Version
PHP 8.0.10 (cli) (built: Aug 26 2021 16:09:29) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.10, Copyright (c) Zend Technologies
    with Zend OPcache v8.0.10, Copyright (c), by Zend Technologies

MS ODBC Driver Version
Microsoft ODBC 17

Problem Description:
I have followed these instructions - https://docs.microsoft.com/en-us/sql/connect/php/installation-tutorial-linux-mac?view=sql-server-ver15 - Linux and macOS Installation Tutorial for the Microsoft Drivers for PHP for SQL Server - specifically the installing on Debian instructions. I have also followed these instructions, as detailed in the Step 2. Install prerequisites (Debian) section - https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15 - Install the Microsoft ODBC driver for SQL Server (Linux) - In the Microsoft ODBC 17 section, in the Debian subsection.
All of the installation steps run smothely. I had to edit my unixodbc_conf.h file to change the SIZEOF_LONG_INT from 4 to 8, before I could get the pecl install of sqlsrv to work. but other than that, no issues with any of the install steps.
I then followed these instructions - https://docs.microsoft.com/en-us/sql/connect/php/installation-tutorial-linux-mac?view=sql-server-ver15#testing-your-installation - Testing your installation and this is where I get the error - [unixODBC][Driver Manager]Can't open lib '/opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.8.so.1.1' : file not found
The file /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.8.so.1.1 does indeed exist in the file system as can be seen below
pi@raspberry:~ $ locate libmsodbcsql-17.8.so.1.1
/opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.8.so.1.1

When I check the dependencies on this file using ldd, I get the following information
pi@raspberry:~ $ ldd /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.8.so.1.1
        linux-vdso.so.1 (0x00007fff6c97d000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f51ffe6c000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f51ffe62000)
        libodbcinst.so.2 => not found
        libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f51ffd82000)
        libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f51ffd35000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f51ffbb1000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f51ffa2c000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f51ffa12000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f51ff9f1000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f51ff830000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f520029d000)
        libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f51ff7fc000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f51ff7f6000)
        libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f51ff7e5000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f51ff7de000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f51ff7c4000)

I am assuming that the error is to do with libodbcinst.so.2 => not found, but after searching here and all over the internet for the past two hours, I am unable to find the answer to how to resolve this.
The file that is missing is located in the following directory
pi@raspberry:~ $ locate libodbcinst.so.2
/usr/lib/i386-linux-gnu/libodbcinst.so.2
/usr/lib/i386-linux-gnu/libodbcinst.so.2.0.0

And I have tried creating a symlnk in the /usr/lib/x86_64-linux-gnu/ folder, but to no avail.
I also have a missing dependency on the sqlcmd file
pi@raspberry:~ $ ldd /opt/mssql-tools/bin/sqlcmd
        linux-vdso.so.1 (0x00007ffd3abaa000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffb89fde000)
        libodbc.so.2 => not found
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffb89e5a000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffb89e40000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffb89e1f000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffb89c5e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ffb8a003000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffb89ad9000)

Again, I have spent the past two hours looking for information on how to fix this fault also, but I cannot find anything to solve this.
Any help would be much appriciated, if any further information is required please let me know and I will get it.

Thanks @yitam. This is a virtual machine, which I originally used this iso to install.
Whilst trying to get this working today, I managed to 'upgrade' the OS somehow, it is now running Debian 10 I believe, instead of Raspbian, although I may have just broken it completly.
I was using this VM as a proof of concept before I looked to move over to an actual Raspberry Pi for further testing, but since seeing that it currently doesnt support a 64 bit OS in Raspbian, I may look at using Ubuntu instead.
It would still be interesting to see if this is able to get working.
As some further information. I did manage to get sqlcmd working on a previous test VM, but I wouldn't be able to accuratly describe the steps I took, I then ran into troubles with the pecl install sqlsrv command which I was not able to complete so started the VM from scratch. I only discovered the PHP instructions, after the msodbcsql17 install instructions.


CocoaPods could not find compatible version for pod 'TensorFlowLiteSwift'- with iOS 8 deployment target

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary):
TensorFlow version:
Python version:
Installed using virtualenv? pip? conda?:
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:

Describe the problem
Provide the exact sequence of commands / steps that you executed before running into the problem
Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

When I install 'TensorFlowLiteSwift' in my swift project with deployment target 8 it give me this error. is there any way we can achieve this?

Can you please elaborate about the issue & the context.Please, provide the exact sequence of commands / steps that you executed before running into the problem.Thanks!

Hi @ravikyram ,  please find my Podfile in the attachment. When I run 'pod install' from my terminal then it gives me this error. Thanks for the response.

Any update on this?

Could you take a look at #32780 and see if that could help?
To give you more background, TensorFlowLiteSwift pod depends on the TensorFlowLiteC binary pod, which also has iOS 9 as the minimum target.
You might also need to edit the Swift API code to remove the GPU delegate related code, and rebuild the TensorFlowSwift pod yourself, according to this guide.

It has been 15 days with no activity and the awaiting response label was assigned. Is this still an issue?

@parascse
Any update on this issue please. Thanks!

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!


JVM crash when training in parallel (EXCEPTION_ACCESS_VIOLATION + SIGSEGV)

Crash happens only when training with MKL, even when utilizing only 1gb out of 60gb memory (small net). So, it does seem to be related to running out of memory, at least.
It also only seems to happen with nets with many inputs. I only get this with my network that has 10k inputs, it never happens with the smaller nets (1000 inputs). Nets tested are RNNs and FFNs.
FWIW by parallel i simply mean doing something like this (i.e. parallel collections, no Spark or anything):
Seq(foo, bar).par.foreach(trainNetwork)

https://gist.github.com/Habitats/bd4b671ef2bf0a2c76ecb01535be0a1f

Seems I get a crash for the same job using OpenBLAS on unix, however, sourcing from libnd4j this time:
https://gist.github.com/Habitats/d58dab6a01e2981c44fef9f46394ecb0

For the latter issue: this may be due to the issue described here: https://github.com/deeplearning4j/nd4j/issues/954
OpenBLAS simply doesn't seem to be properly thread safe on GEMM, whereas MKL seems fine (at least on my machine).

Taking @sadatanwer's advice and trying to limit OMP_NUM_THREAD=1; if I understand this correctly this will limit one OpenMP thread per worker? I.e. if I train 2 nets in parallel I'll have 2 sandboxed OpenMP threads?

Update: When training RNNs in parallel I get a third error: Segmentation fault in the terminal, and no logs anywhere. It seems to be the same issue as https://github.com/deeplearning4j/libnd4j/issues/205.

Since it's behaving the same on Linux, could you try to run that in Valgrind?
https://github.com/deeplearning4j/libnd4j/wiki/Debugging-libnd4j
It will be really really REALLY slow, but if it reports any memory corruption issues, it will be easy to fix.

I have very little time for field work atm (thesis deadline in 2 weeks~), so I don't want to break my builds atm (more than what's already broken):\ I can probably try with Valgrind later this week.
I ran another RNN and this time it trained and finished (12~hours) without problems ... Weird.

This looks like it can be really related to https://github.com/bytedeco/javacpp/issues/103 as the crashes happen when using multiple java threads and a parallel GC. @Habitats is using -XX:+UseConcMarkSweepGC.
I asked him to try the default GC and report back.

I've been training with the default GC on 10 threads for almost 4 hours now; memory is stable and without error.
Looks like this is indeed related to CMS GC.

I think I found what the issue is and a way to reproduce it consistently:
https://gist.github.com/saudet/4de601b9ef25d78b3474c997c368a870
Typical output:
Computing call overhead...
processLongPtr(): 12 ns
processPointer(): 14 ns
Testing crashability... (Make sure to execute with -XX:+UseConcMarkSweepGC)
processPointer()...
processLongPtr()...
[thread 9912 also had an error]
[thread 1184 also had an error]
[thread 10064 also had an error]
[thread 9068 also had an error]
#
# A fatal error has been detected by the Java Runtime Environment:
#
...

What happens basically is that, when we pass only the address of a temporary Pointer object to the JNI function, the object itself is no longer reachable, so there is a possibility for the GC, especially the concurrent one, to deallocate it.
In the case ND4J, based on the stacktrace from @Habitats, this happens when calling the INDArray .transpose() and mmul() methods here for example:
https://github.com/deeplearning4j/deeplearning4j/blob/master/deeplearning4j-core/src/main/java/org/deeplearning4j/nn/layers/BaseOutputLayer.java#L163
A Pointer object is create, the data used inside some native function, and discarded immediately after.
So, I think the best thing to do would be to replace all those long parameters to FloatPointer, DoublePointer, etc. As shown above, the overhead is pretty negligible and it also saves us writing boilerplate code for the cast operations.


add window option to easyffts

Hey, thanks for making a PR! Sorry for taking so long to get it checked out.
While it is often a good idea to do windowing, it seems like a very opinionated step that can be added before computing the FFT. The only benefit I see is that there is some extra details about scaling the result, and doing that for the user is very much in line with the spirit of this package. Is this your thought? That it is cumbersome to get the rescaling right when windowing?
Also, applying windowing by default seems like a very bad idea to me, as I see no reason to expect it as a user. It would make this package go beyong adding convenience, and actually alter the results. Finally, taking on DSP as a dependency is rather significant, which makes this package much much less lightweight. This is my biggest issue with this PR.
All in all, I feel like the cons outweight the pros, and that most of the benefit (making it easier to scale correctly when windowing) could be provided by writing up a copy-pastable example in the docs (readme) showing how to do it. Does providing a documentation-example feel like a decent compomise to you?

Hey, thank you for your extensive response! To be honest I didn't lay so much thought on the changes as you did. I needed windowing and quickly made the changes on top of you package. I thought, it could be useful to add the functionality to the package. I realise now that indeed the windowing could just be applied outside easyfft and response rescaled later.
Now, I very much agree with assessments:

Having windowing in the package can offload the scaling of the response afterwards
Windowing should not be enabled by default

Regarding DSP dependency, we could extend the easyfft in the with an extension when DSP is loaded. However, this would mean you will have to change the julia compat to at least 1.9. If you don't like the extention idea, I am also okay with the documentation-example.

Seeing that the LTS is actually 1.10, I would actually be fine in bumping the version to 1.10. We could then have two signatures: One as it is now, and one that takes a function as the second argument. If a function is provided, it is assumed to be a window function, which returns a window when given the length of the signal (or, in the multidimentional case, the length along each dimension as a tuple). Because this would dispatch on the type of hanning, which is function, it would actually not even require a package extension!

I put something together in #15. Does that look good to you? I am pretty sure that you can uninstall the package, and install it again targeting the https://github.com/KronosTheLate/EasyFFTs.jl/tree/add_windowing_convenience branch to test it locally if you want.


Spotify Artist Playlist Enhancements

Driven by the same issues I encountered with the Liked Songs Library which led me to create this PR: https://github.com/BimmerGestalt/AAIdrive/pull/263 , when the user views an artist playlist there was the main issue that the user could not skip to a specific song in the playlist and it would instead just restart the first song. Utilizing the framework that I created for the Liked Songs Playlist I was able to expand it to the Artist playlists as well so now the user can select any of the tracks they wish to play from that artist.
Additionally another problem I had noticed with artist playlists was that the items in the artist playlist often times were not the correct items that one would expect from the official Spotify experience (sometimes the songs were random or it would pull random playlist items into the playlist). When the user browses to an Artist they are presented with 5 (or 10 on the desktop client) of the artist's top songs. I have utilized the Spotify Web API to get these results which should bring this playlist more in line with what is expected from the official Spotify experience.
Another notable change is when the user searches for a query then selects an artist from the search results, the browse view will redirect to this artist playlist with the artist's top 10 songs. Again this is more similar to the official Spotify experience.
Other smaller noteworthy changes include the redesign of the system to reattempt creating the Liked Songs and Artist Top Songs queues if they fail due to the user not having the Spotify Web API authenticated and then they do authenticate it should auto retry if the queue context is the same. Additionally, the phone interface queue and browse list items have been adjusted to make them easier to click by centering their contents with equal length margins.

Codecov Report

Merging #400 (3bcbefc) into main (f4ff5dc) will decrease coverage by 33.19%.
The diff coverage is 0.95%.

@@              Coverage Diff              @@
##               main     #400       +/-   ##
=============================================
- Coverage     48.65%   15.46%   -33.20%
+ Complexity     1513      297     -1216
=============================================
  Files           207      205        -2
  Lines          9764     9562      -202
  Branches       2068     2030       -38
=============================================
- Hits           4751     1479     -3272
- Misses         3993     7903     +3910
+ Partials       1020      180      -840

Impacted Files
Coverage Δ

...toidrive/music/controllers/SpotifyAppController.kt
0.00% <0.00%> (-60.19%)
:arrow_down:

...n/androidautoidrive/music/spotify/SpotifyWebApi.kt
0.00% <0.00%> (-24.61%)
:arrow_down:

...in/java/me/hufman/androidautoidrive/AppSettings.kt
39.01% <100.00%> (-41.62%)
:arrow_down:

...a/me/hufman/androidautoidrive/music/MusicAction.kt
0.00% <0.00%> (-100.00%)
:arrow_down:

...me/hufman/androidautoidrive/music/QueueMetadata.kt
0.00% <0.00%> (-100.00%)
:arrow_down:

...hufman/androidautoidrive/carapp/maps/MapAppMode.kt
0.00% <0.00%> (-100.00%)
:arrow_down:

...hufman/androidautoidrive/utils/SparseArrayUtils.kt
0.00% <0.00%> (-100.00%)
:arrow_down:

...an/androidautoidrive/carapp/FocusedStateTracker.kt
0.00% <0.00%> (-100.00%)
:arrow_down:

...dautoidrive/phoneui/viewmodels/MapSettingsModel.kt
0.00% <0.00%> (-100.00%)
:arrow_down:

...autoidrive/carapp/notifications/ReplyController.kt
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 90 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f4ff5dc...3bcbefc. Read the comment docs.

That failing unit test seems to only be occurring on the Github job since locally it is passing, looking into this issue 😬

Thank you so much!


Do not create Jobs and the service when JobSet is suspended

To reproduce just create a simple JobSet with spec.suspend=true. In that case Jobs and the service are created.
This is wasteful in case of Jobs which are queued by Kueue, and may stay in the queue for a long time, potentially.

/kind feature

Maybe not a "big" deal, but Kueue is typically used to hold long Kueues of suspended Jobs(or JobSets), say 50k, so would be nice to create them.
I imagine it would be fine to keep a once created service for a JobSet that got suspended (it was running, but got preempted). There should not be too many of them, and we could save on recreation in case the JobSet is quickly re-admitted.

I think the main tricky point would be support for startup policy and suspend.
Our implementation with suspend and startup policy was to resume the replicated jobs in order of their listing.
I guess this could clean this up as we would only create the jobs if they were resumed. But it may be a bit tricky to implement...

/remove-lifecycle stale
I'm investigating this so that we can fix https://github.com/kubernetes-sigs/jobset/pull/625

I don't quite follow why this is required for #625. Can you explain?

I don't quite follow why this is required for #625. Can you explain?

Consider the following chain of states:  suspend - resume 1 - suspend - resume.2.
Here resume 1 and 2 may use different Pod templates (updated during suspend), so we need to recreate the Jobs at some point.
Deleting the in the suspend phase seems simplest. The Job controller also deletes pods on suspend

I see. So Jobs delete pods on a resume/suspend but JobSet was keeping the Jobs around?

Consider the following chain of states: suspend - resume 1 - suspend - resume.2.
Here resume 1 and 2 may use different Pod templates (updated during suspend), so we need to recreate the Jobs at some point.

@mimowo what is the current status of this issue, can we close it? We relaxed pod template mutability constraints in JobSet in order to allow for this integration with Kueue, does this not solve the issue described in your comment above?

Yeah, it solves 98% (guesstimation) of problems. Let me summarize the remaining motivation to still do it (maybe as an extra option not to break the default flow):

transitioning JobSet via the chain (ResourceFlavor1) -> suspend -> resume (ResourceFlavor2) never remove nodeSelectors assigned before (ResourceFlavor1), just override them. This might be a problem if ResourceFlavor1 has nodeSelector: some-key: some-value, but ResourceFlavor2's nodeSelector does not specify the value for some-key. Then, it will still keep the old one in the Pod template, potentially not allowing for kubernetes-level scheduling. However, this is not an issue typically, because the ResourceFlavor2 will typically specify a new value for some-key.

For JobSets with many replicated Jobs it takes a lot of API-server resources even when the JobSet remains in Kueue.

Having said that I'm ok to close it for now, because the remaining issues haven't yet been a problem for any user I know of, so it is not a high priority to solve them proactively.

IIUC it could also be helpful for lazy quota reservation for Execution Policy (see comment), but maybe the KEP will go another way, so not sure.