Bug with last chrome version and slick

Hi,
I found a bug: compatibility problem between slick and last Chrome version.
Slick version: 1.9.0
jQuery version:
Chrome version: 72.0.3626.96
This was working smoothly with the previous version of Chrome, the bug has arrived since the update, without any other changes in code.
In responsive mode, when my desktop is at least 782px, the slick is perfect.
Slick is perfect, no bug
But, when my desktop is less than 782px, the slick is deformed, images are too big (width > 209230px !!)
Slick is deformed, bug
Have you the same problem ?
Thank you

Sorry, I have resolved my pb... it was a css flex bug, not a slick bug.
I close my issue, best regards.

We are experiencing a similar issue it seems.  @gmesnard what did you do to fix, do you remember?


Blade::directive method does not exist

Not a recognised method.

Did you try adding a \ before Blade? and importing the class ofcourse.


Add sphinx documentation bones to templates as optional deployment.

It would be useful if we could provide the option to include the bare bones of sphinx documentation config when deploying a template. What do you think @asbfall , @jacobevansgit ?

As per https://github.com/pyansys/create-python-project/pull/49#issuecomment-1039957154 I will do this. Plan is to setup a  basic sphinx documentation build, with some of the bells and whistles we have at PyAnsys to ensure the easiest documentation writing.

Default theme (alabaster) - pyansys theme should only be used for pyansys products
Autodoc
numpydoc (to be consistent with pyansys)
mdinclude (to allow for the auto-inclusion of the README amongst other things)
A "Getting Started" markdown file explaining how to build the docs

Additionally, I need to go through the templates and add docstrings to any example classes/methods so that there is some lorem ipsum documentation to build right off the bat.


[Analysis] Display borders in network maps

Analysing errors in connection nodes and routes is difficult because it often relies on where the actual border between two operators is located. It would help a lot if the network map showed that border in addition to showing the network and its neighbours.
It is understood that finding what is the right border to display may pose a problem. Often, the name of the network or its operator will give a strong hint but not an immediate mapping. Maybe this could be solved by a table, or by using wikidata.

With the changes to the map styling from issue #410 (which improves how connections to other networks are shown), I guess it is now easier to see the border between networks.  I am closing this issue because I do not see a way to unambiguously draw a border between networks of different operators.


Drop Node 4 support in v0.21.0

As it will reach "end of life" on the 30th April 2018.
https://github.com/nodejs/Release#release-schedule

Hey @lovell , first, fantastic lib just started using, switching over from imgmagik & it's doing wonders.
It seems this release, while it builds & compiles just fine in Node 9.x it broke our (aws) docker containers' build. After upgrading node locally & our docker node image from 9.5.0 to 10.0.0 everything's kosher on v0.21.0 & 22. Just wanted to mention in case anyone else ran into the issue.
 System Linux 4.9.87-linuxkit-aufs
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /app/code/node_modules/sharp
gyp ERR! node -v v9.5.0
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok

If you drop node 4 maybe it's better to do it in a major version don't you think ?

@billouboq sharp has yet to reach v1.0.0 so semver allows for "breaking" changes with a minor version increment.
By way of example, support for Node 0.10 and 0.12, present in sharp v0.16.2, was dropped in v0.17.0. This was discussed in #610.

v0.21.0 now available with this change.


add tiered word list critera

Signed-off-by: Celeste Horgan<EMAIL_ADDRESS>

What's the basis for deciding which tiers apply?  "i know it when I see it" ?

What's the basis for deciding which tiers apply? "i know it when I see it" ?

I would presume that this determination would be made in discussion with the language workstream. There's no one "right" answer to any of these things.

That's fine, if that's what happened, but allowing peopl3e t

What's the basis for deciding which tiers apply? "i know it when I see it" ?

I would presume that this determination would be made in discussion with the language workstream. There's no one "right" answer to any of these things.

That's fine, but it should be explicit in the doc.
We all know the Dewey Decimal system was arbitrary, but it was still incredibly useful.
The doc should say how things were categorized.

That's fine, if that's what happened, but allowing peopl3e t

What's the basis for deciding which tiers apply? "i know it when I see it" ?

I would presume that this determination would be made in discussion with the language workstream. There's no one "right" answer to any of these things.

That's fine, but it should be explicit in the doc.
We all know the Dewey Decimal system was arbitrary, but it was still incredibly useful.
The doc should say how things were categorized.

That's a good point. It could have a statement that the initial list was developed and categorized by X working group or similar, and then specify when new words are being proposed for inclusion the proposing party can suggest the categorization and then it's confirmed by the WG - or some other process as applicable. That way it's clear the history and the process moving forward.

That's fine, if that's what happened, but allowing peopl3e t

What's the basis for deciding which tiers apply? "i know it when I see it" ?

I would presume that this determination would be made in discussion with the language workstream. There's no one "right" answer to any of these things.

That's fine, but it should be explicit in the doc.
We all know the Dewey Decimal system was arbitrary, but it was still incredibly useful.
The doc should say how things were categorized.

At the moment, it would be the call of the language WS and those writing/reviewing the recommendations.

At the moment, it would be the call of the language WS and those writing/reviewing the recommendations.

That's fine, but I would really like it to say something like "Classified by the language workstream of INI" or something.
It also gives us another chance to say "if you don't like it, come join us" :)

Has no-one noticed that this change has broken links to the word-list so it cannot be found now...


Looking to Gain Better Understanding of How to Customize Policy/Role Assignments

Hey All,
I've been pouring over all of the documentation concerning the module on how to customize it to fit my needs. I am struggling a bit to understand how to tailor policy/role definitions and assignments at specific scopes. I am trying to accomplish the following.
Under the Landing Zone Management Group I have (2) custom management groups defined:
Landing Zones
Non-Prod
Production
My desire is to set policy definitions/assignments at these levels because I will have department level management groups below them like so and I would like policy to flow down to those child MGs and their underlying subscriptions
Landing Zones
Non-Prod <--- Non-Prod Policy Defs/Assignments HERE
Department_1_Non-Prod <--- Custom Role Definitions/Assignments HERE (different team/user access, etc)
Department_1_Dev-Test_Subscription
Department_1_Staging_Subscription
Department_2_Non-Prod <--- Custom Role Definitions/Assignments HERE  (different team/user access, etc);
Department_2_Dev-Test_Subscription
Department_2_Staging_Subscription
Production <--- Production Policy Defs/Assignments HERE
Department_1_Production <--- Custom Role Definitions/Assignments HERE (different team/user access, etc)
Department_1_Production_Subscription
Department_2_Production <--- Custom Role Definitions/Assignments HERE  (different team/user access, etc);
Department_2_Production_Subscription'
I have the base of this setup so far in by code, but so far I have had no luck demystifying how to set up the policy and role assignments the way I would like to:
main.tf
....
custom_landing_zones = {
"Non-Prod-MG" = {
display_name               = "Non-Prod"
parent_management_group_id = "${var.root_id}-landing-zones"
subscription_ids           = []
archetype_config = {
archetype_id   = "default_empty"
parameters     = {}
access_control = {}
}
}
"Dept-1-Non-Prod-MG" = {
display_name = "Dept-1-Non-Prod-MG"
parent_management_group_id = "Non-Prod-MG"
subscription_ids = []
archetype_config = {
archetype_id   = "default_empty"
parameters     = {}
access_control = {}
  }
}
"Prod-MG" = {
  display_name               = "Prod"
  parent_management_group_id = "${var.root_id}-landing-zones"
  subscription_ids           = []
  archetype_config = {
  archetype_id   = "default_empty"
  parameters     = {}
  access_control = {}
}
  }
"Dept-1-Prod-MG" = {
  display_name = "Dept-1-Prod-MG"
  parent_management_group_id = "Prod-MG"
  subscription_ids = []
  archetype_config = {
  archetype_id   = "default_empty"
  parameters     = {}
  access_control = {}

  }

Any help in trying to get to this goal would be greatly appreciated.

UPDATE
After some late night tinkering my customization is beginning to take shape.
I have my /lib directory setup with a two archetypes and a role definition like so:
/lib/archetype_definition_non_prod
{
    "non_prod": {
      "policy_assignments": ["Deny-Resource-Locations", "Deny-RSG-Locations"],
      "policy_definitions": [],
      "policy_set_definitions": [],
      "role_definitions": [],
      "archetype_config": {
        "parameters": {
          "Deny-Resource-Locations": {
            "listOfAllowedLocations": [
              "eastus",
              "eastus2"
            ]
          },
          "Deny-RSG-Locations": {
            "listOfAllowedLocations": [
              "eastus",
              "eastus2"
            ]
          }
        },
        "access_control": {}
      }
    }
  }

/lib/archetype_definition_dept1_non_prod
{
    "dept1_non_prod": {
      "policy_assignments": [],
      "policy_definitions": [],
      "policy_set_definitions": [],
      "role_definitions": ["Dept1-Developers"],
      "archetype_config": {
        "parameters": {},
        "access_control": {}
      }
    }
  }

/lib/role_definition_es_dept1_developers.tmpl.json
{
    "name": "c9a07a05-a1fc-53fe-a565-5eed25597c04",
    "type": "Microsoft.Authorization/roleDefinitions",
    "apiVersion": "2018-01-01-preview",
    "properties": {
        "roleName": "Dept1-Developers",
        "description": "Contributor role granted for application/operations team at resource group level",
        "type": "customRole",
        "permissions": [
            {
                "actions": [
                    "*"
                ],
                "notActions": [
                    "Microsoft.Authorization/*/write",
                    "Microsoft.Network/publicIPAddresses/write",
                    "Microsoft.Network/virtualNetworks/write",
                    "Microsoft.KeyVault/locations/deletedVaults/purge/action"
                ],
                "dataActions": [],
                "notDataActions": []
            }
        ],
        "assignableScopes": [
            "${current_scope_resource_id}"
        ]
    }
}

main.tf
....

custom_landing_zones = {
    "Non-Prod-MG" = {
      display_name               = "Non-Prod"
      parent_management_group_id = "${var.root_id}-landing-zones"
      subscription_ids           = []
      archetype_config = {
      archetype_id   = "non_prod"
      parameters     = {}
      access_control = {}
    }
      }
    "Helix-Non-Prod-MG" = {
      display_name = "Dept1-Non-Prod-MG"
      parent_management_group_id = "Non-Prod-MG"
      subscription_ids = []
      archetype_config = {
      archetype_id   = "dept1_non_prod"
      parameters     = {}
      access_control = {}

      }
    }

Questions
Where are the policies defined for the top level built-in management groups (below) and how can I modify/override them? I am asking because there are many defaults that are set that are being pushed down and I would like to tweak (and expand) on those to better suit my environment
Top Level Management Group
  Platform
    Connectivity
    Management
  Landing Zone
  Sandbox

# can I modify the built-in/default policies/role assignments implemented for these management groups


CUDA error when running TheBloke/Chronoboros-33B-GPTQ

System Info
Running on RunPod.io, A6000 and L40 both have the error
Docker image: ghcr.io/huggingface/text-generation-inference:1.0.3
Arguments: --model-id TheBloke/Chronoboros-33B-GPTQ --revision gptq-4bit-32g-actorder_True --hostname <IP_ADDRESS> --port 2345 --quantize gptq  --max-input-length 4095 --max-total-tokens 4096 --max-batch-prefill-tokens 4096 --max-batch-total-tokens 4096
Env: CUDA_LAUNCH_BLOCKING = 1 (as I understand, stack trace is unreliable without this)
Information

[X] Docker
[ ] The CLI directly

Tasks

[X] An officially supported command
[ ] My own modifications

Reproduction
First I had to do the workaround from #1027 to fix GPTQ config error. After that, the model warmup failed.
Usually the only error in the logs was nonspecific Warmup(Generation("transport error")), I accidentaly sent many requests during the warmup and only then got the CUDA error (at the bottom).
Logs
2023-09-14T16:26:08.822613Z  INFO text_generation_launcher: Args { model_id: "TheBloke/Chronoboros-33B-GPTQ", revision: Some("gptq-4bit-32g-actorder_True"), validation_workers: 2, sharded: None, num_shard: None, quantize: Some(Gptq), dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 4095, max_total_tokens: 4096, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: Some(4096), max_waiting_tokens: 20, hostname: "<IP_ADDRESS>", port: 2345, shard_uds_path: "/tmp/text-generation-server", master_addr: "localhost", master_port: 29500, huggingface_hub_cache: Some("/data"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: false, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }
2023-09-14T16:26:31.121335Z  INFO text_generation_launcher: Using exllama kernels
2023-09-14T16:26:31.673141Z  INFO text_generation_launcher: Server started at unix:///tmp/text-generation-server-0
2023-09-14T16:26:31.677564Z  INFO shard-manager: text_generation_launcher: Shard ready in 17.724036519s rank=0
2023-09-14T16:26:31.772486Z  INFO text_generation_launcher: Starting Webserver
2023-09-14T16:26:32.223773Z  INFO text_generation_router: router/src/main.rs:371: Serving revision bb33e4b2806a076ca79a6e31276707c92c0055fb of model TheBloke/Chronoboros-33B-GPTQ
2023-09-14T16:26:32.230922Z  INFO text_generation_router: router/src/main.rs:213: Warming up model
2023-09-14T16:26:39.886131Z ERROR warmup{max_input_length=4095 max_prefill_tokens=4096}:warmup: text_generation_client: router/client/src/lib.rs:33: Server error: transport error
Error: Warmup(Generation("transport error"))
2023-09-14T16:26:39.983305Z ERROR text_generation_launcher: Webserver Crashed
2023-09-14T16:26:39.983335Z  INFO text_generation_launcher: Shutting down shards
2023-09-14T16:26:40.057421Z ERROR shard-manager: text_generation_launcher: Shard complete standard error output:
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. If you see this, DO NOT PANIC! This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=True`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
You are using a model of type llama to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
CUDA Error: an illegal memory access was encountered /usr/src/flash-attention/csrc/layer_norm/ln_fwd_kernels.cuh 236 rank=0
Error: WebserverFailed

Expected behavior
Model should be warmed up without error.

Loading models with act-order false seems to be working correctly, There is some issue with act-order either not being supported correctly or not being read from config

Came here because I got reports - I think from @Pyroserenus  - that act-order True models aren't working any more?
I recently dropped making act-order False models because my understanding was that they weren't needed any more, as all clients properly supported True at full performance.
It'd be great to get an update on what's happening here. I'm trying to cut down on the GPTQ parameter permutations I'm making to reduce cost and maintenance requirements.  All other clients support group_size + act-order without issues so it'd be great if this could be restored to TGI as well.

OK I just tested this myself and can confirm the error with both 1.0.3 and :latest
I believe this is a regression - I am sure this used to work OK.
Error log from :latest, trying to load branch main from TheBloke/Pygmalion-2-13B-SuperCOT-weighed-GPTQ (4-bit / 128g / True):

Error log
Launching TGI with args: --model-id TheBloke/Pygmalion-2-13B-SuperCOT-weighed-GPTQ --max-input-length 3896 --max-concurrent-requests 15 --max-best-of 1 --max-total-tokens 4096 --quantize gptq --huggingface-hub-cache /workspace/huggingface --hostname <IP_ADDRESS> --port 3000
2023-09-17T13:38:50.185049Z  INFO text_generation_launcher: Args { model_id: "TheBloke/Pygmalion-2-13B-SuperCOT-weighed-GPTQ", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: Some(Gptq), dtype: None, trust_remote_code: false, max_concurrent_requests: 15, max_best_of: 1, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 3896, max_total_tokens: 4096, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: "<IP_ADDRESS>", port: 3000, shard_uds_path: "/tmp/text-generation-server", master_addr: "localhost", master_port: 29500, huggingface_hub_cache: Some("/workspace/huggingface"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: false, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }
2023-09-17T13:38:50.185275Z  INFO download: text_generation_launcher: Starting download process.
2023-09-17T13:38:53.656405Z  INFO text_generation_launcher: Download file: model.safetensors

2023-09-17T13:40:07.829347Z  INFO text_generation_launcher: Downloaded /workspace/huggingface/models--TheBloke--Pygmalion-2-13B-SuperCOT-weighed-GPTQ/snapshots/fc95509196df4ed5a55be97da7e35d4330a649a4/model.safetensors in 0:01:14.

2023-09-17T13:40:07.829529Z  INFO text_generation_launcher: Download: [1/1] -- ETA: 0

2023-09-17T13:40:08.359904Z  INFO download: text_generation_launcher: Successfully downloaded weights.
2023-09-17T13:40:08.360460Z  INFO shard-manager: text_generation_launcher: Starting shard rank=0
2023-09-17T13:40:15.354482Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.434238Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.470551Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.503296Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.523954Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.549716Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.570537Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.601930Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.622802Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.649182Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.670121Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.701898Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.722780Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.749114Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.770147Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.796754Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.823831Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.850061Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.870900Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.897341Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.921081Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.952399Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.973250Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:15.999229Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.019960Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.056295Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.088843Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.119539Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.141906Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.167691Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.188971Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.222087Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.243515Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.269779Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.290782Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.320942Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.342082Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.368299Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.389486Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.415536Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.441571Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.467912Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.489434Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.515918Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.541408Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.590103Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.610894Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.636988Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.658005Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.683968Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.716256Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.747509Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.768584Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.794719Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.817744Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.857494Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.891122Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.932027Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.953465Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:16.979875Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.001150Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.033224Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.054131Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.080410Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.101632Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.127771Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.161246Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.202106Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.223214Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.249446Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.270777Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.302789Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.324111Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.350274Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.371695Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.398164Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.432827Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.459071Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.480277Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:17.506744Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:18.026437Z  INFO text_generation_launcher: Server started at unix:///tmp/text-generation-server-0

2023-09-17T13:40:18.072278Z  INFO shard-manager: text_generation_launcher: Shard ready in 9.710261282s rank=0
2023-09-17T13:40:18.169409Z  INFO text_generation_launcher: Starting Webserver
2023-09-17T13:40:18.538238Z  WARN text_generation_router: router/src/main.rs:349: `--revision` is not set
2023-09-17T13:40:18.538265Z  WARN text_generation_router: router/src/main.rs:350: We strongly advise to set it to a known supported commit.
2023-09-17T13:40:18.697236Z  INFO text_generation_router: router/src/main.rs:371: Serving revision fc95509196df4ed5a55be97da7e35d4330a649a4 of model TheBloke/Pygmalion-2-13B-SuperCOT-weighed-GPTQ
2023-09-17T13:40:18.703065Z  INFO text_generation_router: router/src/main.rs:213: Warming up model
2023-09-17T13:40:28.176658Z ERROR text_generation_launcher: Method Warmup encountered an error.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/flash_causal_lm.py", line 753, in warmup
    _, batch = self.generate_token(batch)
  File "/opt/conda/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/flash_causal_lm.py", line 860, in generate_token
    next_input_ids, next_token_logprobs, logprobs = batch.next_token_chooser(
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/tokens.py", line 220, in __call__
    scores = self.watermark_processor(input_ids, scores)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/logits_process.py", line 398, in __call__
    scores[i : i + 1] = processor(input_ids[i : i + 1], scores[i : i + 1])
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/watermark.py", line 88, in __call__
    greenlist_ids = self._get_greenlist_ids(
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/watermark.py", line 62, in _get_greenlist_ids
    self._seed_rng(input_ids)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/watermark.py", line 52, in _seed_rng
    prev_token = input_ids[-1].item()
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/bin/text-generation-server", line 8, in <module>
    sys.exit(app())
  File "/opt/conda/lib/python3.9/site-packages/typer/main.py", line 311, in __call__
    return get_command(self)(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/typer/core.py", line 778, in main
    return _main(
  File "/opt/conda/lib/python3.9/site-packages/typer/core.py", line 216, in _main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/typer/main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/cli.py", line 81, in serve
    server.serve(
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/server.py", line 195, in serve
    asyncio.run(
  File "/opt/conda/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 634, in run_until_complete
    self.run_forever()
  File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
    self._run_once()
  File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
    handle._run()
  File "/opt/conda/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/opt/conda/lib/python3.9/site-packages/grpc_interceptor/server.py", line 159, in invoke_intercept_method
    return await self.intercept(
> File "/opt/conda/lib/python3.9/site-packages/text_generation_server/interceptor.py", line 21, in intercept
    return await response
  File "/opt/conda/lib/python3.9/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 82, in _unary_interceptor
    raise error
  File "/opt/conda/lib/python3.9/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 73, in _unary_interceptor
    return await behavior(request_or_iterator, context)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/server.py", line 66, in Warmup
    max_supported_total_tokens = self.model.warmup(batch)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/flash_causal_lm.py", line 755, in warmup
    raise RuntimeError(
RuntimeError: Not enough memory to handle 4096 prefill tokens. You need to decrease `--max-batch-prefill-tokens`

2023-09-17T13:40:28.177986Z ERROR warmup{max_input_length=3896 max_prefill_tokens=4096}:warmup: text_generation_client: router/client/src/lib.rs:33: Server error: Unexpected <class 'RuntimeError'>: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Error: Warmup(Generation("Unexpected <class 'RuntimeError'>: CUDA error: an illegal memory access was encountered\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n"))
2023-09-17T13:40:28.282520Z ERROR text_generation_launcher: Webserver Crashed
2023-09-17T13:40:28.282565Z  INFO text_generation_launcher: Shutting down shards
2023-09-17T13:40:28.707406Z  INFO shard-manager: text_generation_launcher: Shard terminated rank=0
Error: WebserverFailed
TGI stopped. It will restart in 1 second.
Launching TGI with args: --model-id TheBloke/Pygmalion-2-13B-SuperCOT-weighed-GPTQ --max-input-length 3896 --max-concurrent-requests 15 --max-best-of 1 --max-total-tokens 4096 --quantize gptq --huggingface-hub-cache /workspace/huggingface --hostname <IP_ADDRESS> --port 3000
2023-09-17T13:40:29.747980Z  INFO text_generation_launcher: Args { model_id: "TheBloke/Pygmalion-2-13B-SuperCOT-weighed-GPTQ", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: Some(Gptq), dtype: None, trust_remote_code: false, max_concurrent_requests: 15, max_best_of: 1, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 3896, max_total_tokens: 4096, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: "<IP_ADDRESS>", port: 3000, shard_uds_path: "/tmp/text-generation-server", master_addr: "localhost", master_port: 29500, huggingface_hub_cache: Some("/workspace/huggingface"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: false, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }
2023-09-17T13:40:29.748187Z  INFO download: text_generation_launcher: Starting download process.
2023-09-17T13:40:32.887604Z  INFO text_generation_launcher: Files are already present on the host. Skipping download.

2023-09-17T13:40:33.354085Z  INFO download: text_generation_launcher: Successfully downloaded weights.
2023-09-17T13:40:33.354559Z  INFO shard-manager: text_generation_launcher: Starting shard rank=0
2023-09-17T13:40:38.733101Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:38.825067Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:38.855893Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:38.893413Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:38.913132Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:38.938450Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:38.958335Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:38.987066Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.006917Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.032237Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.052067Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.077336Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.105401Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.131027Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.150935Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.176404Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.196082Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.224966Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.244779Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.270217Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.289994Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.319171Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.350999Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.399122Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.430690Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.479081Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.510790Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.550933Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.571519Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.596852Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.618604Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.647641Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.667874Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.693393Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.713178Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.738666Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.763726Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.789165Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.809328Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.834609Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.855187Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.884562Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.904962Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.930584Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.950750Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.979976Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:39.999786Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.025177Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.045435Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.073015Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.104709Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.143613Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.163896Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.189308Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.209615Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.238621Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.271192Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.319822Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.352283Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.390273Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.410842Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.436931Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.457396Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.487758Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.508369Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.533832Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.554304Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.580141Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.608350Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.633855Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.654337Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.679872Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.701096Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.730594Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.751156Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.776988Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.797328Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.825228Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.845645Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:40.873521Z  INFO text_generation_launcher: Using exllama kernels

2023-09-17T13:40:41.653855Z  INFO text_generation_launcher: Server started at unix:///tmp/text-generation-server-0

2023-09-17T13:40:41.666787Z  INFO shard-manager: text_generation_launcher: Shard ready in 8.310915574s rank=0
2023-09-17T13:40:41.763601Z  INFO text_generation_launcher: Starting Webserver
2023-09-17T13:40:41.931259Z  WARN text_generation_router: router/src/main.rs:349: `--revision` is not set
2023-09-17T13:40:41.931285Z  WARN text_generation_router: router/src/main.rs:350: We strongly advise to set it to a known supported commit.
2023-09-17T13:40:42.067553Z  INFO text_generation_router: router/src/main.rs:371: Serving revision fc95509196df4ed5a55be97da7e35d4330a649a4 of model TheBloke/Pygmalion-2-13B-SuperCOT-weighed-GPTQ
2023-09-17T13:40:42.076579Z  INFO text_generation_router: router/src/main.rs:213: Warming up model
2023-09-17T13:40:45.546296Z ERROR text_generation_launcher: Method Warmup encountered an error.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/flash_causal_lm.py", line 753, in warmup
    _, batch = self.generate_token(batch)
  File "/opt/conda/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/flash_causal_lm.py", line 860, in generate_token
    next_input_ids, next_token_logprobs, logprobs = batch.next_token_chooser(
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/tokens.py", line 220, in __call__
    scores = self.watermark_processor(input_ids, scores)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/logits_process.py", line 398, in __call__
    scores[i : i + 1] = processor(input_ids[i : i + 1], scores[i : i + 1])
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/watermark.py", line 88, in __call__
    greenlist_ids = self._get_greenlist_ids(
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/watermark.py", line 62, in _get_greenlist_ids
    self._seed_rng(input_ids)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/watermark.py", line 52, in _seed_rng
    prev_token = input_ids[-1].item()
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/bin/text-generation-server", line 8, in <module>
    sys.exit(app())
  File "/opt/conda/lib/python3.9/site-packages/typer/main.py", line 311, in __call__
    return get_command(self)(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/typer/core.py", line 778, in main
    return _main(
  File "/opt/conda/lib/python3.9/site-packages/typer/core.py", line 216, in _main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/typer/main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/cli.py", line 81, in serve
    server.serve(
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/server.py", line 195, in serve
    asyncio.run(
  File "/opt/conda/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 634, in run_until_complete
    self.run_forever()
  File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
    self._run_once()
  File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
    handle._run()
  File "/opt/conda/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/opt/conda/lib/python3.9/site-packages/grpc_interceptor/server.py", line 159, in invoke_intercept_method
    return await self.intercept(
> File "/opt/conda/lib/python3.9/site-packages/text_generation_server/interceptor.py", line 21, in intercept
    return await response
  File "/opt/conda/lib/python3.9/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 82, in _unary_interceptor
    raise error
  File "/opt/conda/lib/python3.9/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 73, in _unary_interceptor
    return await behavior(request_or_iterator, context)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/server.py", line 66, in Warmup
    max_supported_total_tokens = self.model.warmup(batch)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/flash_causal_lm.py", line 755, in warmup
    raise RuntimeError(
RuntimeError: Not enough memory to handle 4096 prefill tokens. You need to decrease `--max-batch-prefill-tokens`

2023-09-17T13:40:45.547431Z ERROR warmup{max_input_length=3896 max_prefill_tokens=4096}:warmup: text_generation_client: router/client/src/lib.rs:33: Server error: Unexpected <class 'RuntimeError'>: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Error: Warmup(Generation("Unexpected <class 'RuntimeError'>: CUDA error: an illegal memory access was encountered\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n"))
2023-09-17T13:40:45.577703Z ERROR text_generation_launcher: Webserver Crashed
2023-09-17T13:40:45.577736Z  INFO text_generation_launcher: Shutting down shards

Looks like the bug is with ExLlama.  This workaround appears to be working:  launch with DISABLE_EXLLAMA=True
Still testing, will update in a moment

I have been working with @Pyroserenus trying to narrow down the cause of this.
Tested using:

bare metal A100 SXM 80GB and A100 PCIe 40GB
TGI git+c8a01d759173483efc2135c4e7506b23e14e7fc4 (after 1.0.3)
TheBloke/Airoboros-L2-13B-2.2-GPTQ main and gptq-4bit-32g-actorder_True

Observations:

can only reproduce with act_order = True quants
DISABLE_EXLLAMA=True makes it work
lowering max-batch-prefill-tokens tokens works, the cutoff seems to be inconsistently at around 3200
different group sizes (128g vs 32g) don't have a large impact on the cutoff, if any
the cutoff doesn't change between 80GB and 40GB VRAM models

The last point suggests that this is not a VRAM issue, as it shouldn't be with 13B quants on 40/80GB cards.
My stack trace is identical to yours when it happens.

@TheBloke I'll be able to test later and g4rg is testing now.
If disabling exllama is a substantial performance regression I'll likely just wait for a proper fix for exllama from TGI or exllama2 to be added, whichever comes first, and just host existing models until then. I wouldn't want you to keep doing no act order just for a small handful of users.

lowering max-batch-prefill-tokens tokens works, the cutoff seems to be inconsistently at around 3200

Oh that's interesting. There's an ExLlama kernel paramter which is described in Optimum (https://github.com/huggingface/optimum/blob/c63138778c0c08b96ee49ea35f6346ff42a21437/optimum/gptq/quantizer.py#L112-L114) like so:
 max_input_length (`Optional[int]`, defaults to `None`):
                The maximum input length. This is needed to initialize a buffer that depends on the maximum expected input length.
                It is specific to the exllama backend with act-order.

I wonder if this is connected - like TGI is not setting that maximum input length, or is setting it incorrectly.

If disabling exllama is a substantial performance regression

I don't know what it is in TGI as I've not tested it. But in AutoGPTQ the ExLlama kernels increased token/s by 20 - 30%. So useful, but not life changing.

I'll likely just wait for a proper fix for exllama from TGI or exllama2 to be added, whichever comes first, and just host existing models on horde until then.

Almost certainly the former. EXL2 is effectively a whole new format, still under development. It's likely to be a little while before it makes its way it into a server like TGI.

I wouldn't want you to keep doing no act order just for a small handful of users.

Yeah I won't be now I know it's a bug and there's a workaround. I really don't want to start adding act-order False models again as they're lower quality and I'm trying to reduce my costs in general as they're getting pretty excessive, especially for 34B and 70B models. And soon I'll be adding AWQ and EXL2 versions of every new model as well.

Just confirming that this is still a problem in TGI v1.1.0, and yep the workaround (DISABLE_EXLLAMA=True) results in about ~28% performance drop for me. I'm using 13B models - I haven't tried to replicate the bug with other sizes.

Wondering whether this is related, at all?

https://github.com/huggingface/text-generation-inference/issues/1098

Will test docker image for v1.1.1 once it has built.

Seems to work fine for me without DISABLE_EXLLAMA=True in v1.1.1 - I think this issue can be closed.

We are still experiencing this issue with v1.2.0
Even though ExllamaV2 is now available, this does not help in our case since we need to shard the model which is not supported with v2 at the moment.
The only options still seem to be to fall back to the triton kernels or reduce the prefill tokens to something lower than 3k (which is also not an option in our case because we a trying to use a model with a context length of 8k).

I forgot to update my comment about this, but I'm seeing some errors (not sure if related to this) for specific models only - e.g. a popular-ish one that has the error is MythoMax:
2023-12-07T00:02:37.784973580Z 2023-12-07T00:02:37.784823Z ERROR shard-manager: text_generation_launcher: Shard complete standard error output:
2023-12-07T00:02:37.784995680Z
2023-12-07T00:02:37.784997770Z /opt/conda/lib/python3.10/site-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at /opt/conda/conda-bld/pytorch_1699449201336/work/c10/cuda/CUDAFunctions.cpp:108.)
2023-12-07T00:02:37.785001330Z   return torch._C._cuda_getDeviceCount() > 0
2023-12-07T00:02:37.785002870Z /opt/conda/lib/python3.10/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
2023-12-07T00:02:37.785004830Z   warn("The installed version of bitsandbytes was compiled without GPU support. "
2023-12-07T00:02:37.785006270Z Traceback (most recent call last):
2023-12-07T00:02:37.785007330Z
2023-12-07T00:02:37.785008310Z   File "/opt/conda/bin/text-generation-server", line 8, in <module>
2023-12-07T00:02:37.785016459Z     sys.exit(app())
2023-12-07T00:02:37.785017969Z
2023-12-07T00:02:37.785018969Z   File "/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py", line 83, in serve
2023-12-07T00:02:37.785020449Z     server.serve(
2023-12-07T00:02:37.785021509Z
2023-12-07T00:02:37.785022469Z   File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 207, in serve
2023-12-07T00:02:37.785030309Z     asyncio.run(
2023-12-07T00:02:37.785031979Z
2023-12-07T00:02:37.785032949Z   File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run
2023-12-07T00:02:37.785034089Z     return loop.run_until_complete(main)
2023-12-07T00:02:37.785035179Z
2023-12-07T00:02:37.785036149Z   File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
2023-12-07T00:02:37.785037249Z     return future.result()
2023-12-07T00:02:37.785038679Z
2023-12-07T00:02:37.785039629Z   File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 159, in serve_inner
2023-12-07T00:02:37.785040699Z     model = get_model(
2023-12-07T00:02:37.785041739Z
2023-12-07T00:02:37.785042749Z   File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/__init__.py", line 213, in get_model
2023-12-07T00:02:37.785044259Z     return CausalLM(
2023-12-07T00:02:37.785045299Z
2023-12-07T00:02:37.785046299Z   File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/causal_lm.py", line 492, in __init__
2023-12-07T00:02:37.785047389Z     raise ValueError("quantization is not available on CPU")
2023-12-07T00:02:37.785048909Z
2023-12-07T00:02:37.785049859Z ValueError: quantization is not available on CPU
2023-12-07T00:02:37.785051559Z  rank=0
2023-12-07T00:02:37.884017799Z 2023-12-07T00:02:37.883853Z ERROR text_generation_launcher: Shard 0 failed to start

That's with ghcr.io/huggingface/text-generation-inference:1.2.0 and --quantize gptq --model-id TheBloke/MythoMax-L2-13B-GPTQ --max-total-tokens 4096 --port 3000 on a Runpod 3090.
But other Llama 13B models work fine - e.g. TheBloke/LLaMA2-13B-Tiefighter-GPTQ works, which is strange because I was under the impression that they're the same model, just with different weights.
Again, not sure if the above issue is related to the other issues people are having on this thread. This issue isn't blocking me at all - just adding what little evidence I have to help others debug.

I meet the issue too, use llama with num-shard 4 quantize gptq


feat: add mail deliverybox

@tillwestpfahl Is this ready for review?

Nope, I was waiting for the Readme-Issue to be fixed to continue with this.


Move  to the end of the class name, not the beginning

Motivation

Test plan

See included automated tests.

We have a policy of testing changes to Sorbet against Stripe's codebase before
merging them. I've kicked off a test run for the current PR.  When the build
finishes, I'll share with you whether or how it failed. Thanks!
Stripe employees can see the build results here:
→ https://go/builds/bui_PZOkB0cYYESest
→ https://go/builds/bui_PZOkAitm3vMHRe
→ https://go/builds/bui_PZOkFzvWtTPVVB

This changes so little at this point that I might just drop the changes to add <PackageSpec> to the names by way of rewriting the trees, but leave the change to start determining symbol ownership by package hierarchy, because I think that I'm going to have to revert the tree rewriting aspects when I move package defining into namer.

So feel free to review for the time being, but know that I might hold off on landing this until I know more about whether it's going to want to be in this shape long term.

We have a policy of testing changes to Sorbet against Stripe's codebase before
merging them. I've kicked off a test run for the current PR.  When the build
finishes, I'll share with you whether or how it failed. Thanks!
Stripe employees can see the build results here:
→ https://go/builds/bui_PZkPEH8XkByQUe
→ https://go/builds/bui_PZkPcSfJklkrO8
→ https://go/builds/bui_PZkPdMrwaBkYwM

We have a policy of testing changes to Sorbet against Stripe's codebase before
merging them. I've kicked off a test run for the current PR.  When the build
finishes, I'll share with you whether or how it failed. Thanks!
Stripe employees can see the build results here:
→ https://go/builds/bui_PZkaBBvqmqp3An
→ https://go/builds/bui_PZkaHI5GRyiGUR
→ https://go/builds/bui_PZkaQ648WhxP3M

We have a policy of testing changes to Sorbet against Stripe's codebase before
merging them. I've kicked off a test run for the current PR.  When the build
finishes, I'll share with you whether or how it failed. Thanks!
Stripe employees can see the build results here:
→ https://go/builds/bui_PZkfVxslLflUJR
→ https://go/builds/bui_PZkgC1dN2vOrTj
→ https://go/builds/bui_PZkg9RROGveXns


Add LIGHT space

Add space for ...
Etherscan: https://cn.etherscan.com/token/0x0dBB4b96a23Ea7943BB72a8F16cc2c248F372935
Website: https://lights.finance/#/home
Twitter: https://twitter.com/FinanceLights

Mark has already been adjusted


Update habitat build of openresty to <IP_ADDRESS>

The omnibus build was already updated to <IP_ADDRESS>, but we should
generally bump the habitat build at the same time.
This is mostly bugfixes over the previous <IP_ADDRESS> (which had already
patched for nginx CVE-2021-23017).

Adhoc - https://buildkite.com/chef/chef-chef-server-main-omnibus-adhoc/builds/3814


documentation not entirely clear about how to implement recurring jobs

First off: thanks for your efforts.
I use Rails6 and delayed_job and want a job that runs daily at 7:00 am.
I use dokku do deploy and do not want to set up an trigger to enque the jobs.
I will figure out how delayed_cron_jobs can help me. I have to say from the README itself I cannot figure that out. If you accept PRs, I will draft a documentation PR. I sense that it might also be desirable to have a generator for creating app/jobs/cron_job.rb and a def cron_expression=value in it, but I will have to dig deeper to understand if that would be really helpful.
Please give me a feedback before I write up some documentation draft.

Sure, feel free to improve whatever you find. Please separate the single issues into single pull requests.

okay, I'll have a look. I think, what was missing for me was an example of a CronJob subclass (with self.cron_expression = '* * * * *'.
I would put my explanations in doc/rails6_example.md for now, which is .gitignored at the moment. It might end up in the README but I would leave that up to you. Do you object doc/*md ?

I prefer to have stuff simple. Just put it in the README. Otherwise, if you think of an extended example for a special use case, the wiki might be another option.

Ya, I dont like documentation very much outside of the repo (
https://github.com/fwolfst/your-repository/issues/2) or the gem (
https://github.com/fwolfst/your-repository/issues/1). Will try to make it
concise so it fits the readme.
On Mon, Apr 6, 2020 at 12:15 PM z<EMAIL_ADDRESS>wrote:

I prefer to have stuff simple. Just put it in the README. Otherwise, if
you think of an extended example for a special use case, the wiki might be
another option.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/codez/delayed_cron_job/issues/19#issuecomment-609705036,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABVR3LC5GJA7T2NBD7YH4LRLGTUVANCNFSM4MA7YO7Q
.


bump-formula-pr: more refactoring

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[ ] Have you successfully run brew tests with your changes locally?

This will be useful for brew bump-cask-pr (#7986)
utils/github: add forked_repo_info!

Also deleted occurrences of GitHub. and GitHub:: that are redundant
Replaced api_errors method with constant API_ERRORS

utils/inreplace: add inreplace_pairs

#8177 introduced a bug:

Also updates inreplace_pairs to check for --write in addition to --dry-run

inreplace_pairs already checked for --write, just not in the if statement

Closing for now


Can it support M1 Mac ?

and I had this problem and solved by changing Wifi setting
Pip error, OSError Errno 22 invalid argument
Python version: 3.11.2
OS: macOS Ventura 13.2 M1

That would be awesome!

No, get a new computer and stop asking people to support your bad decision.

No, get a new computer and stop asking people to support your bad decision.

Not a bad descision to support a apple silicon chip. So maybe u stop commenting on such things.

Please explain why time should be wasted accommodating a minuscule portion of Apple's 8% market share. Why should every completed project have to waste time making it work for you? Why don't you learn how to use Xcode & Rosetta and do it yourself?
I use Windows, but you don't see me in threads asking people to port to Docker or whatever VENV I prefer, I get off my rear and do it myself. Welcome to Github.

Please explain why time should be wasted accommodating a minuscule portion of Apple's 8% market share. Why should every completed project have to waste time making it work for you? Why don't you learn how to use Xcode & Rosetta and do it yourself?
I use Windows, but you don't see me in threads asking people to port to Docker or whatever VENV I prefer, I get off my rear and do it myself. Welcome to Github.

ur the one wasting time here by posting this stuff. Newbies exist bro. maybe ur just pissed because you cant afford a mac. dont take it out here

Please explain why time should be wasted accommodating a minuscule portion of Apple's 8% market share. Why should every completed project have to waste time making it work for you? Why don't you learn how to use Xcode & Rosetta and do it yourself?
I use Windows, but you don't see me in threads asking people to port to Docker or whatever VENV I prefer, I get off my rear and do it myself. Welcome to Github.

Respectfully, here's the explanation:

Market share is not a great proxy for your argument. This is a dev heavy website, but you're using a % which represents the global population. Look at the stack overflow developer survey and you'll see ~32% share of devs using it for personal/work.

Ollama is one of the most popular ways to run LLMs locally, and this is one of its most requested features (Windows support): https://github.com/ollama/ollama/issues/403

"I get off my rear and do it myself. Welcome to Github." -- you have zero contributions in your profile; maybe this is a dummy profile, who knows. Maybe, through that same logic, you should have got off your rear and created the code for Shap-E yourself?

Open-source is about sharing contributions and building on others' work. Requesting features helps devs gauge interest—OP did so respectfully.


Enable OSX builds on Travis (#660).

#660
cc @victorhurdugaci @halter73

:shipit: if it passes

Will the travis badge in our README only stay green if both builds pass?

Yes.


Fix race condition in resident_runner

Don't set the vmServices member variable until it's fully initialized.
Add a timeout to the future that sends the 'started' event to the IDE

https://github.com/flutter/flutter/issues/16604

LGTM (assuming 1 minute is reasonable for the work being done there)


Debugger won't attach

Tried various settings in the launch.json files. Can't get it do attach or hit any breakpoints.

What platform are you on?
What version of Unity?

OS X 10.10.5 and Unity 5.2.1f1

Same here. Same versions.

There were apparently a bunch of fixes in 0.9.0 to help with attaching / etc.
Can you possibly turn on the output messages option and see if it is finding the port? and if that port is being written to your launch settings file for vscode.

Yeah, I verified that the launch.json file is being written with the correct port:
"version":"0.1.0",
"configurations":[
{
"name":"Unity",
"type":"mono",
"address":"localhost",
"port":56139
}
]

Damn! Just upgraded to the new (today) VS Code 0.9.1 and the debugger worked perfectly the first time! ...and have NOT been able to get it to attach since. :-( Tried restarting apps, comp, etc. No go. There must be some cache file it's referencing??

OK, I was finally able to solve this! Not sure the exact file that was the culprit but basically dumped all of the generated code files from the Unity project folder (.csproj, obj/Debug). You can even delete the Project Settings and Library folder if you want. Unity will re-create everything and the VSCode attaches and works perfectly fine.

Sounds like the files were being locked and not able to be overwritten?
Version control? Perforce?

Hi I'm unable to connect debugger too. If I try to start debug I get a "Debug adapter process has terminated unexpectedly" error on vscode.
I'm using latest vscode 0.10.3, latest unity 5.2.3f1 and Yosemite os

Can the problem be related to the mono version installed in the system? I'm not speaking about unity monodevelop version.
Having no luck with your plugin I've found this: http://forum.unity3d.com/threads/vs-code-unity-debugger-extension-preview.369775/
Following the instructions I've installed latest monodevelop version (and the vscode plugin provided). After that I was able to successfully connect the debugger also using your plugin

Indeed, Unity's plugin provides the right debugger framework (instead of relying on Mono internal). I've been chatting with them about the future of my plugin.
Ultimately they won't be able to get their support 'fully' for another 6 months and it sounds like they'd like mine to stick around for that time. One thing that I'd like to add to mine is installing that debugger extension for you automatically, etc.
We'll see ... I haven't had time to evaluate doing that yet, work keeps my occupied.

I'm going to close this now with the addition of the Unity Debugger side of things to the plugin. My suggestion is to use it :)

This is still a continuing problem, and I'm now no longer able to fix it by deleting the aforementioned files. I have the VSCode debugger installed, as well as the VSCode editor extension and all the boxes checked. The debugger spits out a message in the console that it attached, and then about 3 seconds later it disconnects. Can't hit any breakpoints. :-(

@zsculpt
buddy, do you solve that attached at first and then disconnect ?  same problem i meet :-(

No - I gave up trying to use the debugger until someone can provide any more troubleshooting steps. It just doesn't work.