[CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…

…rt overwrite partition
Why is this PR needed?
If update/delete some records on one partition, then inserts overwrite another partition. Deleted records before will be queried out.
What changes were proposed in this PR?
When insert overwrite (or drop) partition, the updatestatusfilename in tablestatus will be tampered, which is fixed in this PR.
Does this PR introduce any user interface change?

No

Is any new testcase added?

Yes

Build Failed  with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbon_PR_Builder_2.4.5/2654/

retest this please

Build Failed  with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbon_PR_Builder_2.4.5/2656/

Build Failed  with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/4410/

Build Failed  with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/4412/

Build Failed  with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbon_PR_Builder_2.4.5/2661/

Build Failed  with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/4415/

retest this please

Build Failed  with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/4418/

Build Failed  with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbon_PR_Builder_2.4.5/2664/

retest this please

Build Failed  with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbon_PR_Builder_2.4.5/2668/

Build Failed  with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/4422/

Build Success with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/4431/

Build Failed  with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbon_PR_Builder_2.4.5/2679/

retest this please

Build Failed  with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbon_PR_Builder_2.4.5/2686/

retest this please

Build Success with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/4472/

retest this please

Build Success with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/4499/

Build Success with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbon_PR_Builder_2.4.5/2745/

Build Success with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbon_PR_Builder_2.4.5/2790/

Build Failed  with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/4544/

retest this please

Build Failed  with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbon_PR_Builder_2.4.5/2801/

Build Failed  with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/4552/

please do rebase

Build Failed  with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/4671/

Build Failed  with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbon_PR_Builder_2.4.5/2915/

Build Success with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/4674/

Build Success with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbon_PR_Builder_2.4.5/2917/

retest this please

please do rebase

I have do rebase according to your suggestion

Build Success with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbon_PR_Builder_2.4.5/2934/

Build Success with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/4691/

@marchpure please correct the PR title and please give detailed explanation of the issue and the fix in the PR description.

@marchpure please correct the PR title and please give detailed explanation of the issue and the fix in the PR description.

Yes. I have modified the PR desc as your suggestion

retest this please

Build Success with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbon_PR_Builder_2.4.5/2952/

@marchpure as from the description, i understood that the table status file after insert overwrite do not contain the updatetablestatusfile  name. So why do we need to add boolean variable, and send this boolean from many places. This may lead to confusion later also. So why dont we take care it just while updating the table status in case of insert overwrite?
I mean to say, while updating the status after insert overwrite, we can first copy the existing meta and then prepare new entry and update case can be handled there, this way, it will lead to less changes and less confusion later.

@marchpure please correct the PR title and please give detailed explanation of the issue and the fix in the PR description.

Yes. I have modified the PR desc as your suggestion

PR title is still not yet formatted, please check

@marchpure : This is already handled in #3999 , need to close this PR ?


kotlin file hash mismatch

Description
The Packer build runs and when it gets to installing JetBrains/kotlin it seems the FileHash is not matching what the release shows on kotlin's repository.
Checking kotlin's release from 10h ago the SHA the script expects is: 15A8A2825B74CCF6C44E04E97672DB802D2DF75CE2FBB63EF0539BF3AE5006F0 (which is the one showing on the assets).
The script however, seems to use a different hash when it runs in our self-hosted agent (Linux):
FullyQualifiedErrorId : Checksum verification failed. Expected hash: kotlincompiler1920zip; Actual hash: 15A8A28
25B74CCF6C44E04E97672DB802D2DF75CE2FBB63EF0539BF3AE5006F0

Platforms affected

[X] Azure DevOps
[ ] GitHub Actions - Standard Runners
[ ] GitHub Actions - Larger Runners

Runner images affected

[ ] Ubuntu 20.04
[ ] Ubuntu 22.04
[ ] macOS 11
[ ] macOS 12
[ ] macOS 13
[ ] Windows Server 2019
[X] Windows Server 2022

Image version and build link
We don't have anything public. Our pipeline pulls this repository down (latest from main branch) and runs the helper script.
This failure is running on a Linux self-hosted agent (image built in same manner) and attempting to build a Windows VM Image.
Is it regression?
No
Expected behavior
File hash matches and step proceeds
Actual behavior
==> azure-arm.image: Provisioning with powershell script: /agent/_work/1/s/RunnerImages/images/win/scripts/Installers/Install-Kotlin.ps1
    azure-arm.image: Downloading package from: https://github.com/JetBrains/kotlin/releases/download/v1.9.20/kotlin-compiler-1.9.20.zip to path C:\Users\packer\AppData\Local\Temp\kotlin-compiler.zip .
    azure-arm.image: Package downloaded successfully in 2.04 seconds
==> azure-arm.image: Checksum verification failed. Expected hash: kotlincompiler1920zip; Actual hash:
==> azure-arm.image: 15A8A2825B74CCF6C44E04E97672DB802D2DF75CE2FBB63EF0539BF3AE5006F0.
==> azure-arm.image: At C:\Program Files\WindowsPowerShell\Modules\ImageHelpers\InstallHelpers.ps1:673 char:9
==> azure-arm.image: +         throw "Checksum verification failed. Expected hash: $Distribu ...
==> azure-arm.image: +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
==> azure-arm.image:     + CategoryInfo          : OperationStopped: (Checksum verifi...539BF3AE5006F0.:String) [], RuntimeException
==> azure-arm.image:     + FullyQualifiedErrorId : Checksum verification failed. Expected hash: kotlincompiler1920zip; Actual hash: 15A8A28
==> azure-arm.image:    25B74CCF6C44E04E97672DB802D2DF75CE2FBB63EF0539BF3AE5006F0.
==> azure-arm.image:
==> azure-arm.image: Script exited with non-zero exit status: 1. Allowed exit codes are: [0]
==> azure-arm.image: Step "StepProvision" failed
==> azure-arm.image: Provisioning step had errors: Running the cleanup provisioner, if present...

Step fails due to the hash not matching
Repro steps
Our process calls the helper function for calling the Packer process in this repository:
    Push-Location -Path $RunnerImagesRepositoryPath

    Write-Verbose "Packer build starting"
    $env:PACKER_LOG = 1
    $env:PACKER_LOG_PATH = "$pwd/packer-log.txt"
    $helperScript = [IO.Path]::Join($RunnerImagesRepositoryPath, 'helpers', 'GenerateResourcesAndImage.ps1')
    Import-Module $helperScript -ErrorAction Stop

    $packerBuildParams = @{
        SubscriptionID                = $subscriptionId
        ResourceGroupName             = $ResourceGroupName
        ImageType                     = $ImageType.ToLower()
        ManagedImageName              = $packerManagedImageName
        AzureLocation                 = "West US 2"
        ImageGenerationRepositoryRoot = "$($pwd)"
        AzureTenantId                 = $tenantId
        ReuseResourceGroup            = $true
        Tags                          = @{ ExcludeMdeAutoProvisioning = "true" }
    }

    <#
        Retrieve secrets for the Service Connection from kvdeployment<env>unique
    #>
    $keyVaultName = "kvdeployment$($environmentCode)unique"
    $clientAppId = Get-AzKeyVaultSecret -VaultName $keyVaultName -Name "serviceConnectionAppId" -AsPlainText
    $clientSecret = Get-AzKeyVaultSecret -VaultName $keyVaultName -Name "serviceConnectionAgentSecret" -AsPlainText
    $packerBuildParams.Add('AzureClientId', $clientAppId)
    $packerBuildParams.Add('AzureClientSecret', $clientSecret)
    GenerateResourcesAndImage @packerBuildParams -ErrorAction Stop | ForEach-Object {
        $currentString = $_
        if ($currentString -match '(OSDiskUri|OSDiskUriReadOnlySas|TemplateUri|TemplateUriReadOnlySas|AMI|ManagedImageId|ManagedImageName|ManagedImageResourceGroupName|ManagedImageLocation|ManagedImageSharedImageGalleryId): (.*)') {
            $varName = $matches[1]
            $varValue = $matches[2]

            Write-Verbose "Variable [$($varName):$($varValue)] created"
            New-Variable -Name $varName -Value $varValue -Force

            Write-Host "##vso[task.setvariable variable=$varName;isOutput=true]$varValue"
            <#
            Packer execution above will output the following variables
                $ManagedImageResourceGroupName: 'rg-example-azusw2-dvo-rprd'
                $ManagedImageName: 'Ubuntu2004-2023.09.22'
                $ManagedImageId: '/subscriptions/00000000000000000/resourceGroups/rg-example-azusw2-dvo-rprd/providers/Microsoft.Compute/images/Ubuntu2004-2023.09.22'
                $ManagedImageLocation: 'West US 2'
            #>
        }
        Write-Output $_
    }
    Write-Verbose "Packer build finished"

Hey @wsmelton! Sorry to say it but @nlighten was 5 minutes faster. 😄
Closing item as a duplicate https://github.com/actions/runner-images/issues/8707

All good 🥳


Support for custom base_api

Feature request
Custom the base_api url for OpenAI
Motivation
I use OneAPI to distribute api keys. It requires to sending requests to its website instead of the official URL.
Your contribution
Hmmm

hey @JackuXL ,
You can create an instance of OpenAIClient with a different apiBaseUrl like this:
final openai = OpenAI(
  apiClient: OpenAIClient.instanceFor(
    apiKey: 'api-key',
    apiBaseUrl: 'your-api-base-url',
    organization: 'your-organization',
  ),
);

@davidmigloz
Thank you for your answer🙏
Btw, I'm wondering if you have any plan for serpapi or google-search wrapper?

@JackuXL We should definitely add support for it, but I haven't had time to implement it. Would you like to contribute?

@davidmigloz
I'll give it a shot, but I'm not well-versed in Dart, so no promises. 😢

Awesome! Let me know if you need some guidance 🙂


Triton ensemble model configuration for transducer models

Hello, could you please give a reference configuration for the ensemble transducer model in the example repository for a pure triton-based inference? Specifically, how do we interface the variable y from the scorer back to the decoder input? Also, the template misses the joiner_decoder_proj and joiner_decoder_proj parts.

I am working on it now, and managed to follow the docs to make triton server up and running(all components including ensemble transducer are up and running), but when I am try to start client script it fails.
python3 decode_manifest_triton.py --manifest-filename /mnt/efs/dspavankumar/e/tamil_icefall/data/test_re/icefall_manifests/cuts_1.jsonl.gz --server-addr <IP_ADDRESS> --server-port 8001 --streaming  --model-name transducer --chunk_size 16 --context 2

tritonclient.utils.InferenceServerException: [StatusCode.INVALID_ARGUMENT] in ensemble 'transducer', inference request for sequence 10107 to model 'feature_extractor' must specify the START flag on the first request of the sequence

One weird thing I noticed was that when I start the server, I see these warnings/errors
Cleaning up...
free(): invalid pointer
free(): invalid pointer

Is this possibly related to memory leak issue ?
https://github.com/triton-inference-server/server/issues/3777

Also, I am trying pretrained model from section "Deploy onnx with arbitrary pruned_transducer_stateless_X(2,3,4,5) model for Chinese or English recipes"
at https://github.com/k2-fsa/sherpa/tree/master/triton
After downloading the model files, I am getting the following error:
./pruned_transducer_stateless3/export_onnx.py \
    --exp-dir ./icefall_librispeech_streaming_pruned_transducer_stateless3_giga_0.9_20220625/exp \
    --tokenizer-file ./icefall_librispeech_streaming_pruned_transducer_stateless3_giga_0.9_20220625/data/lang_bpe_500/bpe.model \
    --epoch 999 \
    --avg 1 \
    --streaming-model 1 \
    --causal-convolution 1 \
    --onnx 1 \
    --left-context 64 \
    --right-context 4 \
    --fp16

    sp.load(params.tokenizer_file)
File "/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/site-packages/sentencepiece/__init__.py", line 905, in Load
  return self.LoadFromFile(model_file)
File "/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/site-packages/sentencepiece/__init__.py", line 310, in LoadFromFile
  return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
RuntimeError: Internal: src/sentencepiece_processor.cc(1101) [model_proto->ParseFromArray(serialized.data(), serialized.size())]

@csukuangfj
I am getting this error when I try to run default streaming example provided in sherpa/triton folder (https://github.com/k2-fsa/sherpa/tree/master/triton/model_repo_streaming)
tritonclient.utils.InferenceServerException: [StatusCode.INTERNAL] in ensemble 'transducer', Failed to process the request(s) for model instance 'feature_extractor_0_0', message: Exception: ('Invalid first chunk size', 9360, 14880)
https://github.com/k2-fsa/sherpa/blob/master/triton/model_repo_streaming/feature_extractor/1/model.py#L46
Did you guys encounter this issue as well ? What is your current status on transducer setup for triton, is it stable for you guys ?
I would appreciate if you can suggest some pointers to address this issue, I can spend some time to fix issues if there any known issues, or if you want me try something else to make it work.
Also, as mentioned in previous comment(https://github.com/k2-fsa/sherpa/issues/371#issuecomment-1530912862), I suspect its related kaldifeats library memory leak issue, if this is a known issue, do you suggest to try using a different library for feature extraction ?

@yuekaizhang
Could you help to have a look at this issue?

I am working on it now, and managed to follow the docs to make triton server up and running(all components including ensemble transducer are up and running), but when I am try to start client script it fails.
python3 decode_manifest_triton.py --manifest-filename /mnt/efs/dspavankumar/e/tamil_icefall/data/test_re/icefall_manifests/cuts_1.jsonl.gz --server-addr <IP_ADDRESS> --server-port 8001 --streaming  --model-name transducer --chunk_size 16 --context 2

tritonclient.utils.InferenceServerException: [StatusCode.INVALID_ARGUMENT] in ensemble 'transducer', inference request for sequence 10107 to model 'feature_extractor' must specify the START flag on the first request of the sequence

One weird thing I noticed was that when I start the server, I see these warnings/errors
Cleaning up...
free(): invalid pointer
free(): invalid pointer

Is this possibly related to memory leak issue ? triton-inference-server/server#3777

Hi,thanks for trying this triton recipe.

inference request for sequence 10107 to model 'feature_extractor' must specify the START flag on the first request of the sequence
This error may be caused by outdated request. At the beginning of the service startup, due to insufficient warming up, if a request is cleared due to timeout, it will cause later arriving chunks to lose their start flag.  You may first try to warmup service with small batch size and concurrency.
free(): invalid pointer This warning (which I have no idea yet) should be fine.
tritonclient.utils.InferenceServerException: [StatusCode.INTERNAL] in ensemble 'transducer', Failed to process the request(s) for model instance 'feature_extractor_0_0', message: Exception: ('Invalid first chunk size', 9360, 14880) This issue is caused by --context 2, you should use --encoder_right_context which is for icefall models. https://github.com/k2-fsa/sherpa/blob/master/triton/client/decode_manifest_triton.py#L161 This is for wenet models.

Hi @yuekaizhang
thanks for your response.
I ensured that the config parameters in $model_repo_path/*/config.pbtxt are matching properties as per onnx export log file
icefall_librispeech_streaming_pruned_transducer_stateless3_giga_0.9_20220625/exp/onnx_export.log
For reference:
ENCODER_LEFT_CONTEXT: 64
ENCODER_RIGHT_CONTEXT: 4
ENCODER_DIM: 512
DECODER_DIM: 512
VOCAB_SIZE: 500
DECODER_CONTEXT_SIZE: 2
CNN_MODULE_KERNEL: 31
ENCODER_LAYERS: 12
All params:{'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 50, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'encoder_dim': 512, 'nhead': 8, 'dim_feedforward': 2048, 'num_encoder_layers': 12, 'decoder_dim': 512, 'joiner_dim': 512, 'model_warm_step': 3000, 'env_info': {'k2-version': '1.23.4', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '62e404dd3f3a811d73e424199b3408e309c06e1a', 'k2-git-date': 'Mon Jan 30 02:26:16 2023', 'lhotse-version': '1.12.0', 'torch-version': '1.13.0', 'torch-cuda-available': True, 'torch-cuda-version': '11.6', 'python-version': '3.1', 'icefall-git-branch': None, 'icefall-git-sha1': None, 'icefall-git-date': None, 'icefall-path': '/mnt/efs/dspavankumar/tools/icefall', 'k2-path': '/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/site-packages/k2/__init__.py', 'lhotse-path': '/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/site-packages/lhotse/__init__.py', 'hostname': 'ip-10-40-5-20', 'IP address': '<IP_ADDRESS>'}, 'epoch': 1111, 'iter': 0, 'avg': 1, 'exp_dir': PosixPath('icefall_librispeech_streaming_pruned_transducer_stateless3_giga_0.9_20220625/exp'), 'tokenizer_file': './icefall_librispeech_streaming_pruned_transducer_stateless3_giga_0.9_20220625/data/lang_bpe_500/bpe.model', 'onnx': True, 'context_size': 2, 'left_context': 64, 'right_context': 4, 'streaming_model': True, 'fp16': True, 'dynamic_chunk_training': False, 'causal_convolution': True, 'short_chunk_size': 25, 'num_left_chunks': 4, 'blank_id': 0, 'vocab_size': 500}

Then I ran the client again python3 decode_manifest_triton.py --encoder_right_context 4 --chunk_size 16 --manifest-filename /mnt/efs/dspavankumar/e/tamil_icefall/data/test_re/icefall_manifests/cuts.jsonl.gz --server-addr <IP_ADDRESS> --server-port 8001 --streaming --model-name transducer
Still getting same error
task-48: 0/221
task-49: 0/221
Traceback (most recent call last):
  File "/mnt/efs/dspavankumar/tools/sherpa/triton/client/decode_manifest_triton.py", line 485, in <module>
    asyncio.run(main())
  File "/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/asyncio/base_events.py", line 649, in run_until_c
omplete
    return future.result()
  File "/mnt/efs/dspavankumar/tools/sherpa/triton/client/decode_manifest_triton.py", line 433, in main
    ans_list = await asyncio.gather(*tasks)
  File "/mnt/efs/dspavankumar/tools/sherpa/triton/client/decode_manifest_triton.py", line 316, in send_streaming
    response = await triton_client.infer(model_name,
  File "/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/site-packages/tritonclient/grpc/aio/__init__.py",
 line 727, in infer
    raise_error_grpc(rpc_error)
  File "/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/site-packages/tritonclient/grpc/__init__.py", lin
e 62, in raise_error_grpc
    raise get_error_grpc(rpc_error) from None
tritonclient.utils.InferenceServerException: [StatusCode.INTERNAL] in ensemble 'transducer', Failed to process the request(s) fo
r model instance 'feature_extractor_0_1', message: Exception: ('Invalid first chunk size', 12640, 14880)

At:
  /mnt/efs/dspavankumar/tools/sherpa/triton/model_repo_streaming_pretrained/feature_extractor/1/model.py(47): add_wavs
  /mnt/efs/dspavankumar/tools/sherpa/triton/model_repo_streaming_pretrained/feature_extractor/1/model.py(221): execute

Hi @yuekaizhang thanks for your response.
I ensured that the config parameters in $model_repo_path/*/config.pbtxt are matching properties as per onnx export log file icefall_librispeech_streaming_pruned_transducer_stateless3_giga_0.9_20220625/exp/onnx_export.log
For reference:
ENCODER_LEFT_CONTEXT: 64
ENCODER_RIGHT_CONTEXT: 4
ENCODER_DIM: 512
DECODER_DIM: 512
VOCAB_SIZE: 500
DECODER_CONTEXT_SIZE: 2
CNN_MODULE_KERNEL: 31
ENCODER_LAYERS: 12
All params:{'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 50, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'encoder_dim': 512, 'nhead': 8, 'dim_feedforward': 2048, 'num_encoder_layers': 12, 'decoder_dim': 512, 'joiner_dim': 512, 'model_warm_step': 3000, 'env_info': {'k2-version': '1.23.4', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '62e404dd3f3a811d73e424199b3408e309c06e1a', 'k2-git-date': 'Mon Jan 30 02:26:16 2023', 'lhotse-version': '1.12.0', 'torch-version': '1.13.0', 'torch-cuda-available': True, 'torch-cuda-version': '11.6', 'python-version': '3.1', 'icefall-git-branch': None, 'icefall-git-sha1': None, 'icefall-git-date': None, 'icefall-path': '/mnt/efs/dspavankumar/tools/icefall', 'k2-path': '/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/site-packages/k2/__init__.py', 'lhotse-path': '/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/site-packages/lhotse/__init__.py', 'hostname': 'ip-10-40-5-20', 'IP address': '<IP_ADDRESS>'}, 'epoch': 1111, 'iter': 0, 'avg': 1, 'exp_dir': PosixPath('icefall_librispeech_streaming_pruned_transducer_stateless3_giga_0.9_20220625/exp'), 'tokenizer_file': './icefall_librispeech_streaming_pruned_transducer_stateless3_giga_0.9_20220625/data/lang_bpe_500/bpe.model', 'onnx': True, 'context_size': 2, 'left_context': 64, 'right_context': 4, 'streaming_model': True, 'fp16': True, 'dynamic_chunk_training': False, 'causal_convolution': True, 'short_chunk_size': 25, 'num_left_chunks': 4, 'blank_id': 0, 'vocab_size': 500}

Then I ran the client again python3 decode_manifest_triton.py --encoder_right_context 4 --chunk_size 16 --manifest-filename /mnt/efs/dspavankumar/e/tamil_icefall/data/test_re/icefall_manifests/cuts.jsonl.gz --server-addr <IP_ADDRESS> --server-port 8001 --streaming --model-name transducer
Still getting same error
task-48: 0/221
task-49: 0/221
Traceback (most recent call last):
  File "/mnt/efs/dspavankumar/tools/sherpa/triton/client/decode_manifest_triton.py", line 485, in <module>
    asyncio.run(main())
  File "/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/asyncio/base_events.py", line 649, in run_until_c
omplete
    return future.result()
  File "/mnt/efs/dspavankumar/tools/sherpa/triton/client/decode_manifest_triton.py", line 433, in main
    ans_list = await asyncio.gather(*tasks)
  File "/mnt/efs/dspavankumar/tools/sherpa/triton/client/decode_manifest_triton.py", line 316, in send_streaming
    response = await triton_client.infer(model_name,
  File "/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/site-packages/tritonclient/grpc/aio/__init__.py",
 line 727, in infer
    raise_error_grpc(rpc_error)
  File "/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/site-packages/tritonclient/grpc/__init__.py", lin
e 62, in raise_error_grpc
    raise get_error_grpc(rpc_error) from None
tritonclient.utils.InferenceServerException: [StatusCode.INTERNAL] in ensemble 'transducer', Failed to process the request(s) fo
r model instance 'feature_extractor_0_1', message: Exception: ('Invalid first chunk size', 12640, 14880)

At:
  /mnt/efs/dspavankumar/tools/sherpa/triton/model_repo_streaming_pretrained/feature_extractor/1/model.py(47): add_wavs
  /mnt/efs/dspavankumar/tools/sherpa/triton/model_repo_streaming_pretrained/feature_extractor/1/model.py(221): execute

" This error may be caused by outdated request. At the beginning of the service startup, due to insufficient warming up, if a request is cleared due to timeout, it will cause later arriving chunks to lose their start flag. You may first try to warmup service with small batch size and concurrency."
I also tried with --num-tasks 1 argument, but it still fails.
/mnt/efs/dspavankumar/tools/sherpa/triton/client$ python3 decode_manifest_triton.py --num-tasks 1 --encoder_right_context 4 --chunk_size 16 --manifest-filename /mnt/efs/dspavankumar/e/tamil_icefall/data/test_re/icefall_manifests/cuts.jsonl.gz --server-addr <IP_ADDRESS> --server-port 8001 --streaming --model-name transducer
task-0: 0/11077
/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/site-packages/lhotse/audio.py:164: UserWarning: You requested a subset of a recording that is read from disk via a bash command. Expect large I/O overhead if you are going to read many chunks like these, since every time we will read the whole file rather than its subset.
  warnings.warn(
Traceback (most recent call last):
  File "/mnt/efs/dspavankumar/tools/sherpa/triton/client/decode_manifest_triton.py", line 485, in <module>
    asyncio.run(main())
  File "/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/mnt/efs/dspavankumar/tools/sherpa/triton/client/decode_manifest_triton.py", line 433, in main
    ans_list = await asyncio.gather(*tasks)
  File "/mnt/efs/dspavankumar/tools/sherpa/triton/client/decode_manifest_triton.py", line 316, in send_streaming
    response = await triton_client.infer(model_name,
  File "/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/site-packages/tritonclient/grpc/aio/__init__.py", line 727, in infer
    raise_error_grpc(rpc_error)
  File "/mnt/efs/dspavankumar/tools/miniconda3/envs/icefall_env/lib/python3.10/site-packages/tritonclient/grpc/__init__.py", line 62, in raise_error_grpc
    raise get_error_grpc(rpc_error) from None
tritonclient.utils.InferenceServerException: [StatusCode.INVALID_ARGUMENT] in ensemble 'transducer', inference request for sequence 10086 to model 'feature_extractor' must specify the START flag on the first request of the sequence

https://github.com/k2-fsa/sherpa/blob/master/triton/client/decode_manifest_triton.py#L381-L383
Here, please check your first_chunk_ms, decoding_window_length, decode_window_length = (args.chunk_size + 2 + args.encoder_right_context) * args.subsampling + 3 decode_window_length should be (16 + 2 + 4)*4 + 3 = 91, first_chunk_ms = (decode_window_length + add_frames) * frame_shift_ms add_frames should be 2.

I have checked, these values are correct.
ipdb> decode_window_length
91
ipdb> print(args.chunk_size )
16
ipdb> print(args.encoder_right_context)
4
ipdb> print(args.subsampling)
4
ipdb> print(add_frames)
2
ipdb> print(frame_shift_ms)
10

Failed to process the request(s) fo
r model instance 'feature_extractor_0_1', message: Exception: ('Invalid first chunk size', 12640, 14880)

If the values are correct, could you trace back to figure out how do you get this 12640 number?

Hi @yuekaizhang
I noticed that in the wav_segs(https://github.com/k2-fsa/sherpa/blob/master/triton/client/decode_manifest_triton.py#L269), in the last segment, the number of samples(length) are different from all other segments, causing an issue.
So after adding
del(wav_segs[-1])
at https://github.com/k2-fsa/sherpa/blob/master/triton/client/decode_manifest_triton.py#L282 problem is fixed.
Do you think this is a bug?

I am not sure. If it is a bug, it will exist in feature_extractor/1/model.py rather than this client here. Could you make sure that https://github.com/k2-fsa/sherpa/blob/master/triton/model_repo_streaming/feature_extractor/1/model.py#L52 here assert len(self.wav) > 0 always hold? Otherwise, there is a problem somewhere.

Good observation,
so that error was caused due to few very short cuts present in jsonl, I used a different cuts file(which didnt have very short (<0.3 seconds cuts)), and I think that might have fixed that particular issue.

Okay, close the issue since it is fixed.


can you give me some paper or references of UVa/Padova simulitor

can you give me some paper or references of UVa/Padova simulitor,I can learn a lot params and formula
of your code

The link to UVA Padova 2013 paper is in the README:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4454102/
or
https://journals.sagepub.com/doi/pdf/10.1177/1932296813514502
This paper (UVA Padova 2013) includes the Glucagon pathway as well, but simglucose is based on the 2008 version, which only has the insulin pathway. However, you'll find everything you might want to know about the 2008 version in the 2013 paper. Hope this helps!

Thanks @sumanabasu for the paper pointers. Will close this issue.


Custom filter implementation

This probably is a dumb question, but I am stuck with this for a while so going to ask it anyway.
I'm trying to implement a Hudson/Nashville filter on a pet project. I googled a little and checkout out a few open-source projects and found some Objective-C (which I don't understand) based projects. They do have the filters implemented using GPUImage, but I wasn't sure about their approach.
I have the overlay and other images that they have used and the GLSL files.
So my question is how do I go about using this images and shader files to implement a custom filter?
Note: I tried using the LookupFilter approach as suggested, but the result wasn't so good. I know I am asking too much, but it would be super helpful if you can show me some code. Thanks

I understand your problem, this is the case, CPUimage generation and second generation is more different, you can use photoshop action file and lookup file to do differentiated approach, other methods for the time being do not know.

@BradLarson @henvyluk What I am trying to understand this. Given a custom shader like the one below, how am I supposed to pass the input images for uniform inputImageTexture2, inputImageTexture3 & inputImageTexture4. Do I pass it as a PictureInput to BasicOperation by subclassing it? If so, how? What am I missing? I haven't been able to walk through the code much because of the lack of a proper documentation. I have read up on shaders and its different components now, but still not able to figure out a way to work with custom filters on GPUImage2. Please help.
precision highp float;

varying highp vec2 textureCoordinate;

uniform sampler2D inputImageTexture;
uniform sampler2D inputImageTexture2; //blowout;
uniform sampler2D inputImageTexture3; //overlay;
uniform sampler2D inputImageTexture4; //map

uniform float strength;

void main()
{
    vec4 originColor = texture2D(inputImageTexture, textureCoordinate);

    vec4 texel = texture2D(inputImageTexture, textureCoordinate);

    vec3 bbTexel = texture2D(inputImageTexture2, textureCoordinate).rgb;

    texel.r = texture2D(inputImageTexture3, vec2(bbTexel.r, texel.r)).r;
    texel.g = texture2D(inputImageTexture3, vec2(bbTexel.g, texel.g)).g;
    texel.b = texture2D(inputImageTexture3, vec2(bbTexel.b, texel.b)).b;

    vec4 mapped;
    mapped.r = texture2D(inputImageTexture4, vec2(texel.r, .16666)).r;
    mapped.g = texture2D(inputImageTexture4, vec2(texel.g, .5)).g;
    mapped.b = texture2D(inputImageTexture4, vec2(texel.b, .83333)).b;
    mapped.a = 1.0;

    mapped.rgb = mix(originColor.rgb, mapped.rgb, strength);

    gl_FragColor = mapped;
}

These shader file in gpuimage2 is useless, at least I do not know how to deal with. Can try my method:
1, import lookup.png to ps after processing derived newlookup.png
2, the establishment of BasicOperation subclass that is your custom filter
3, add the LookupFilter property of the lookupImage property that newlookup.png
BradLarson's explanation is that practice

Finally was able to figure out something.
public class CustomFilter: BasicOperation {

    public var strength: Float = 1.0 { didSet { uniformSettings["strength"] = strength } }
    public var inputs: [PictureInput]? {
        didSet {
            if let inputs_ = inputs {
                for (index, input) in inputs_.enumerated() {
                    input.addTarget(self, atTargetIndex: UInt(index + 1))
                    input.processImage()
                }
            }
        }
    }

    public init(fragmentShader: String, numberOfInputs: Int = 1) {
        super.init(fragmentShader: fragmentShader, numberOfInputs: UInt(numberOfInputs))
        ({strength = 1.0})()
    }
}

Ways to use:
public class HudsonFilter: CustomFilter {

    public override init(fragmentShader: String, numberOfInputs: Int) {
        super.init(fragmentShader: fragmentShader, numberOfInputs: numberOfInputs)

        ({strength = 1.0})()
        ({ inputs = [PictureInput(imageName:"hudsonbackground.png"), PictureInput(imageName:"overlaymap.png"), PictureInput(imageName:"hudsonmap.png")] })()
    }

    convenience init() {
        self.init(fragmentShader: hudsonFragmentShader, numberOfInputs: 4)
    }
}

A new Hudson filter object:
let filter = HudsonFilter()

@henvyluk - The shader files are not "useless", they are used to generate the filters that are built into the framework. A script called ShaderConverter.sh compiles them into string constants that are used within the processing operations.
They're good templates to use to build your own custom shaders, which you can load via your own BasicOperation (or the like) instances.

I'll answer the remainder on Stack Overflow: https://stackoverflow.com/questions/44871171/implementing-custom-filters-using-gpuimage-swift-library , since this isn't really a bug or enhancement request.

@BradLarson  I am sorry that I have tried to use this method, but did not reasonably solve. Well, it is not enough depth. I use the ps Action to achieve.


Why there are 4 dashes in comments instead of words?

I was wondered to see ---- instead some words in code's documentation. The dashes make it harder to understand the comments.
For example:
https://github.com/microsoft/referencesource/blob/e0bf122d0e52a42688b92bb4be2cfd66ca3c2f07/mscorlib/system/threading/Tasks/Task.cs#L273

I guess Microsoft automatically censors swear words from public sources and "race" somehow ended up in the blacklist.

Duplicate of #68

Closed as duplicate of #68.


Access blocked

same problem here

Same. Crazy thing is that it was working just fine a few days ago.  Been running cloud servers on runpod - was working, now is not. I've tried again and again.

same error

same error

same

same error.

Same error.

Same here

Hi, if anyone has managed to pass this issue please share the tips :)

same problem

Fixed in https://github.com/prasmussen/gdrive/issues/645

If you've logged in before, you can copy the token file ${HOME}/.gdrive/token_v2.json to the same place in your new machine, then you can skip the login process.

not an automated solution, but if you want to download a file from google drive to a remote server, you can launch a web browser on that server
mkdir ~/downloads/
chmod 777 ~/downloads/
# LAUNCH_URL should be in quotes and set to the link of your google drive file
# VNC_PW is set to password
sudo docker run --rm -it --shm-size=512m -p 6901:6901 -v ~/downloads/:/home/kasm-user/Downloads -e VNC_PW=YOUR_PASSWORD -e LAUNCH_URL='LINK_TO_YOUR_FILE' kasmweb/chrome:1.11.0

then you can navigate to https://PUBLIC_IP_OF_YOUR_SERVER:6901 log in with kasm_user and your password and complete whatever captchas / questions google presents.
more information on how the end result looks is here https://hub.docker.com/r/kasmweb/chrome

Still the same problem today.

You can use this Repository. Don't forget to hit Star.

@Msameim181 first build it then ask for stars lol

I am also experiencing the same issue.

Seems this is due to this https://developers.google.com/identity/protocols/oauth2/resources/oob-migration
October 3, 2022 - the OOB flow is deprecated for OAuth clients created before February 28, 2022

I downloaded gdrive 2.1.1 for Windows from here - https://github.com/prasmussen/gdrive/releases and the problem still exists. After running gdrive about I get an "Authentication needed" notice with a URL, but when I go to the URL I get the Access blocked issue. Should I download from somewhere else? Or should I do anything locally?

I downloaded gdrive 2.1.1 for Windows from here - https://github.com/prasmussen/gdrive/releases and the problem still exists. After running gdrive about I get an "Authentication needed" notice with a URL, but when I go to the URL I get the Access blocked issue. Should I download from somewhere else? Or should I do anything locally?

This problem is because of changes happens in google policy I think. So owners of this package have to have some change on their package.

Guys, I think I am going to try using Dropbox API.

You can also try using rclone
It has support for Google Drive too along with many others.
I used to upload my backup to the Google Drive and it works like a charm

rclone worked for me 👍

same problem here

So this problem is caused by Google changing the OAuth method. It would just mean you cannot get the token, so if you have already got a token to your drive previously, and still have it stored, you can reuse it  as @knwng stated.
Someone else (either on this repo or on gshell's repo) also mentioned spawning a web server that loops back the request (which is apparently what's the new guidelines require), but they didn't provide how to do it, and I wasn't able to replicate it. I'll link to it later if I can find the comment.
Someone will just have to migrate the code and follow their new guidelines.
https://developers.google.com/identity/protocols/oauth2/resources/oob-migration
I want to do it, sadly I genuinely do not understand how it works after reading the source code myself.

Ok just reading through the issues list, I have tried it and #645 worked
Just follow the steps listed, but you need to install go as well as enable google drive API
Link to the fork is https://github.com/carstentrink/gdrive

You can try this method. It's working.
Generating an OAuth token

Go to OAuth 2.0 Playground https://developers.google.com/oauthplayground/
In the Select the Scope box, paste https://www.googleapis.com/auth/drive.readonly
Click Authorize APIs and then Exchange authorization code for tokens
Copy the Access token
Run in terminal

curl -H "Authorization: Bearer ACCESS_TOKEN" https://www.googleapis.com/drive/v3/files/FILE_ID?alt=media -o FILE_NAME
source

You can try this method. It's working.
Generating an OAuth token

Go to OAuth 2.0 Playground https://developers.google.com/oauthplayground/
In the Select the Scope box, paste https://www.googleapis.com/auth/drive.readonly
Click Authorize APIs and then Exchange authorization code for tokens
Copy the Access token
Run in terminal

curl -H "Authorization: Bearer ACCESS_TOKEN" https://www.googleapis.com/drive/v3/files/FILE_ID?alt=media -o FILE_NAME
source

But this is solution for download, what about upload ?

 I can't get into my domain.

FYI: Is this still happening.

I have the same issue 🤦♀

WTF

Use this: https://github.com/BugCode1/gdrive/releases/tag/2.1.2
its working for me.
verification_code:
just copy the verification code (****** - for example) from this url after redirecting
http://localhost:1/?state=state&code=******&scope=https://www.googleapis.com/auth/drive


Update default installed version

Hi,
Koa 2 has been released by some time now, could you please update the default installed koa-router version according to this package docs:

When koa 2 is released, 7.x will be tagged with latest and installed by default by NPM.

(Relates to #333)

I hope to see this resolved soon 😄

@alexmingoia or @tj do you want me to send a pr?

@alexmingoia who has publishing permissions?

@jbielick Sorry I thought I had added you but was mistaken. You should be able to publish on npm now. I went ahead and published 5.4.1 and 7.1.1

Thanks, @alexmingoia !

@alexmingoia and @jbielick so shall we close this issue now? Cheers

But Koa v2 was released and koa-router@latest still targets Koa v1. This doesn't resolve the original issue 😢

> npm dist-tag add<EMAIL_ADDRESS>latest
+latest<EMAIL_ADDRESS>> npm dist-tag ls
latest: 7.1.1
next: 7.1.1

should be good now. Do we need to rm the @next tag?

It is also needed to set master as the default branch in this repository

That I cannot do

Latest should be publishing correctly, for the default branch, we'll can track on #352


Calling an action fails when sending a Content-Type header

When calling an action and sending a Content-Type header the server always responds with 406-Not Acceptable. If the Content-Type header is removed then the actions work fine.

@sduplooy1 I've had on my list for a while to push out some updates from work we've done on this internally - might include a fix for this, I need to check. (I recall seeing this issue at some point too.)
From your perspective is this an annoyance or a major issue?

I don't think that's the proper fix. When I've seen this before the issue has been setting the Content-Type header when there isn't actually any content (e.g., for a GET request) - can you confirm that's the case?

Yes correct.
A test scenario is in the phonebook sample, when you do a GET to the
$find-by-phone action:
GET:
http://localhost:52773/csp/ISC-REST/phonebook-sample/api/contact/$find-by-phone?phoneFragment=962
If Content-Type is set it fails.
On Thu, Feb 8, 2024 at 3:40 PM Tim Leavitt @.***> wrote:

I don't think that's the proper fix. When I've seen this before the issue
has been setting the Content-Type header when there isn't actually any
content (e.g., for a GET request) - can you confirm that's the case?
—
Reply to this email directly, view it on GitHub
https://github.com/intersystems/isc-rest/issues/26#issuecomment-1934146834,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA54QHFMKC47KEROEM5PRTLYSTIVTAVCNFSM6AAAAABC7XHS46VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZUGE2DMOBTGQ
.
You are receiving this because you were mentioned.Message ID:
@.***>


expand tilde for home directory

Otherwise vim would try to open literally ~/.backup/yankring_... and fail with error:
Error detected while processing function 103_YRInit..103_YRHistoryRead..103_YRHistorySave:
line   13:
E482: Can't create file ~/.backup/yankring_history_v2.txt
YRHistorySave: Unable to save yankring history file: ~/.backup/yankring_history_v2.txt

interesting.... I don't have this error. wander why... ;)


Could a useStardustThemeContext Hook be added?

Feature Request
Currently ThemeContext is exported from here:
https://github.com/stardust-ui/react/blob/3b21ea894ac8e16e15d559a888f796be9cd52718/docs/src/context/ThemeContext.tsx#L31
Could this also export a useStardustThemeContext Hook? This way this be composed into Functional Components more easily.
Problem description
Currently it's a little cumbersome to compose in the current Stardust Theme Context in a Functional Component.
Proposed solution
Add an exported Hook called useStardustThemeContext that looks similar to:
export const ThemeContext = React.createContext<ThemeContextData>(themeContextDefaults)

export const useStardustThemeContext = () => React.useContext<ThemeContextData>(ThemeContext);

The useStyles hook should be used to get computed styles. We do not want to expose internals (theme object), just the computed values


chore: setup the project

add dojo as cairo dependency
setup correct folders

use dojo-starter once kari fixes it

done


Debugging slow parsing

Checklist

[ ] I've read and followed the Contribution Guidelines
[ ] step.yml and README.md is updated with the changes (if needed)

Version

Requires a MAJOR/MINOR/PATCH version update
Context

Changes

Investigation details

Decisions

Closing this in favor of #292, that PR contains the go-xcode lib changes without the other debug logging changes.


Skeleton of plugin interface

:tada: This PR is included in version 1.1.2 :tada:
Your semantic-release bot :package::rocket:


Fix : Differentiate recent and all in study panel

Context
This PR tries to solve : https://github.com/OHIF/Viewers/issues/3714
It were remaining a partial implementation in which the "recent" tab and "all" tab was displaying the same content.
Changes & Results
For Primary tab => No changes
For Recent => It now list studies prior to the oldest primary study with a limit time of 1y
For All => It now list all patient's studies without restrictions (including primary studies)
Testing
This change does not affect Basic mode as this mode uses PanelStudyBrowserTracking, while this PR updated PanelStudyBrowser
@sedghi is there any plane to make this two sharing at least a common code base ?  There a lot of duplicated code between both.
Checklist
PR
Code

[] My code has been well-documented (function documentation, inline comments,
etc.)

Public Documentation Updates

[] The documentation page has been updated as necessary for any public API
additions or removals.

Tested Environment
Chrome Latest

I'm gonna open another PR for this

closing in favor of https://github.com/OHIF/Viewers/pull/4242


getting frames from a video using ffmpeg through node

Using this command "ffmpeg -i inputfile.avi -f image2 image-%07d.png" i'm able to get frames for video, but how to implement the same in node.js

Just use ffmpeg('inputfile.avi').save('image-%07d.png');. ffmpeg will guess the rest, I think, or maybe you will have to add .toFormat('image2'). Don't forget to add event handlers (esp. for errors).

thankyou, i will apply this
Thanks and Regards,
Vamsi Perivilli.
On Wed, Dec 9, 2015 at 3:22 PM, Nicolas Joyard<EMAIL_ADDRESS>wrote:

Just use ffmpeg('inputfile.avi').save('image-%07d.png');. ffmpeg will
guess the rest, I think, or maybe you will have to add .toFormat('image2').
Don't forget to add event handlers (esp. for errors).
—
Reply to this email directly or view it on GitHub
https://github.com/fluent-ffmpeg/node-fluent-ffmpeg/issues/480#issuecomment-163166183
.


fix: #5831 HOWTO Dutch. flag emoji code en > us

What does this PR do?
Improve repo
For resources
Description
Hotfix to resolve https://github.com/EbookFoundation/free-programming-books/commit/d775cb34fbf5eadbdaa465bf46510cc4974151f8#commitcomment-59537594
:en: emoji flag markup doesn't work. Use :us: :us: or :gb: :gb:
Source PR: #5831.
https://github.com/EbookFoundation/free-programming-books/commit/d775cb34fbf5eadbdaa465bf46510cc4974151f8#diff-b35eb40c148657d4ea1361c44496e11c5625f484a4c6040c022770e88cbcfdccR8-R10
Checklist:

[x] Read our contributing guidelines
[x] Search for duplicates. #5831
[x] Add needed indications (PDF, access notes, under construction)

Follow-up

Check the status of GitHub Actions and resolve any reported warnings!

:warning: Locked by #6614. Merge after it

added to #6614


[bug]: patchmatch failed to load or compile (/usr/lib64/libtiff.so.6: undefined symbol)

Is there an existing issue for this problem?

[X] I have searched the existing issues

Operating system
Linux
GPU vendor
AMD (ROCm)
GPU model
RX 7800 XT
GPU VRAM
16GB
Version number
4.2.0
Browser
LibreWolf 125.0.3-1 (Firefox-based Browser)
Python dependencies
{
  "accelerate": "0.29.2",
  "compel": "2.0.2",
  "cuda": null,
  "diffusers": "0.27.2",
  "numpy": "1.26.4",
  "opencv": "<IP_ADDRESS>",
  "onnx": "1.15.0",
  "pillow": "10.3.0",
  "python": "3.11.9",
  "torch": "2.2.2+rocm5.6",
  "torchvision": "0.17.2+rocm5.6",
  "transformers": "4.39.3",
  "xformers": null
}

What happened
For some reason I get a patchmatch error in my terminal about /usr/lib64/libtiff.so.6 having an undefined symbol preventing it from loading:
Generate images with a browser-based interface
>> patchmatch.patch_match: ERROR - patchmatch failed to load or compile (/usr/lib64/libtiff.so.6: undefined symbol: jpeg12_read_raw_data, version LIBJPEG_6.2).
>> patchmatch.patch_match: INFO - Refer to https://invoke-ai.github.io/InvokeAI/installation/060_INSTALL_PATCHMATCH/ for installation instructions.
[2024-05-12 13:08:57,141]::[InvokeAI]::INFO --> Patchmatch not loaded (nonfatal)
[2024-05-12 13:09:04,040]::[InvokeAI]::INFO --> Using torch device: AMD Radeon RX 7800 XT
[2024-05-12 13:09:04,256]::[InvokeAI]::INFO --> cuDNN version: 2020000
[2024-05-12 13:09:04,508]::[uvicorn.error]::INFO --> Started server process [5848]
[2024-05-12 13:09:04,508]::[uvicorn.error]::INFO --> Waiting for application startup.
[2024-05-12 13:09:04,508]::[InvokeAI]::INFO --> InvokeAI version 4.2.0
[2024-05-12 13:09:04,508]::[InvokeAI]::INFO --> Root directory = /mnt/chonker/InvokeAI/InstallDir
[2024-05-12 13:09:04,528]::[InvokeAI]::INFO --> Initializing database at /mnt/chonker/InvokeAI/InstallDir/databases/invokeai.db
[2024-05-12 13:09:06,138]::[uvicorn.error]::INFO --> Application startup complete.
[2024-05-12 13:09:06,138]::[uvicorn.error]::INFO --> Uvicorn running on http://<IP_ADDRESS>:9090 (Press CTRL+C to quit)

What you expected to happen
I expected no patchmatch error to occur.
How to reproduce the problem

Install media-libs/tiff with jpeg image support
Install opencv
Symlink /usr/lib/pkgconfig/opencv4.pc to /usr/lib/pkgconfig/opencv.pc
Install and set openblas as the blas implementation
Install InvokeAI
Launch the InvokeAI start script

Additional context
OS: Gentoo Linux (LLVM17-based)
System libtiff version: 4.6.0
Discord username
No response

having same issue trying to resolve the "patchmatch.patch_match: ERROR - patchmatch failed to load or compile (Command 'make clean && make' returned non-zero exit status 2.)." , followed the instructions on tuto and know i have this error
Actually patchmatch never worked for me, linux fedora, last invokeai version , rtx 3090


Plugins API: ability to add query methods

For now, css selectors is the only abailable method of querying elements. We need plugin API which allows users to add custom query methods.

@SevInf do you think implementing this functionality as plugin is the best way?
I was thinking, maybe it should be configurable by option in .gemini.yml (queryType, for example) and maybe (not sure if needed), it could be made possible to override it specifically in methods?
Possible selenium queries in wd are:
elementByClassName(value, cb) -> cb(err, element)
elementByCssSelector(value, cb) -> cb(err, element)
elementById(value, cb) -> cb(err, element)
elementByName(value, cb) -> cb(err, element)
elementByLinkText(value, cb) -> cb(err, element)
elementByPartialLinkText(value, cb) -> cb(err, element)
elementByTagName(value, cb) -> cb(err, element)
elementByXPath(value, cb) -> cb(err, element)
elementByCss(value, cb) -> cb(err, element)
elementByIosUIAutomation(value, cb) -> cb(err, element)
elementByAndroidUIAutomator(value, cb) -> cb(err, element)
elementByAccessibilityId(value, cb) -> cb(err, element)

and obviously lots of them are interchangeable by css/sizzle (className, CssSelector, Id…)
Maybe it's just more useful to implement those step-by-step in a core gemini, and made configurable, with default as css selectors / sizzle (for backward compatibility and as most capable one) ?


add rlp decode subcommand

PR description
Fixed Issue(s)

Adds decode subcommand to rlp based on this Issue
Example
besu rlp decode --from=<input-file> --to=<output-file> --type=<IBFT_EXTRA_DATA|QBFT_EXTRA_DATA>

Thanks for sending a pull request! Have you done the following?

[ ] Checked out our contribution guidelines?
[x] Considered documentation and added the doc-change-required label to this PR if updates are required.
[x] Considered the changelog and included an update if required.
[ ] For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests

Locally, you can run these tests to catch failures early:

[ ] unit tests: ./gradlew build
[ ] acceptance tests: ./gradlew acceptanceTest
[ ] integration tests: ./gradlew integrationTest
[ ] reference tests: ./gradlew ethereum:referenceTests:referenceTests

sounds good, will add tests around it and update the PR. Thank you so much for the feedback

@macfarla Updated the PR with tests. Would be grateful if you could give it a look. The tests might need approval to run and build. Thank you

looks good @Brindrajsinh-Chauhan - thanks for your contribution!
May be helpful for docs - some local testing I did that shows more usage options
➜  b2 git:(rlp-decode) ✗ echo 0xf853a00000000000000000000000000000000000000000000000000000000000000000ea94be068f726a13c8d46c44be6ce9d275600e1735a4945ff6f4b66a46a2b2310a6f3a93aaddc0d9a1c193808400000000c0 > a.txt
besu rlp decode --from=a.txt --to=b.txt --type=IBFT_EXTRA_DATA
➜  b2 git:(rlp-decode) ✗ less b.txt
➜  b2 git:(rlp-decode) ✗ besu rlp decode --from=a.txt --type=IBFT_EXTRA_DATA
[0xbe068f726a13c8d46c44be6ce9d275600e1735a4, 0x5ff6f4b66a46a2b2310a6f3a93aaddc0d9a1c193]
➜  b2 git:(rlp-decode) ✗ cat a.txt | besu rlp decode
[0xbe068f726a13c8d46c44be6ce9d275600e1735a4, 0x5ff6f4b66a46a2b2310a6f3a93aaddc0d9a1c193]
➜  b2 git:(rlp-decode) ✗ cat a.txt | besu rlp decode --type=IBFT_EXTRA_DATA
[0xbe068f726a13c8d46c44be6ce9d275600e1735a4, 0x5ff6f4b66a46a2b2310a6f3a93aaddc0d9a1c193]
➜  b2 git:(rlp-decode) ✗ cat a.txt | besu rlp decode --type=QBFT_EXTRA_DATA
org.hyperledger.besu.ethereum.rlp.RLPException: Expected current item to be a list, but it is: SHORT_ELEMENT (at bytes 78-79: ...d9a1c193[80]84000000...)

Thank you @macfarla for the review. Should I open a PR for docs update of is the process something different.

Thank you @macfarla for the review. Should I open a PR for docs update of is the process something different.

You are welcome to open a PR in the docs repo - some of the linting is a bit fiddly but docs team can definitely help with that to get it over the line! That would be much appreciated!
Otherwise it goes on the (quite long!) backlog for the docs team (based on the label).
Either way, it will expedite things if we can at least draft the content - usage examples and description.

@macfarla could you please approve the workflow run again, had to merge with main for it to automerge


refactor(test): rename e2e_test/source to source_legacy

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?

Checklist

[ ] I have written necessary rustdoc comments
[ ] I have added necessary unit tests and integration tests
[ ] I have added test labels as necessary. See details.
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[ ] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

#18890  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @xxchan and the rest of your teammates on  Graphite

So "legacy" is referring to shared/non-shared, instead of inline/non-inline tests? Can you add some description in PR body?

Good question, thanks.
No, "legacy" means non-inline style, not related with shared. (Although I gain the motivation from working on shared source. I need to modify some tests, but I'm reluctant to touch legacy tests, and want to modernize them first.)
I found original PR https://github.com/risingwavelabs/risingwave/pull/18870 not trivial to review the diff, so split it into steps.


[cmake] don't find_library for libm to avoid absolute path in OpenEXRTargets.cmake

As is, if you do a
cmake -Bbuild -DBUILD_SHARED_LIBS=OFF -GNinja --fresh && DESTDIR=install ninja -C build install

then build/install/usr/local/lib/cmake/OpenEXR/OpenEXRTargets.cmake will reference the absolute path to libm, which is problematic for my usecase as I'm compiling against a sysroot then copying it and building against it elsewhere.
This fixes that problem by just linking to m instead of the path

Thanks for the fix and the contribution. The ASWF policy is to require signed commits, could you possibly re-commit this with a -s?  That will add the "signed-off-by" line. The best thing to do is to overwrite this commit with a force push, something like "git reset HEAD~1; git commit -s -m ; git push -f".
Alternatively, I'm happy to submit a separate PR with the change, to save you the hassle if you prefer.  Thanks!

You don't need the git reset part. The following will do:
git commit --amend -s

(and then the force push)
This avoids losing any commit message you had originally.

@russelltg, this slipped through the cracks but we're about to make a release and would like to include this. Could you take a minute and amend the commit with the signoff? Or let me know and I'll submit a duplicate PR myself. Thanks!

@russelltg, one more ping before the upcoming release, can you kindly sign the commit? Thanks!

Yep, sorry about the delay. Thanks for the pings :)


KWM does not account for dock position

Kwm 4.0.4
I prefer the dock to be on the left of my screen, but Kwm assumes the dock is on the bottom and leaves some padding for it. It does not detect where the dock is and assumes it is on the bottom so the window is resized and overlaps under the dock. This results in me not able to click the X button easily:

See there is no padding on the left side so the window is not accounting for that size difference with the dock there.

padding is configurable. see http://koekeishiya.github.io/kwm/kwmc#global-defaults

Oh! That is awesome, I did not realize
So I guess this is what I should edit:
Set default padding
kwmc config padding {padding}
{padding}: top bottom left right
I tried
./kwmc config padding left
But it results in an error
Unknown config padding top value 'left'

you actually need to sub in all four values. like kwmc config padding 0 0 20 0

Thank you so much It worked


Update 9.5.0 release notes with known issue

Hi Jonathan, @jveldhuizen
Since this is a known issue in 9.5 should this be published immediately?
I'll need to take out the fix information and add it later.
Do you have a zendesk ticket or JIRA KNOWN_PUBLIC_ISSUE for this?
Let me know what you want to do with this - I'll mark it for 9.6 for now until I know what the best solution is.
Mark

Hi @MarkvanMents , preferable, we add this as soon as possible to the 9.5 release notes. The Jira issue is RTI-308, the currently known ticket numbers 129493, 129605 and 129902.

OK - I"ll remove the fix information and publish now.

Thanks!


Row component has mx property set to -32 by default

Title says everything :)
Here is the sandbox with reproduced bug and few examples

Setting mx to 0 solves the problem, other fields not.
Also, wrapping with Provider and theme.space = [0] solve problem a little. Giving mx = -3 which is still weird.
Provider also seems to have default theme, such as fonts etc. This must be described somewhere so people know defaults.

@nfcampos My guess is that Row is meant to be used with Column which has as postive padding px=3. Which is a common pattern to achieve a grid with gutters and evenly sized columns.

Yeah, sorry the docs are sort of light around the components at the moment. As @ferdinandsalis said, <Row /> is meant to be used with <Column /> as a simple, no configuration sort of component. They both extend <Flex /> and <Box /> respectively, which are more generic and flexible.
The <Provider /> is documented here: https://github.com/jxnblk/rebass#provider-

@ferdinandsalis I think you've mentioned me by mistake?

@nfcampos sorry, yes! Happens quite easily …

@ferdinandsalis no problem!


Update branding

This organization is referred as:

"The Nibbstack group" https://twitter.com/0xcert/status/1463467336493678602
"nibbstack" https://github.com/nibbstack
"Nibbstack" https://nibbstack.com

Please pick one a representation for the brand and update all to be consistent.

Done :)


fix(a11y): Pass colour contrast audit

This PR addresses colour contrast accessibility issues with the current docs theme in light mode.
It adds:

Darker text accent colour (links etc.).
I chose a darker, but also marginally redder hue as that helps with contrast without needing to go too dark and lose vibrancy. I think it still vibes with the accent orange, but feedback welcome.

Slightly darker code block background colour.

Tweaked two text colours used for syntax highlighting to improve contrast

Tested using in-browser colour contrast tools in Chrome and Firefox.
I have more cosmetic improvements I’d like to propose, but I’ll add those in a separate PR to make this (hopefully) quick to merge and address the accessibility issues.

Hey @delucis ! Can you confirm that you can view the deploy preview here? I don't know why, but I'm getting netlify.app can't handle the request. (Not sure if it's a Netlify problem or a me problem.) Sure the ONE pr where I really want to SEE the difference...  ;)

While Netlify is down, here’s a side-by-side of one page to give an idea of what’s going on:

Before
After

LGTM! Thanks for the audit @delucis!


Update default.nix

https://bun.sh/
0.6.7 is out for bun. This package is a bit dated. I would like to update it so its usable for people who want to work with or contribute to bun
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[x] x86_64-darwin
[ ] aarch64-darwin

[x] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

@DAlperin - this package is a bit outdated - I'm new to contributing in nix, and pretty new to open source in general. Could you guide me how to get this across the line? Reaching out b/c I see you're marked as a maintainer here. I would love to help out if needed

bun 0.6.7 is available on the unstable branches

Thank you so much for this comment. I couldn't figure out for the life of me why so many packages seemed out of date.


support Exception serialize/deserialize?

I want serialize/deserialize Exception and derived types.
The initial idea is implement ExceptionFormatter, like below.
public sealed class ExceptionFormatter<T> : IFormatter<T> where T : Exception {
   public void Serialize(ref byte[] buffer, ref int offset, T value) {
      var binaryFormatter = new BinaryFormatter();
      ... append byte[] generated by binaryFormatter to the buffer...
   }
   public void Deserialize(byte[] buffer, ref int offset, ref T value) {
      var binaryFormatter = new BinaryFormatter();
      ... use the binaryFormatter to read the buffer to deserialize value...
   }
}

BinaryFormatter seems too fat.
Did I miss the reference Exception?
Any suggestions?
thanks.

Should be no problem to make this work somehow.
What exactly are you looking for though?
Are you looking for a "specialized" serialization that only includes stuff you care about?
I suppose you obviously want the actual Type of the exception and its Message, and the InnerException is also essential...
But what about other things? StackTrace is very important in most scenarios, so that should be included, right? But maybe you don't need that?
Or are you simply looking for a perfect copy? That should be totally doable as well...
Did you try just giving the exception instance directly to Ceras? Are there some problems with doing that? Personally I have not yet tried to serialize exceptions, so I'm not sure if it works.
But even if there are some issues, I'm certain they can be fixed more or less easily :)
Let me know what you think.
I will have more time in a few hours; then I'll try to serialize some exceptions and see what happens.

I am looking for something to replace my WCF.
This includes some serializer instead of DataContractSerializer.
DataContractSerializer is nice but not friendly.
For exception, I hope serializer can keep detailed information so that the caller can re-throw the exception, so there must has Type and a Message.
It would be better if it could include StackTrace and InnerException, which can help developers.
It would be better if Ceras could build support for exceptions.
But if the inside is implemented with BinaryFormatter, I think it will affect the cleanliness of Ceras.
But at the moment I can't think of a way to practice without using BinaryFormatter.
What do you think?

I have considered it too much.
Very simple to solve with the following.
new CerasSerializer(new SerializerConfig { DefaultTargets = TargetMember.All })
Sorry for disturbing you. 😄

Good that it works for you.
If there is something still missing or not working, be sure to let me know :D


Ollama reports an error when running the AI model using GPU

What is the issue?
When I use the command :
’docker run --gpus all -d -v f:/ai/ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama‘
the following error will occur,
“docker:Error response from daemon:failed to create task for container:failed to create shim task:OCIruntime create failed:runc create failed:unable to start container process:error during container init:error runninghook #0:error running hook:exit status 1,stdout:stderr:Auto-detectedmode as 'legacy nvidia-container-cli:initialization error:load library failed:libnvidia-ml.so.1:cannot open shared object file:no such file or directory:unknown.”How should we handle this issue?
OS
No response
GPU
Nvidia
CPU
Intel
Ollama version
No response

This error appears to be coming from Docker or the Nvidia container runtime.  It looks like it happens before ollama starts running.
Please make sure you have GPU support configured and working in with Docker.  https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
A simple test to verify things are working properly at the Docker + Nvidia level without Ollama involved is:
% docker run --gpus all ubuntu nvidia-smi
Tue May 21 23:54:36 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.105.01   Driver Version: 515.105.01   CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0 Off |                  N/A |
| 35%   31C    P8    N/A /  19W |      1MiB /  4096MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+


Spheres of influence update

Updated to work with VIC3 version 1.7
Transfer trade power now creates a trade agreement like steer trade does.
Decentralized World updated to fit new states in Eastern Europe, Siberia, and the Middle East
Country mappings updated for new nations added to VIC3

In Europa Expanded Kazan and Tatarstan are separate tags, so that could be problematic...
Also, I would change the IUS mapping to Volga-Ural, since Idel-Ural is just the tatar name for it


Improvement In HexCodeGenerator.

Refer this attachment for what I have changed.

My Update changes

I have successfully resolved the issue no #301
please check and assign me.

Attach screenshots of improvements you have made
Attach both previous and present project screenshots

Attach screenshots of improvements you have made Attach both previous and present project screenshots

Allow me a minute I'm just doing.

I have successfully resolved the issue no #301 please check and assign me.

@TusharKesarwani by mistaken I have closed it,
Please have a look done changes.

@TusharKesarwani please review and merge it.

Changes are not required


Change issue templates for forms. Disabled empty issue template option.

Pre-submit checklist:

Branch

[ ] ~Tests are provided (if possible)~
[x] Commit sequence broadly makes sense
[x] Key commits have useful messages
[ ] ~Relevant tickets are mentioned in commit messages~
[x] Formatting, materialized Nix files, PNG optimization, etc. are updated

PR

[x] Self-reviewed the diff
[x] Useful pull request description
[x] Reviewer requested

This is amazing! Instead of renaming the other file just feel free to delete it.
So excited for this :)
:heart:
@silky Resolved all comments, please re-review.


Loss value on training and validation set not accurate

Hi, I notice that value of loss function at the end of training model isn't same as manually calculated error on same dataset (after model is learned).
Then I created callback on_epoch_end where I measure the error at the and of every epoch. The results are never the same as in training log. Loss function is MAE (mean absolute error). Here is the example:

Epoch
loss MAE
manually MAE

22
0.11139457036931492
0.09953024542751052

23
0.09754104633130029
0.08839598977022033

24
0.09538907789202652
0.08944885550643963

For calculating MAE manually I even tried to use keras.losses.mean_absolute_error but again
values are not the same. Manually calculated value are always lower than value of loss function.

I have also had issues with the val_loss. Eventhough my val_accuracy remains at 0.88, the val_loss, which uses log loss, is extremily low, almost zero. The loss on the training data is 0.28 with 0.88 accuracy, which seems closer to reality.

It seems that my issue goes away when downgrading keras to 2.25

I'm also having some val_loss related issue. In particular, I have a multi-loss model and the weighted validation loss of all the outputs don't add up correctly in val_loss. But loss (training loss) is ok.


Example app on how to get things running in Rails

I'm not much of a JS hero, so all the build tools: Webpack, Postcss etc feel like one big magic show to me. Is there anyone around here that has more experience with these tools that could create an example app with the bare configuration that is required for Ruby on Rails?
In return I'll make sure that the steps to set it up are added to the docs :-)

Laravel Mix makes webpack a no-brainer. It is not coupled to PHP or Laravel at all, you can use it in any project.
Repo: https://github.com/JeffreyWay/laravel-mix
Installation: https://github.com/JeffreyWay/laravel-mix/blob/master/docs/installation.md

@jvanbaarsen depending on which version of Rails (4.2+) you're using, the Webpacker gem makes this pretty straightforward.
Please note: I haven't used this in production, and I only fired up a Rails app to start playing with Tailwind. Also, I'm also not much of a JS hero either. 😎
This is a very basic process I followed and was able to get up and running pretty quick. Someone may have a more efficient way to do this. If I had more time today, I'd work on making this part of the Webpack build process.

Add Webpacker to your Gemfile.

gem 'webpacker', '~> 3.0'

Install the Gem

bundle install

Install (and configure) Webpacker

bundle exec rake webpacker:install
Webpacker will add the required files for Webpack, as well as create the app/javascript directory.

Create a tailwind.js file. I used:

./node_modules/.bin/tailwind init tailwind.js

Create a file inside app/javascript/src named styles.css

app/javascript/src/styles.css
Paste in the contents of this file.

In app/views/layouts/application.html.erb' add this in the ` (or wherever you want)

<%= javascript_pack_tag 'application' %>
<%= stylesheet_pack_tag 'application' %>

Now, hop back in a terminal and run:

./node_modules/.bin/tailwind build app/javascript/src/styles.css -o app/javascript/src/application.css
This builds the CSS for the application.

Inside app/javascript/packs/application.js add:

import "src/application.css"

Fire up your servers:

In one terminal window: ./bin/webpack-dev-server' In another: rails s`

Bask in Webpacker + Tailwind CSS 😎

I hope this helps. As I mentioned, this is just my first attempt at it. I'm glad to share with you (and hope you'll share with me!) as I experiment more with Rails + Tailwind.

@jasoncharnes Thanks! Yeah I have been trying to do this with Webpacker, but I wasn't sure where to place the code that is in the docs for Webpack (The tailwind docs that is).
I tried placing that in a new file called: postcss.config.js but nothing happend 🙈 .
I'll give this a shot and see if I can get it deployed to Heroku!

@jasoncharnes Great, I got this working locally. Only thing that I had to change was the following:
"yarn add tailwindcss --dev" -> When I ran that command the tailwindcss was not added to package.json. So I manually added it to that file to the dependency group, and ran: yarn install.
It does seem that the config js (The one in the root tailwind.js) is not picked up. I'm looking into ways of getting this to work :)

Nice glad you guys are figuring this out! Once you feel like you've got it ironed out pretty well, would you mind sharing a simple bare bones Rails repo with everything setup? Would be great for helping out other trying to integrate with Rails 👍

@adamwathan Yes! Collaborative learning 💪

@jasoncharnes / @adamwathan  It looks like I have the full app running now! I'll do another test with a brand new app, after that is working I'll post it as an example app.

@jasoncharnes I got it all working. For now I quickly created a Heroku app and an example repo without any readme:
Repo: https://github.com/jvanbaarsen/tailwindcss-rails-example
URL: https://tailwind-rails-example.herokuapp.com/
All the configuration has been done in this commit: https://github.com/jvanbaarsen/tailwindcss-rails-example/commit/9e00e2719570bfe06299fcf8499df63f98563c3d
The most import thing is to remove the .postcssrc file, and replace it with the postcss.config.js file.
Tomorrow I'll write some proper documentation with that readme, and if @adamwathan would accept a PR for it a small headline in the official docs on how to set it up and a link to that example repo (Or even better, maybe they can import it under the tailwindcss namespace and collect a bunch of example apps)

@jvanbaarsen Could you add your example to the awesome-tailwind list?

@m1guelpf Where do you want me to add it? Under Resources or Open Source?

@jvanbaarsen I'd put it under resources, but creating a quickstart subsection in the resources section would be the best thing.

@jvanbaarsen I got Tailwind to work with Rails (5.1.4) and Webpacker gem without replacing the .postcssrc file.
Here is my .postcssrc: https://github.com/jonaskay/tailwind-rails-blog/blob/master/.postcssrc.yml

@jonaskay Thanks for sharing :-) Yeah I got that figured out in the end as well! The syntax feels so weird :P

@jvanbaarsen Can you update the steps for a successful deploy to heroku? I've had no issues running tailwindcss locally but fail to deploy via heroku or aws.

@DawgOnKing I'd be happy to help.  I'm successfully deploying Rails 5.2/Edge apps using Tailwind to Heroku right now.  What version of Rails are you using and are you using Webpacker to manage everything?

@joeybeninghove Greatly appreciated! I'm running "locally" on Cloud9 via Rails 5.2 and webpacker gem.

@DawgOnKing Oh ok, well I'm only working with Heroku at the moment, not Cloud9 or AWS directly.  I'll still try to push up a full repo later if I have a chance, but for now, here is a quick Gist showing the relevant files that I've set up to work: https://gist.github.com/joeybeninghove/882ca5a2ea2917ffca087609a7e4c181
This a pretty vanilla set up honestly, as Heroku takes care of automatically running the webpack compile during deployment.

@joeybeninghove - Thanks again for your help. At quick glance, my setup is exactly the same as yours. I will create another app and post specific errors I'm getting from heroku during compile (which is where it's failing).

@DawgOnKing Did you setup the buildpacks for Heroku?

@jvanbaarsen - short answer, no. I attempted to via console but attempts have been unsuccessful. Did you use CLI?

Here's a breakdown for getting Tailwind up and running on Cloud9 machines. This isn't the AWS setup although I have that running as well. I just need additional steps for deploying via Heroku and AWS CodeDeploy which I will work on tomorrow.
Thanks again for assistance with deployment!
https://github.com/DawgOnKing/cloud9-tailwindcss

@DawgOnKing I believe I set the Buildpacks in the Heroku interface itself. It is very important that the Nodejs buildpack comes before the ruby buildpack. Otherwise it won't work.

@DawgOnKing @jvanbaarsen That’s true about the buildpacks, but in my recent deployments to Heroku, they just figured that out for me automatically.  I didn’t have to do anything special to manually configure the buildpacks.  I’m not sure how or what about the Rails apps I’m deploying that makes that the case, but I’m happy it’s working like that for me.  :)
Sent with GitHawk

@jvanbaarsen I set both of those via the CLI and they are loading in order according. Here's the output of running heroku buildpacks via CLI:
=== stormy-sands-xxxxx Buildpack URLs

heroku/nodejs
heroku/ruby

When I attempt to push project to Heroku, I'm getting errors/failures during compile. But warnings as well when building dependencies:
-----> Building dependencies
remote:        [3/4] Linking dependencies...
remote:        warning "@rails/webpacker ><EMAIL_ADDRESS>has unmet peer dependency<EMAIL_ADDRESS>remote:        warning " ><EMAIL_ADDRESS>has unmet peer dependency<EMAIL_ADDRESS>|| ^3.0.0".
remote:        warning "webpack-dev-server ><EMAIL_ADDRESS>has unmet peer dependency<EMAIL_ADDRESS>|| ^2.0.0 || ^3.0.0".
remote:        [4/4] Building fresh packages...
remote:        Done in 21.69s.
AND FINALLY THE FAILURES DURING COMPILE:
[2] ./node_modules/css-loader??ref--1-2!./node_modules/postcss-loader/lib??ref--1-3!./app/javascript/src/application.css 799 bytes [built] [failed] [1 error]
ERROR in ./node_modules/css-loader??ref--1-2!./node_modules/postcss-loader/lib??ref--1-3!./app/javascript/src/application.css
remote:        Module build failed: Error: Loading PostCSS Plugin failed: Cannot find module 'autoprefixer'
Module build failed: ModuleBuildError: Module build failed: Error: Loading PostCSS Plugin failed: Cannot find module 'autoprefixer'
IN SUMMARY
Looks like these issues are being created by autoprefixer which is a devDependency. I'm not sure why it would be installed in production environment as it's my understanding that only dependencies should be.

@jvanbaarsen
What command are you using to generate tailwind in CLI?
Did you manually create and update contents of your postcss.config.js file?

@DawgOnKing I'm using Webpacker for this. See my example repo: https://github.com/jvanbaarsen/tailwindcss-rails-example

Finally solved by setting NODE ENV production variable via CLI. Thanks to all for assistance!

Note, the current version of Tailwind (v0.5.1) now requires your CSS file to look like this:
@tailwind preflight;

@tailwind components;  // new!

@tailwind utilities;

If you use the old config, most classes will be missing and you'll likely experience a prolonged spike in blood pressure.

I manage to put together a gem to easily install tailwindcss with Rails - https://github.com/IcaliaLabs/tailwindcss-rails - please let me know how it goes!

Thanks @jasoncharnes
I was losing tailwind configuration of my Rails 6 website and this fix it.
rails webpacker:info
Ruby: ruby 2.6.4p104 (2019-08-28 revision 67798) [x86_64-linux]
Rails: 6.0.0
Webpacker: 4.0.7
Node: v8.16.1
Yarn: 1.19.0
@rails/webpacker:
<EMAIL_ADDRESS>/var/www/html/MRCOM
└──<EMAIL_ADDRESS>Is bin/webpack present?: true
Is bin/webpack-dev-server present?: true
Is bin/yarn present?: true
Thanks @jasoncharnes again!

@coopeu:
Have you deployed this to production anywhere? I've been trying to get the assets:precompile job to work for over an hour now, and nothing. I've tried everything suggested here, and when RAILS_ENV=development bundle exec rails assets:precompile works fine, but when RAILS_ENV=production, I get an error:
ERROR in ./app/javascript/css/application.css
Module build failed (from ./node_modules/mini-css-extract-plugin/dist/loader.js):
ModuleBuildError: Module build failed (from ./node_modules/postcss-loader/src/index.js):
Error: Cannot find module 'tailwindcss'

# ...

If anyone here has a Rails 6 app with Tailwind that can successfully precompile assets on production, I would love to see your configuration. I cannot for the life of me find the issue.
FWIW:
$ rails webpacker:info
Ruby: ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-darwin17]
Rails: 6.0.0
Webpacker: 4.0.7
Node: v12.10.0
Yarn: 1.17.3

@rails/webpacker:
<EMAIL_ADDRESS>/Users/margheim/Projects/graphstimate
└──<EMAIL_ADDRESS>
Is bin/webpack present?: true
Is bin/webpack-dev-server present?: true
Is bin/yarn present?: true


feat: docker image

Overview

What you are trying to do: docker image for ape
Why Ape's current functionality is inadequate to address your goal: needs better docker image for compatibility with docker and easier setup and containerization

Specification
This should be a docker image which has all the apeworx plugins installed and ready to go.
Dependencies
This should be able to do on linux but more difficult on windows because an issue with dependency: https://github.com/ApeWorX/ape/issues/91

@sabotagebeats note that using the Docker image in windows doesn't mean the base needs to be windows, it can be a linux base

https://hub.docker.com/layers/apeworx/eth-ape-docker/0.1.0a22/

https://hub.docker.com/layers/apeworx/eth-ape-docker/0.1.0a22/

Awesome! Can the image just be ape though?

https://hub.docker.com/layers/apeworx/eth-ape-docker/0.1.0a22/

Awesome! Can the image just be ape though?

I've made it just be ape but i'm having an issue getting it to compile solidity.

(brownie) sabotage@sabotage:~/ape/token$ docker run --volume $HOME/.ape:/root/ape --volume $HOME/.vvm:/root/.vvm --volume $HOME/.solcx:/root/.solcx --volume $PWD:/root/project --workdir /root/project apeworx/ape ape compile
INFO: Compiling 'contracts/Token.sol'
INFO: Compiling 'contracts/SafeMath.sol'
Traceback (most recent call last):
  File "/usr/local/bin/ape", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/ape_compile/_cli.py", line 75, in cli
    contract_types = project.load_contracts(use_cache)
  File "/usr/local/lib/python3.8/dist-packages/ape/managers/project.py", line 117, in load_contracts
    contract_types.update(self.compilers.compile(list(needs_compiling)))
  File "/usr/local/lib/python3.8/dist-packages/ape/managers/compilers.py", line 49, in compile
    for contract_type in self.registered_compilers[extension].compile(paths_to_compile):
  File "/usr/local/lib/python3.8/dist-packages/ape_solidity/compiler.py", line 77, in compile
    for result in solcx.compile_source(
  File "/usr/local/lib/python3.8/dist-packages/solcx/main.py", line 112, in compile_source
    return _compile_combined_json(
  File "/usr/local/lib/python3.8/dist-packages/solcx/main.py", line 288, in _compile_combined_json
    stdoutdata, stderrdata, command, proc = wrapper.solc_wrapper(
  File "/usr/local/lib/python3.8/dist-packages/solcx/wrapper.py", line 158, in solc_wrapper
    raise SolcError(
solcx.exceptions.SolcError: An error occurred during execution
> command: `/root/.solcx/solc-v0.6.12 --combined-json abi,bin,bin-runtime,devdoc,userdoc -`
> return code: `1`
> stdout:

> stderr:
Error: Source "SafeMath.sol" not found: File not found.
 --> <stdin>:5:1:
  |
5 | import "./SafeMath.sol";
  | ^^^^^^^^^^^^^^^^^^^^^^^^

oh weird, so the solidity binary is not finding the local referenced file?

this issue has been resolved in new version of ape-solidity.

https://github.com/ApeWorX/ape/pull/165


defaultText fix for dropdownList method

Use the placeholder value as defaultText for dropdownList method

Where did you find placeholder attribute in select/dropdown?

Yes
Zelenin\yii\SemanticUI\widgets\ActiveField
You need to append ‘defaultText’ just like below.
Best Regards,
Mir Adnan
<EMAIL_ADDRESS>
On Feb 6, 2017, at 5:15 AM, Aleksandr Zelenin<EMAIL_ADDRESS>wrote:
Where did you find placeholder attribute in select/dropdown?

Where did you find a placeholder attribute in select/dropdown?

I found it in Zelenin\yii\SemanticUI\widgets\ActiveField dropDownList method.

A html select hasn't a placeholder. I will not add it. You shoud use a native api for it (see semantic ui docs).

Okay. How do I add defaultText then?

Ah, I see. Dropdown is not select and has not a default option.
Ok, use options like here https://github.com/zelenin/yii2-semantic-ui/blob/master/widgets/ActiveField.php#L91 and I will merge it
$defaultText = ArrayHelper::remove($options, 'defaultText', '');

Done

Thanks!

Awesome! Thanks


On some chains even though requests get fulfilled the script doesn't pick that up

When making requests on arbv4 sometimes it seems like the request didn't work because it can stall indefinetely waiting for fulfillment or even return a price of zero. When I go back and try to see if the request was fulfilled correctly though by querying for the requestId again a second time it is always correct.
Sometimes I would notice that the request would be fulfilled correctly and the ethers filtering picks it up but when querying past events with web3 it cannot find the ClientRequestFulfilled event yet for a very short time. So there seems to be issues with querying arbitrum events with both Ethers and Web3, with them disagreeing on if an event was emitted sometimes or other times ethers just misses the event completely.
So overall the issue seems to be with latency and emitted event querying on arbitrum, airnode works fine if you just wait a few seconds after fulfillment to query the request output value (sometimes it works instantly just fine though). This can end up in the airnode-starter test appearing as if it didn't work.
If you let me know a way to debug this better happy to work some more on it.

Thanks for the write-up. I think the main problem is that subscription patterns (.once() in https://github.com/api3dao/airnode-starter/blob/main/scripts/make-request.js#L39) always end up suffering from some edge cases, which is why Airnode is designed to poll for requests, that's just more robust.
We can simply replace the line linked above to poll every second for the event instead, but there are a ton of other subscriptions in the @api3/airnode-admin package (to get the requester index after creating the requester, etc.) so it's not a complete solution. I think airnode-starter appearing to fail sometimes without actually failing due to an issue with Arbitrum that they may end up solving anyway is not a serious problem. Let's keep this issue open and see if it happens on other chains/to others on Arbitrum.

This issue was originally opened for Arbitrum v4 but I updated the name in case we have the same problem in other chains.


ListView Component large dataset with external images bug

Hey there,
I noticed while implementing a list view with your library that you statically render the entire list inside of a view, instead of the React Native ListView component?
Is there a reason for this? You could just as easily wrap around the ListView component, which has multiple performance improvements, including initial list size. This would help when loading large lists with external image resources.
If you'd like I'd be more than willing to open a PR.
Mo

Hey @mobinni, check out https://github.com/react-native-community/react-native-elements#using-rn-listview-implemented-with-link-and-avatar, you're welcome to implement with either a map function or a ListView, it's really up to you! Let me know if I may have missed something or misunderstood your question. thanks!


Feat/component colors

Fixes CardList for simple card and image cards.
Fixes ImageCard
Add Subheading title color

:+1:


Region of interest

Adds arguments --region-of-interest as xmin ymin xmax ymax and --frame-id as integer. This is branched of the ionosphere branch so that should be merged first (mainly on account of github actions fixed there).

Once the tests have passed, I am going to merge this. This will provide better testing capabilities for raider and ionosphere.
Lots more tests have to be included within this repo (via pytest), but again, can't hold this up. Rather have things break so we can continue to debug the various pieces of the new processing workflow.