This is the continuation of : https://github.com/banzaicloud/bank-vaults/issues/1722

We have upgraded the vault image to 1.13.1 but we are not able to insert the flag "iss validation"
We are able to modify the existing key values but not able to add the new values. Could you please suggest.
vault kv put auth/kubernetes/config kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}<EMAIL_ADDRESS>disable_iss_validation=true
/ # vault kv get -format=json auth/kubernetes/config {   "request_id": "674646467-3a37-e36b-d262-9265c068c1af",   "lease_id": "",   "lease_duration": 0,   "renewable": false,   "data": {     "issuer": "",     "kubernetes_ca_cert": "-----BEGIN CERTIFICATE-----\nMIIDjDCCAnSgAwIBAgIUTS25upWdU6QcWkHW1fLg8zjGPxUwDQYJKoZIhvcNAQEL\nBQAwXjELMAkGA1UEBhMCVVMxDjAMBgNVBAgTBVRleGFzMQ8wDQYDVQQHEwZBdXN0\naW4xDzANBgNVBAoTBk9yYWNsZTEMMAoGA1UECxMDT0RYMQ8wDQYDVQQDEwZLOFMg\\\nY2xlMQwwCgYDVQQLEwNPRFgxDzANBgNVBAMTBks4UyBDQTCCASIwDQYJKoZIhvcN\nAQEBBQADggEPADCCAQoCggEBALnkheJYL8sbfbx2UEZGHhAdG2sDDrqY5vCCvn77\n8JvUsJzMRdY2mWM+JxepmOzmnlCDpJlaDPbcl+J1kSYTkyx8Tclz/0dgv7Nosv9e\nmC5AgwhIe7sO4xdOpL90/ixKmt/+bkjhzdkjhkjhsjkhdkjhzkdhkhzd\nQyWAT+zdlkhlkdzkljzkjjzlzjklzd\nRL/SA0S1mjl7umswpt1BdrjFyHDs+SrD/WK3m9+uOxvaerUT0v7rjnElvE7DXZEM\na2ffHbqdR0df7tXgXKSt9fLjSkpB5qkmVUldx+10Fc/M6QECAwEAAaNCMEAwDgYD\zdl/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFOzvai7hXPD0\n/8D+huX/5T1SIRMbMA0GCSqGSIb3DQEBCwUAA4IBAQALbI7xwG/yJZdv584xD9ya\nvbkOzfLd7QLR438VJllVwAm5tyhBmTxl8jct2UAlYqZjMlGqo+/9nHcQC4nz5hFJ\n9J/zdlkjzkkzdjkzljkzd+quBLcpeU/OExcdts76B1kHlLAiaX\nWm/Z9Tz6nxzNjj31LiMXYUCFBb8YxfbrZggNlV53+oKzcdsopI17Lj9auLg8ubeh\n5RS3presd1EJlF1PFQuyVDYIy/zdlkjlkzjzjzkljzlkd\n0OAe6jZeiEIyQzNfTWOXkFqfGGQRntdKv8/7q40EHrErr/9K58lcCaebISTU6LDe\n-----END CERTIFICATE-----\n",     "kubernetes_host": "https://<IP_ADDRESS>",     "pem_keys": []   },   "warnings": null }
Kuberentes cluster version: v1.21.5
###################################################################

Bugs should be filed for issues encountered whilst operating bank-vaults.
You should first attempt to resolve your issues through the community support
channels, e.g. Slack, in order to rule out individual configuration errors.
Please provide as much detail as possible.

Describe the bug:
A clear and concise description of what the bug is.
Expected behaviour:
A concise description of what you expected to happen.
Steps to reproduce the bug:
Steps to reproduce the bug should be clear and easily reproducible to help people
gain an understanding of the problem.
Additional context:
Add any other context about the problem here.
Environment details:

Kubernetes version (e.g. v1.10.2):
Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc):
bank-vaults version (e.g. 0.4.17):
Install method (e.g. helm or static manifests):
Logs from the misbehaving component (and any other relevant logs):
Resource definition (possibly in YAML format) that caused the issue, without sensitive data:

/kind bug

Hi @signinred, sorry for replying this late.
Could you please specify the Vault image version, and Bank-Vaults version you are using? If you are still using Vault version 1.2.3 that might be an issue because this config did not exist back then. Otherwise I did a quick test locally and the field disable_iss_validation (and also disable_local_ca_jwt) was present and modifiable each time using various combination of Vault (1.6.2 and newer) and Bank-Vaults (1.13.1 and newer) versions. As far as I can tell, Bank-Vaults doesn't even touch this configuration, this field comes with Vault by default, so I am not sure what went wrong if not because of old version - can you provide more information on how to reproduce this issue? Thanks!

This issue seems to be stale so closing it, if @signinred you still have questions, feel free to open again or ask them in a separate issue/thread!


Authentication / accounts

Implement authentication and accounts.
It is a bit beyond what I'm currently doing here, but it would be cool to have this deployed as a service that other people could use. At the very least it could serve as an example of how to do authentication with Starlette.

I would like to work on this, do you have more details on this matter, which kind of auth would you like to have in here? E.g.: Implicit?, basic auth?
Would you like to have user groups? if so it will be spliced across different submissions
Do you want to limit access for users as groups or by individuals?
Will such access be by endpoint or by roles?
Best regards

@sjortiz I don't have any strong preferences as to the particulars, but ideally I think this could serve a community platform in addition to a personal notebook. I haven't really had a chance to explore the auth aspect at this point, do you have any suggestions?

we can start with basic auth and build on top to extend the methods later

@sjortiz sounds good. Haven't had a lot of time to work on this, but if you put forward a PR I can take a look.


How do you do .ThenInclude from F#?

After figuring out how to get the proxies to work in F# https://github.com/aspnet/EntityFrameworkCore/issues/12923, I tried to create queries using something like .Include(fun blog -> blog.Posts).ThenInclude(fun post -> post.Author), but post ends up being a collection. It fails in IntelliSense and during compilation. How do I make it work? Is there a workaround?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: bbd36f35-e257-d0ed-8f28-f539ddcb71ec
Version Independent ID: 093903d5-f7c7-5a78-c0e5-0e2662dcedef
Content: Loading Related Data - EF Core
Content Source: entity-framework/core/querying/related-data.md
Product: entity-framework
GitHub Login: @rowanmiller
Microsoft Alias: divega

@ctaggart We can look into this, but two things I can mention off the top of my head:

Visual Studio IntelliSense doesn't work for this in C#--see https://github.com/dotnet/roslyn/issues/8237 (ReSharper/Ryder IntelliSense works fine for me.)
You could try using string-based Include. That is, dot-separated navigation properties: .Include("Posts.Author")

Has anyone got this working? I am stuck trying to get child properties loading in F#. The string version doesn't seem to help, the child property is always null.

I can confirm that the string version of Include is working in F#.

@manchoz's comment shows how to do it without hardcoding strings:

F# won't automatically upcast List<T> to IEnumerable<T>. Explicitly upcasting the .Include() lambda to IEnumberable<T> seems to resolve the issue:

let q =
    db.Schemas
       .Include(fun sch -> sch.Entities :> IEnumerable<_>)
           .ThenInclude(fun (ents : Entity) -> ents.Fields)
       .ToList()


Dio freeze on widget test

i use Dio in my skeleton and iam trying to write widget test for my function which use dio for api call and show toast if there was error while making api request **the problem is widget test code freeze when i start making api request using dio **, i made a simple code just to view the problem
main() {
  // mock dio
  var dio = Dio();
  final dioAdapter = DioAdapter(dio: dio);

  testWidgets('test 500 error',(tester) async {
    // *) pump dummy widget
    await tester.pumpWidget(const Center(child: Text('haha'),));

    // *) dummy route
    String route = 'https://www.facebook/emadbeltaje.com';

    // *) simulate server 500 internal error
    dioAdapter.onGet(route, (server) {
      server.reply(500, null);
    });

    // *) perform api request on our dummy url
    try {
      // !) "" -> CODE FREEZE HERE <- ""
      Response response = await dio.get(route);
    }catch(error) {
      expect((error as DioError).response?.statusCode,500);
    }
  });
}

@AlexV525 why was this closed? I can still reproduce this on latest version

you switched from testWidget to just test,
but I solved it by wrapping the contents of my test with tester.runAsync and changing HttpOverrides.global to allow http requests

@ahmednfwela
iam sorry i ment this class here

`
void main() async {
// work fine
test('dio get test', () async {
await dioGetTest();
});
// work bad, always get 400 status code
test('dio get test', () async {
TestWidgetsFlutterBinding.ensureInitialized();
await dioGetTest();
});
// work fine
test('dio get test', () async {
IntegrationTestWidgetsFlutterBinding.ensureInitialized();
await dioGetTest();
});
}
Future dioGetTest() async {
final dio = Dio();
final res = await dio.get('https://xxx');
print(res);
}
`
What could be the reason behind this error?

This is normal behavior for Flutter widget tests. There are tons of related issues in the Flutter repository or on Stackoverflow.


ocnli dataset is broken

>>> datalabs.load_dataset("ocnli", None)
Couldn't find a directory or a dataset named 'ocnli' in this version. It was picked from the master branch on github instead.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/gneubig/opt/anaconda3/envs/explainaboard_web/lib/python3.10/site-packages/datalabs/load.py", line 2093, in load_dataset
    builder_instance = load_dataset_builder(
  File "/Users/gneubig/opt/anaconda3/envs/explainaboard_web/lib/python3.10/site-packages/datalabs/load.py", line 1859, in load_dataset_builder
    builder_cls = import_main_class(dataset_module.module_path)
  File "/Users/gneubig/opt/anaconda3/envs/explainaboard_web/lib/python3.10/site-packages/datalabs/load.py", line 121, in import_main_class
    module = importlib.import_module(module_path)
  File "/Users/gneubig/opt/anaconda3/envs/explainaboard_web/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/gneubig/.cache/expressai/modules/datasets_modules/datalab/ocnli/63bf5c94a45cee453a5f6232c2357ff17366279a49a24fc4d33bef4a60860d3a/ocnli.py", line 20, in <module>
    from datalabs.tasks import TextMatching
ImportError: cannot import name 'TextMatching' from 'datalabs.tasks' (/Users/gneubig/opt/anaconda3/envs/explainaboard_web/lib/python3.10/site-packages/datalabs/tasks/__init__.py)

Closing in favor of the more general issue https://github.com/ExpressAI/DataLab/issues/397


AB#29646 Add change password tests

Describe your changes
Adds change password tests for Portalicious
Checklist before requesting a review

[x] I have performed a self-review of my code
[x] I have added tests wherever relevant
[ ] I have made sure that all automated checks pass before requesting a review
[x] I do not need any deviation from our PR guidelines

Converting to draft because the tests are failing in CI (and also there are some rebase conflicts due to recent merges on main).

Converting to draft because the tests are failing in CI (and also there are some rebase conflicts due to recent merges on main).

Must have happened after the rebase I'll take care of it @aberonni


docs: add some documentation in case some problems arise with poetry

This is a tragedy because the colab notebook requires 3.7

Oh no! But I could not make it work with 3.7 at least in Mac M1. I can remove the part from .python-version but keep the update on the HACKING.md

Oh no! But I could not make it work with 3.7 at least in Mac M1. I can remove the part from .python-version but keep the update on the HACKING.md

Can we try debugging the error you're encountering with an M1? What's the error message and environment setup?

Closing this at it arose from using poetry installed via brew


Use MPEG4 Container

Depending on #229, running the tests to confirm that the reliability of video encoding has improved now there is timebase conversion in #229.

@facebook-github-bot import

Accepted on Phab


Allows empty EF field in spec details

Thanks for your work on this library !
We used this library with some producer where the "EF" fields is not provided. This merge-request allows this library to work in this particular case.

Thanks you for your pull request, I just merged it


bug(COMPONENT): Table border offsets when scrolling up and down

Is this a regression?

[ ] Yes, this behavior used to work in the previous version

The previous version in which this bug was not present was
No response
Description
When scrolling down the table, table offsets 1px up

Reproduction
Steps to reproduce:

Go to https://material.angular.io/components/table/examples
Locate Table with sticky header
Set the table css in DevTools border: 1px solid red so that it is easier to notice the issue
Scroll down
Notice that the table top border also moves up 1px.
6 Scroll all the way down to the bottom
Scroll up
Notice that the table bottom border also moves down 1px

Expected Behavior
Table border should never offset
Actual Behavior
Border top and bottom offsets 1px down and up along with the scrolling up and down
Environment

Angular:
CDK/Material:
Browser(s):
Operating System (e.g. Windows, macOS, Ubuntu):

You could add for the head some style with
position: sticky;
z-index: 10;

Sounds little bit hacky especially since you already provided a property for being sticky. But at least this fixes this problem. I believe as well that #15885 is somehow related to this issue.


Can you add proxy function ?

I can not visit www.packtpub.com without proxy ,so i wonder if you can add proxy function.
like socks5 or http, thanks

I have no plan for this right now, You can open pul lrequest and add it on your own :)


[WIP] Working on separating parsing/lexing/semantic logic

Trying to grok the parsing system, with the eventual goal of providing a relatively complete language specification.
So far:

move lexer to own module

Thank you!
I'm looking forward to the relatively complete language specification.


No option to see the full emailaddresses in Google Groups

There is an option in Gam to configure to view the members of a list, this option only gives partial emailaddresses visibility in Google groups.
There needs to be a additional option set "View Email Addresses" in the web interface to see the email addresses completely. This option is not yet available in GAM

What command are you issuing? What email addresses are missing?
GAM has nothing to do with the Web interface, if you want a change there you'll have to ask Google.
Ross

Maybe my request was not clear enough, nothing is broken, just a configure option missing in GAM which you can set in Groups web interface.
As I am adding hundreds of groups I would like the set this setting in bulk, not by clicky clicky via the web for all these groups.
When you issue command : gam update group XXX who_can_view_membership all_in_domain_can_view
Everybody can see the list of members of the group, but by default is is in a masked mode like<EMAIL_ADDRESS>in the web there an additional option to set "view Email address" to uncover the mask and see the whole emailaddress.
I looked yesterday quickly through the API's of groups but could not immediately find this specific option back. Not sure if it is something new, I never noticed this option before in Groups.

There's no API support for this setting, thus nothing GAM can do here. Closing.


tutorials/peft-curation-with-sdg Failing with Runtime Exceptions.

We have attempted to run tutorials/peft-curation-with-sdg and facing runtime errors, details are mentioned below with the environment setup information we tried.
python ./main.py \
	--api-key <Token From: https://build.nvidia.com/nvidia/nemotron-4-340b-instruct?integrate_nim=true&hosted_api=true > \
	--device gpu \
	--synth-gen-rounds 1 \
	--synth-gen-ratio 0.001 \
	--synth-gen-model "nvidia/nemotron-4-340b-instruct"

Environment Setup
conda create --name nemo python==3.10.12
conda activate nemo

apt-get update && apt-get install -y libsndfile1 ffmpeg
pip install Cython packaging
pip install nemo_toolkit['all']

git clone https://github.com/NVIDIA/NeMo.git

cd ./NeMo

apt install gcc-12 g++-12

# Then you need to set this version as default one:
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 1

conda activate nemo

cd ./NeMo

python ./setup.py install

conda install rapidsai::cudf
conda install rapidsai::dask-cudf
conda install rapidsai::cuml

pip install annotated-types
pip install pydantic-core
pip install httpcore
pip install lxml-html-clean

Errors
Errors for Dataset: https://huggingface.co/datasets/ymoslem/MedicalSciences-StackExchange/resolve/main/medical.stackexchange-questions-answers.json?download=true
Key:       ('lambda-cfabe91fdbaed95ddbb48379ec3121d3', 9)
State:     executing
Function:  execute_task
args:      ((<function reify at 0x7d3869cdfbe0>, (<function map_chunk at 0x7d3869cfc040>, <function SemanticClusterLevelDedup.compute_semantic_match_dfs.<locals>.<lambda> at 0x7d36e7f8fbe0>, [[9]], None, {})))
kwargs:    {}
Exception: 'ValueError("invalid literal for int() with base 10: \'law-stackexchange-qa-9905\'")'
Traceback: '  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/bag/core.py", line 1868, in reify\n    seq = list(seq)\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/bag/core.py", line 2056, in __next__\n    return self.f(*vals)\n  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/modules/semantic_dedup.py", line 456, in <lambda>\n    lambda cluster_id: get_semantic_matches_per_cluster(\n  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/utils/semdedup_utils.py", line 252, in get_semantic_matches_per_cluster\n    text_ids = cluster_i[:, 0].astype(id_col_type)\n'

2024-08-14 12:04:51,964 - distributed.worker - WARNING - Compute Failed
Key:       ('lambda-cfabe91fdbaed95ddbb48379ec3121d3', 8)
State:     executing
Function:  execute_task
args:      ((<function reify at 0x7d3869cdfbe0>, (<function map_chunk at 0x7d3869cfc040>, <function SemanticClusterLevelDedup.compute_semantic_match_dfs.<locals>.<lambda> at 0x7d36f199f490>, [[8]], None, {})))
kwargs:    {}
Exception: 'ValueError("invalid literal for int() with base 10: \'law-stackexchange-qa-19252\'")'
Traceback: '  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/bag/core.py", line 1868, in reify\n    seq = list(seq)\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/bag/core.py", line 2056, in __next__\n    return self.f(*vals)\n  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/modules/semantic_dedup.py", line 456, in <lambda>\n    lambda cluster_id: get_semantic_matches_per_cluster(\n  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/utils/semdedup_utils.py", line 252, in get_semantic_matches_per_cluster\n    text_ids = cluster_i[:, 0].astype(id_col_type)\n'

Traceback (most recent call last):
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py", line 415, in <module>
    main()
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py", line 406, in main
    train_fp_curated = run_pipeline(args, train_fp)
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py", line 310, in run_pipeline
    dataset_df, n_rows_before, n_rows_after = run_curation_pipeline(args, jsonl_fp)
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py", line 227, in run_curation_pipeline
    dataset = gpu_curation_steps(dataset)
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/modules/meta.py", line 22, in __call__
    dataset = module(dataset)
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py", line 141, in semantic_dedupe
    dedup_ids = semdup(dataset)
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/modules/semantic_dedup.py", line 570, in __call__
    self.semantic_cluster_dedup.compute_semantic_match_dfs(self.eps_thresholds)
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/modules/semantic_dedup.py", line 467, in compute_semantic_match_dfs
    tasks.compute()
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/base.py", line 376, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/base.py", line 662, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/bag/core.py", line 1868, in reify
    seq = list(seq)
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/bag/core.py", line 2056, in __next__
    return self.f(*vals)
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/modules/semantic_dedup.py", line 456, in <lambda>
    lambda cluster_id: get_semantic_matches_per_cluster(
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/utils/semdedup_utils.py", line 252, in get_semantic_matches_per_cluster
    text_ids = cluster_i[:, 0].astype(id_col_type)
ValueError: invalid literal for int() with base 10: 'law-stackexchange-qa-9905'
ERROR conda.cli.main_run:execute(125): `conda run python /data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py --api-key Ynvapi-iGgFsXrdl4FkGA7MFPzNgL3r8o2c_dcZ7SGT8Yx7ZlYNWNix2xnX8IezbxTVbcPW --device gpu --synth-gen-rounds 1 --synth-gen-ratio 0.001 --synth-gen-model nvidia/nemotron-4-340b-instruct` failed. (See above for error)

Errors for Dataset: https://huggingface.co/datasets/ymoslem/Law-StackExchange/resolve/main/law-stackexchange-questions-answers.json
future: <Task finished name='Task-8451' coro=<tqdm_asyncio.gather.<locals>.wrap_awaitable() done, defined at /home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/tqdm/asyncio.py:75> exception=AuthenticationError('Error code: 401')>
Traceback (most recent call last):
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/tqdm/asyncio.py", line 76, in wrap_awaitable
    return i, await f
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/synthetic_gen.py", line 221, in _prompt_model
    gen_title, gen_question, gen_answer = await asyncio.gather(
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/synthetic/async_nemotron.py", line 421, in generate_closed_qa_instructions
    openline_response = await self._prompt(
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/synthetic/async_nemotron.py", line 77, in _prompt
    return await self.client.query_model(
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/services/openai_client.py", line 123, in query_model
    response = await self.client.chat.completions.create(
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/openai-1.40.2-py3.10.egg/openai/resources/chat/completions.py", line 1339, in create
    return await self._post(
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/openai-1.40.2-py3.10.egg/openai/_base_client.py", line 1815, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/openai-1.40.2-py3.10.egg/openai/_base_client.py", line 1509, in request
    return await self._request(
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/openai-1.40.2-py3.10.egg/openai/_base_client.py", line 1610, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.AuthenticationError: Error code: 401
Task exception was never retrieved
future: <Task finished name='Task-8452' coro=<tqdm_asyncio.gather.<locals>.wrap_awaitable() done, defined at /home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/tqdm/asyncio.py:75> exception=AuthenticationError('Error code: 401')>
Traceback (most recent call last):
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/tqdm/asyncio.py", line 76, in wrap_awaitable
    return i, await f
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/synthetic_gen.py", line 221, in _prompt_model
    gen_title, gen_question, gen_answer = await asyncio.gather(
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/synthetic/async_nemotron.py", line 421, in generate_closed_qa_instructions
    openline_response = await self._prompt(
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/synthetic/async_nemotron.py", line 77, in _prompt
    return await self.client.query_model(
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/services/openai_client.py", line 123, in query_model
    response = await self.client.chat.completions.create(
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/openai-1.40.2-py3.10.egg/openai/resources/chat/completions.py", line 1339, in create
    return await self._post(
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/openai-1.40.2-py3.10.egg/openai/_base_client.py", line 1815, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/openai-1.40.2-py3.10.egg/openai/_base_client.py", line 1509, in request
    return await self._request(
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/openai-1.40.2-py3.10.egg/openai/_base_client.py", line 1610, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.AuthenticationError: Error code: 401
Task exception was never retrieved
future: <Task finished name='Task-8455' coro=<tqdm_asyncio.gather.<locals>.wrap_awaitable() done, defined at /home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/tqdm/asyncio.py:75> exception=AuthenticationError('Error code: 401')>
Traceback (most recent call last):
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/tqdm/asyncio.py", line 76, in wrap_awaitable
    return i, await f
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/synthetic_gen.py", line 221, in _prompt_model
    gen_title, gen_question, gen_answer = await asyncio.gather(
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/synthetic/async_nemotron.py", line 421, in generate_closed_qa_instructions
    openline_response = await self._prompt(
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/synthetic/async_nemotron.py", line 77, in _prompt
    return await self.client.query_model(
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/services/openai_client.py", line 123, in query_model
    response = await self.client.chat.completions.create(
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/openai-1.40.2-py3.10.egg/openai/resources/chat/completions.py", line 1339, in create
    return await self._post(
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/openai-1.40.2-py3.10.egg/openai/_base_client.py", line 1815, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/openai-1.40.2-py3.10.egg/openai/_base_client.py", line 1509, in request
    return await self._request(
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/openai-1.40.2-py3.10.egg/openai/_base_client.py", line 1610, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.AuthenticationError: Error code: 401
Task exception was never retrieved
future: <Task finished name='Task-8458' coro=<tqdm_asyncio.gather.<locals>.wrap_awaitable() done, defined at /home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/tqdm/asyncio.py:75> exception=AuthenticationError('Error code: 401')>
Traceback (most recent call last):
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/tqdm/asyncio.py", line 76, in wrap_awaitable
    return i, await f
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/synthetic_gen.py", line 221, in _prompt_model
    gen_title, gen_question, gen_answer = await asyncio.gather(
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/synthetic/async_nemotron.py", line 421, in generate_closed_qa_instructions
    openline_response = await self._prompt(
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/synthetic/async_nemotron.py", line 77, in _prompt
    return await self.client.query_model(
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/services/openai_client.py", line 123, in query_model
    response = await self.client.chat.completions.create(
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/openai-1.40.2-py3.10.egg/openai/resources/chat/completions.py", line 1339, in create
    return await self._post(
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/openai-1.40.2-py3.10.egg/openai/_base_client.py", line 1815, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/openai-1.40.2-py3.10.egg/openai/_base_client.py", line 1509, in request
    return await self._request(
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/openai-1.40.2-py3.10.egg/openai/_base_client.py", line 1610, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.AuthenticationError: Error code: 401

Errors for Dataset: Created a custom dataset based on law-stackexchange-questions-answers.json with one single record.
Reading 1 files
Traceback (most recent call last):
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
    return self._engine.get_loc(casted_key)
  File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'title'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py", line 415, in <module>
    main()
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py", line 406, in main
    train_fp_curated = run_pipeline(args, train_fp)
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py", line 310, in run_pipeline
    dataset_df, n_rows_before, n_rows_after = run_curation_pipeline(args, jsonl_fp)
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py", line 207, in run_curation_pipeline
    dataset = cpu_curation_steps(dataset)
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/modules/meta.py", line 22, in __call__
    dataset = module(dataset)
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/modules/modify.py", line 31, in __call__
    dataset.df[self.text_field] = dataset.df[self.text_field].apply(
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/dataframe/core.py", line 4951, in __getitem__
    meta = self._meta[_extract_meta(key)]
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/pandas/core/frame.py", line 4102, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
    raise KeyError(key) from err
KeyError: 'title'
ERROR conda.cli.main_run:execute(125): `conda run python /data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py --api-key Ynvapi-iGgFsXrdl4FkGA7MFPzNgL3r8o2c_dcZ7SGT8Yx7ZlYNWNix2xnX8IezbxTVbcPW --device gpu --synth-gen-rounds 1 --synth-gen-ratio 0.001 --synth-gen-model nvidia/nemotron-4-340b-instruct` failed. (See above for error)

Errors for Dataset: Created a custom dataset based on law-stackexchange-questions-answers.json with 100k+ entries.
Reading 1 files
/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
2024-08-14 14:13:03,126 - distributed.worker - WARNING - Compute Failed
Key:       ('to-parquet-fc0472286679d49f53313f93027e9065', 0)
State:     executing
Function:  subgraph_callable-5e534f6954054123de078e717e9367e9
args:      ((0,), '<crossfit.backend.torch.op.base.Predictor object a-18befc5a9c8e11d3a125f4e2f611e20c', {'number': 0, 'division': None}, '<crossfit.op.tokenize.Tokenizer object at 0x7cca6c-3510ecebf68286e44567efef058f7f2b', 'to_cudf_dispatch-25a6cae5e12e52fe804538d37bb046b3', 'assign-42e4fd5d877ed0406e2080192bc11744', 'text', 'answer', '\n', 'title', 'question', <bound method FilterLowScores.keep_document of <filters.FilterLowScores object at 0x737a3c503310>>, 'apply-f00d3b1f0bc5365043d5ad449cc081ad', <bound method FilterLowScores.score_document of <filters.FilterLowScores object at 0x737a3c503310>>, 'getitem-340ac611e67c7235a472afe0fa6529fa', 'answer_score', <bound method FilterLowScores.keep_document of <filters.FilterLowScores object at 0x737a3c503370>>, 'apply-5d07bbb647c3e1c3ea837331d3b646b3', <bound method FilterLowScores.score_document of <filters.FilterLowScores object at 0x737a3c503370>>, 'getitem-ad0338cf88d799fa6c5767bd2922a2f4', 'question_score', <bound method WordCountFilter.keep_do
kwargs:    {}
Exception: "IndexError('list index out of range')"
Traceback: '  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/optimization.py", line 1001, in __call__\n    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/core.py", line 157, in get\n    result = _execute_task(task, cache)\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/core.py", line 127, in _execute_task\n    return func(*(_execute_task(a, cache) for a in args))\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/core.py", line 127, in <genexpr>\n    return func(*(_execute_task(a, cache) for a in args))\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/core.py", line 127, in _execute_task\n    return func(*(_execute_task(a, cache) for a in args))\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/core.py", line 127, in <genexpr>\n    return func(*(_execute_task(a, cache) for a in args))\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/core.py", line 121, in _execute_task\n    return [_execute_task(a, cache) for a in arg]\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/core.py", line 121, in <listcomp>\n    return [_execute_task(a, cache) for a in arg]\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/core.py", line 127, in _execute_task\n    return func(*(_execute_task(a, cache) for a in args))\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/core.py", line 127, in <genexpr>\n    return func(*(_execute_task(a, cache) for a in args))\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/core.py", line 121, in _execute_task\n    return [_execute_task(a, cache) for a in arg]\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/core.py", line 121, in <listcomp>\n    return [_execute_task(a, cache) for a in arg]\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/core.py", line 127, in _execute_task\n    return func(*(_execute_task(a, cache) for a in args))\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/utils.py", line 78, in apply\n    return func(*args, **kwargs)\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/dataframe/core.py", line 7164, in apply_and_enforce\n    df = func(*args, **kwargs)\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/dataframe/core.py", line 7073, in func\n    return orig_func(*args, **kwargs, partition_info=partition_info)\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/crossfit-0.0.4-py3.10.egg/crossfit/op/base.py", line 96, in __call__\n    output = self.call(data, *args, **kwargs)\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/crossfit-0.0.4-py3.10.egg/crossfit/op/tokenize.py", line 152, in call\n    input_ids, attention_mask = self.call_column(data[col])\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/crossfit-0.0.4-py3.10.egg/crossfit/op/tokenize.py", line 117, in call_column\n    tokenized_data = self.tokenize_strings(text).copy()\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/crossfit-0.0.4-py3.10.egg/crossfit/op/tokenize.py", line 68, in tokenize_strings\n    tokenized_data = tokenizer.batch_encode_plus(\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3338, in batch_encode_plus\n    return self._batch_encode_plus(\n  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 561, in _batch_encode_plus\n    for key in tokens_and_encodings[0][0].keys():\n'

Traceback (most recent call last):
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py", line 415, in <module>
    main()
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py", line 406, in main
    train_fp_curated = run_pipeline(args, train_fp)
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py", line 310, in run_pipeline
    dataset_df, n_rows_before, n_rows_after = run_curation_pipeline(args, jsonl_fp)
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py", line 227, in run_curation_pipeline
    dataset = gpu_curation_steps(dataset)
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/modules/meta.py", line 22, in __call__
    dataset = module(dataset)
  File "/data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py", line 141, in semantic_dedupe
    dedup_ids = semdup(dataset)
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/modules/semantic_dedup.py", line 568, in __call__
    embeddings_dataset = self.embedding_creator(dataset)
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/modules/semantic_dedup.py", line 202, in __call__
    write_to_disk(
  File "/data/ops/roshan/remote/nemo-curator/nemo_curator/utils/distributed_utils.py", line 519, in write_to_disk
    df.to_parquet(output_file_dir, write_index=False)
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/cudf/utils/performance_tracking.py", line 51, in wrapper
    return func(*args, **kwargs)
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask_cudf/core.py", line 283, in to_parquet
    return to_parquet(self, path, *args, **kwargs)
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/dataframe/io/parquet/core.py", line 1047, in to_parquet
    out = out.compute(**compute_kwargs)
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/base.py", line 376, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/dask/base.py", line 662, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/crossfit-0.0.4-py3.10.egg/crossfit/op/base.py", line 96, in __call__
    output = self.call(data, *args, **kwargs)
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/crossfit-0.0.4-py3.10.egg/crossfit/op/tokenize.py", line 152, in call
    input_ids, attention_mask = self.call_column(data[col])
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/crossfit-0.0.4-py3.10.egg/crossfit/op/tokenize.py", line 117, in call_column
    tokenized_data = self.tokenize_strings(text).copy()
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/crossfit-0.0.4-py3.10.egg/crossfit/op/tokenize.py", line 68, in tokenize_strings
    tokenized_data = tokenizer.batch_encode_plus(
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3338, in batch_encode_plus
    return self._batch_encode_plus(
  File "/home/praxi-usr/anaconda3/envs/nemo/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 561, in _batch_encode_plus
    for key in tokens_and_encodings[0][0].keys():
IndexError: list index out of range
ERROR conda.cli.main_run:execute(125): `conda run python /data/ops/roshan/remote/nemo-curator/tutorials/peft-curation-with-sdg/main.py --api-key Ynvapi-iGgFsXrdl4FkGA7MFPzNgL3r8o2c_dcZ7SGT8Yx7ZlYNWNix2xnX8IezbxTVbcPW --device gpu --synth-gen-rounds 1 --synth-gen-ratio 0.001 --synth-gen-model nvidia/nemotron-4-340b-instruct` failed. (See above for error)

Hi @praxi-roshan, thanks for opening this issue and providing detailed repro steps. I just attempted running the tutorial on the latest main commit with a freshly setup environment and it seems to work fine for me.
What I tried:
python3 --version # mine was 3.10.13
python3 -m venv ./venv
source ./venv/bin/activate
git clone https://github.com/NVIDIA/NeMo-Curator.git
cd NeMo-Curator
pip install --extra-index-url https://pypi.nvidia.com ".[cuda12x]"
python tutorials/peft-curation-with-sdg/main.py --api-key <my_key> --device gpu

A few questions for you, so I can better assist you:

Did you happen to try running the tutorial without any modifications to see if works at all for you?
On that note, did you try running without providing an API key to see whether the regular data curation bits work as expected? I see some authentication errors, which may suggest your build.nvidia.com key may not have worked properly.
Similarly, did you happen to try with --device cpu to further isolate the scope of the issue?
I noticed that you included your command for installing the NeMo framework (pip install nemo_toolkit['all']), but I didn't quite figure out how the NeMo Curator was installed. How was it installed?

Hi @Maghoumi,
Thank you for the instructions, i tried installing based on your steps I'm getting a different error now:
OSError: libcudart.so: cannot open shared object file: No such file or directory
I will troubleshoot and provide more feedback.
4. I noticed that you included your command for installing the NeMo framework (pip install nemo_toolkit['all']), but I didn't quite figure out how the NeMo Curator was installed. How was it installed?
Apologies in the post i mentioned wrong repo...! (https://github.com/NVIDIA/NeMo.git), This is how i installed
conda create --name nemo python==3.10.12
conda activate nemo

apt-get update && apt-get install -y libsndfile1 ffmpeg
pip install Cython packaging
pip install nemo_toolkit['all']

git clone https://github.com/NVIDIA/NeMo-Curator.git

cd ./NeMo-Curator

apt install gcc-12 g++-12

# Then you need to set this version as default one:
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 1

conda activate nemo

cd ./NeMo-Curator

python ./setup.py install

conda install rapidsai::cudf
conda install rapidsai::dask-cudf
conda install rapidsai::cuml

pip install annotated-types
pip install pydantic-core
pip install httpcore
pip install lxml-html-clean

Thank you for the instructions, i tried installing based on your steps I'm getting a different error now:
OSError: libcudart.so: cannot open shared object file: No such file or directory

At which stage of running the tutorial do you encounter this issue?
This error typically indicates an issue with CUDA installation and would mean the CUDA library files were not found. Since this is OS dependent, I would advise ensuring CUDA is properly installed in your environment before running the code. I often tend to use the following snippet to ensure CUDA is installed properly when I setup new environments (assuming that you have PyTorch installed):
import torch
print(torch.cuda.is_available())  # Should print True if CUDA is installed properly

Looking at your install instructions, I see you have:
conda activate nemo
cd ./NeMo-Curator
python ./setup.py install  # < -- have you tried `pip install .` instead?

Asking because pip install . should ensure dependencies are also handled properly.

Hi,
I tried on a new Ubuntu 23.10 Server environment with the below mentioned environment setup,
and apparently it worked without any errors and created the curated files.
I have attached the complete log file of the curation process for your reference to confirm that it worked fine.
logs.log

Looks like synthetic data generation not possible without an API key, could you please let us known is there a workaround that we can generate synthetic data without an API Key?
Is there a way we can generate synthetic data without OpenAI client and using a locally downloaded LLM model?

conda create --name nemo python==3.10.13

conda deactivate
conda activate nemo

pip install Cython packaging

# Installed this to resolve this error: "OSError: libcudart.so: cannot open shared object file: No such file or directory"
conda install cudatoolkit=11.8

git clone https://github.com/NVIDIA/NeMo-Curator.git

cd ./NeMo-Curator

apt install gcc-12 g++-12

# Then you need to set this version as default one:
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 1

pip install --extra-index-url https://pypi.nvidia.com ".[cuda12x]"

python tutorials/peft-curation-with-sdg/main.py --api-key <my_key> --device gpu**

Glad to hear that you were able to follow along after using another environment. Regarding your questions:

Looks like synthetic data generation not possible without an API key, could you please let us known is there a workaround that we can generate synthetic data without an API Key?

As you may have noticed already, currently the framework assumes that the LLM model that can generate synthetic data is accessible via some sort of an HTTP endpoint, which is either compatible with the OpenAI API, or the NeMo Framework API.
The default example aims to show the usage of externally hosted synthetic generator models (e.g. models hosted on on build.nvidia.com, or some other commercial gateways). This requires the usage of an API key to be able to be able to use that model. Which brings us to your next question:

Is there a way we can generate synthetic data without OpenAI client and using a locally downloaded LLM model?

You can certainly use locally downloaded models. The easiest path to do that would be to deploy the model yourself and make it available through an HTTP endpoint. This can be done via Nvidia NIMs (sign up for early access at https://developer.nvidia.com/nemo-microservices), or other solutions such as vLLM or TGI.
Once you deploy the model, you can treat it like any other externally hosted LLM and start querying it via the OpenAI API and your local endpoint (e.g. http://localhost:8000). Make sure to update the endpoint locations inside main.py so that it knows where to look for.
Hope this helps!

Hi @praxi-roshan, let me know the status of this issue, and whether we are good to close it.
Also, I also opened a new PR to improve the documentation for using LLM endpoints for synthetic data generation, hopefully that should make it a bit more clear for anybody else who's looking for using arbitrary LLM model deployments to run the tutorial: https://github.com/NVIDIA/NeMo-Curator/pull/301

Hi @Maghoumi,
I will close the ticket, thank you for the assistance.


SetSubMatrix problem

Hello
I'm using Math.Net to perform FEM code, and i get some issuse in usage of SetSubMatrix in Matrix objects.
I'm creating 6 Dense Matrices 6x6, and I'm composing from them 24x24 matrix.
But, the end Matrix, after composing contains "holes" I've use DelimitedWriter.Write to export matrix to Excel and this is what I get...

composing of 24x24 matrix part looks like this. All matrix K1,K2...K6 (which are 6x6)are fulfilled with doubles...
//Composing Ke matrix
// * Ke = {K1, K2, K3, K4
//  *      K2T, K5, K6, K3T
// *       K3T, K6, K5T, K2T
// *       K4, K3, K2, K1T}
//Row 0
Ke.SetSubMatrix(0, 0, K1);
Ke.SetSubMatrix(0, 6, K2);
Ke.SetSubMatrix(0, 12, K3);
Ke.SetSubMatrix(0, 18, K4);
//Row 6
Ke.SetSubMatrix(6, 0, K2.Transpose());
Ke.SetSubMatrix(6, 6, K5);
Ke.SetSubMatrix(6, 12, K6);
Ke.SetSubMatrix(6, 18, K3.Transpose());
////Row 12
Ke.SetSubMatrix(12, 0, K3.Transpose());
Ke.SetSubMatrix(12, 6, K6);
Ke.SetSubMatrix(12, 12, K5.Transpose());
Ke.SetSubMatrix(12, 18, K2.Transpose());
////Row 18
Ke.SetSubMatrix(18, 0, K4);
Ke.SetSubMatrix(18, 6, K3);
Ke.SetSubMatrix(18, 12, K2);
Ke.SetSubMatrix(18, 18, K1.Transpose());
If I comment randomly any setsubmatrix from above, it gives different result....
As you see in excel, empty places are not even Zeros... that's really strange behaviour.
If you will need I can send you whole class.
Could you investigate what is wrong with this function....
Thank You

Ugh, I'll have a look!

Hm, I cannot reproduce so far. This is what I tried with Math.NET Numerics 3.4.0 in LinqPad (hence the Dump calls):
var k = Matrix<double>.Build.Dense(6,6,(i,j) => 100.0 + 10.0*i+j);

// simple way
var m1 = Matrix<double>.Build.DenseOfMatrixArray(new[,]
    {{k,k*2,k*3,k*4},
    {k*5,k*6,k*7,k*8},
    {k*9,k*10,k*11,k*12},
    {k*13,k*14,k*15,k*16}});
m1.ToArray().Dump();

// doing it manually
var m2 = Matrix<double>.Build.Dense(24,24);
m2.SetSubMatrix(0,0,k);
m2.SetSubMatrix(0,6,k*2);
m2.SetSubMatrix(0,12,k*3);
m2.SetSubMatrix(0,18,k*4);
m2.SetSubMatrix(6,0,k*5);
m2.SetSubMatrix(6,6,k*6);
m2.SetSubMatrix(6,12,k*7);
m2.SetSubMatrix(6,18,k*8);
m2.SetSubMatrix(12,0,k*9);
m2.SetSubMatrix(12,6,k*10);
m2.SetSubMatrix(12,12,k*11);
m2.SetSubMatrix(12,18,k*12);
m2.SetSubMatrix(18,0,k*13);
m2.SetSubMatrix(18,6,k*14);
m2.SetSubMatrix(18,12,k*15);
m2.SetSubMatrix(18,18,k*16);
m2.ToArray().Dump();

// difference is all 0
(m1 - m2).ToArray().Dump();

// verify DelimitedWriter can print it properly
var writer = new StringWriter();
DelimitedWriter.Write(writer,m1);
writer.ToString().Dump();

What version of Math.NET Numerics are you using?

Since dense matrices cannot technically have "no value" at some place this was more likely an issue outside of Math.NET Numerics (maybe an old version of the text data package?). I'm closing this, but please reopen if you can indeed reproduce with current versions.


Added PowerShell details for Windows 7

Added details to resolve https://github.com/geerlingguy/drupal-vm/issues/231

Since Windows 7 is a few versions old at this point, and this issue shouldn't affect users with Windows 8 and (sooner or later, once all the tools work on it, Windows 10), can you add this note in the docs/other/windows.md documentation, under a new section, and take the change out of README.txt?

Thanks!


cargo update

@bors r+

:pushpin: Commit a555e1ec25389ab870009387812d07237badfbcd has been approved by RalfJung

:hourglass: Testing commit a555e1ec25389ab870009387812d07237badfbcd with merge 3c444bf6a6cff3b9014005f21cc44995b34862ce...

:sunny: Test successful - checks-travis, status-appveyor
Approved by: RalfJung
Pushing 3c444bf6a6cff3b9014005f21cc44995b34862ce to master...


v0.0.5

Added Scroll Axis for Infinite Page View

Codecov Report
Patch coverage: 100.00% and project coverage change: +0.35% :tada:

Comparison is base (e73f16a) 81.59% compared to head (29b5a9b) 81.95%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master       #3      +/-   ##
==========================================
+ Coverage   81.59%   81.95%   +0.35%
==========================================
  Files           3        3
  Lines         201      205       +4
==========================================
+ Hits          164      168       +4
  Misses         37       37

Files Changed
Coverage Δ

...ib/infinite_page_view/_infinite_page_provider.dart
69.67% <100.00%> (ø)

lib/infinite_page_view/infinite_page_view.dart
100.00% <100.00%> (ø)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


DialogV2: re-set focusTrap when footer content changes

Closes https://github.com/github/primer/issues/3358, https://github.com/github/primer/issues/3356, https://github.com/github/primer/issues/3357

Adds footer as a dependency of useFocusTrap, ensures that if content is changed within either, focus trap will properly focus newly added focusable elements. Example story: Repro Multistep Dialog With Conditional Footer.
Changelog

Changed

Adds footer to useFocusTrap dependencies
Removes autofocus prop from stories

Rollout strategy

[ ] Patch release
[x] Minor release
[ ] Major release; if selected, include a written rollout or migration plan
[ ] None; if selected, include a brief description as to why

Testing & Reviewing

Merge checklist

[ ] Added/updated tests
[ ] Added/updated documentation
[x] Added/updated previews (Storybook)
[ ] Changes are SSR compatible
[x] Tested in Chrome
[ ] Tested in Firefox
[ ] Tested in Safari
[ ] Tested in Edge
[ ] (GitHub staff only) Integration tests pass at github/github (Learn more about how to run integration tests)

I had to utilize querySelector to search for the close button. I'm wondering if there'd be any benefit to letting consumers add a ref directly to it via a prop (e.g. closeButtonRef)? This fix is only required for a story, so it's low priority.

Logic is sound. Just need to test for any performance implications

That is fair! Is there a good way to go about this outside of just testing existing dialogs? Another option is making this opt-in, but not too sure if that's the best route to take if we want this functionality by default 🤔

That is fair! Is there a good way to go about this outside of just testing existing dialogs?

Can see how if the hook is called only when it should or also on every render or on unrelated changes inside the Dialog content.

Another option is making this opt-in, but not too sure if that's the best route to take if we want this functionality by default 🤔

I think this should be default as well

That is fair! Is there a good way to go about this outside of just testing existing dialogs?

Can see how if the hook is called only when it should or also on every render or on unrelated changes inside the Dialog content. Dependency on a const body is looks doubtful to me

Another option is making this opt-in, but not too sure if that's the best route to take if we want this functionality by default 🤔

I think this should be default as well

I removed body as a dependency as I don't think we actually need it. I believe that focus-trap automatically adds elements that are appended to the container, so we don't have to add it as a dependency, as it would only be redundant. If I had to guess, content inside of the footer is added after the last focused element when the focus trap is initialized, meaning it wont be included in the focus trap, only elements that are within the first focusable element and the last focusable elements are included. This is only an assumption looking at the focus-trap behavior 😅 .
I left footer as a dependency as we'd want to include content here if it's added after the fact, like in the repro multistep story. This is also less risky when it comes to performance, as the footer is less likely to have re-renders like body would (if any) after the initial render. We could also include header here, as it will most likely fall into the same group as the footer.

I added some state in the story to show the effect of dependency (committed the change to story), now if you add a console.log to the line where focusTrap is called, you see how many times it is called.
Video description:
Part 1, with dependencies:

When the Dialog opens, you see "setting focus trap" printed twice (let's ignore the second one for now)
With dependencies like footer or body added without "memo", you see "setting focus trap" printed once more. This is good! We want the focus trap to be set again for changed footer content.
Entering text into the input causes the component to re-render and all the variables inside it too. You see "setting focus trap" printed on every keystroke! This removing and attaching event listeners can cause performance regressions.

Part 2, after commenting out dependencies

When the Dialog opens, you see "setting focus trap" printed twice (same as before)
With no dependencies array, you don't see any more "setting focus trap" printed. Focus trap is not resetting when the footer content changed
Entering text into the input causes the component to re-render and all the variables inside it too. But, you don't see "setting focus trap" printed because we are not passing a dependencies to useFocusTrap

https://github.com/user-attachments/assets/d4faf9d1-9307-43f5-9320-1ba973a0a4e1

Thanks a ton for the notes and video @siddharthkp, it helps a lot to see where the problem is ✨  The current route is definitely not the best for performance as we're reinitializing the focus trap each render 😬
Digging into this a bit more, I have a few alternatives without us utilizing dependencies, and I would love your opinion!

Utilize a mutation observer in focus-trap behavior

We could use a mutation observer in our focus trap behavior to account for new focusable elements added. We do this in focus-zone already, so we'd be able to take from an existing implementation. There is still some performance concern, but I don't think it'd be as bad if we're handling it correctly. This also means that on re-renders we wouldn't be initializing the focus trap once set.

Adjust "sentinel" to always be at the start/end of the container

When the dialog footer is conditionally rendered, the ending sentinel is before the footer, meaning that it's not encompassing the entirety of the dialog's content. Technically, it should encompass all elements within the container passed to focusTrap. We could adjust the behavior to always be at the start and end of the container so that any elements within are always accounted for.

These are just two options, both requiring changes to the behavior, rather than the useFocusTrap hook. I'm still digging to see if there are any other routes we could take, or if it's possible to keep the dependency option, but only run the focus trap behavior when needed (i.e. footer/header has changed significantly from the previous render).

We do this in focus-zone already, so we'd be able to take from an existing implementation. There is still some performance concern, but I don't think it'd be as bad if we're handling it correctly. This also means that on re-renders we wouldn't be initializing the focus trap once set

Ooh i really like that idea!
I wonder how eager should the observer be. We haven't received any performance concerns about focus zone yet, but it's also really difficult to attribute it to that 😅

Quick update, working on a fix in primer/behaviors: https://github.com/primer/behaviors/pull/429. Basically adding a mutation observer that checks if the start or end sentinels are no longer at the start/end.

@siddharthkp, tested this with the updated behaviors version & very lightly in dotcom (though I couldn't find too many dialogs). Doesn't look like there's much performance hit, as the likelihood of content being added before/after sentinels is low. In cases where elements are added after the focus trap, they will only be moved once once it's asserted that either the first child or last child in the focus trap is a .sentinel.
I'm curious if there's any additional testing we should do, or if we should ship this change (with the behaviors bump).

Great, thanks for making sure! I think it sounds good, let's ship it


Short discussions don't work with line breaking

Using a slightly modified example from the documentation (with an added newline):
+ knock knock
- Who's \n
^ there?

+ *
% who * there
- <star> who?

+ *
% * who
- LOL! <star>! That's funny!

If I now follow the script, the final statement (LOL ...) is never reached. I suspect it is probably due to how the regex matching is being done here https://github.com/aichaos/rivescript-python/blob/master/rivescript/brain.py#L503. Adding the re.DOTALL flag would most likely fix this issue, but not sure if that is the correct answer here (I'm relatively new to rivescript). Any thoughts?

Assuming there is no substitution to convert from who's to who is, this script doesn't work on my computer. When I keep the script into the explicit form who is,
+ knock knock
- Who is \n
^ there?

+ *
% who * there
- <star> who?

+ *
% * who
- LOL! <star>! That's funny!

The flow of this discussion works again.
Type /quit when you're done to exit this example.

You> knock knock
Bot> Who is
there?
You> orange
Bot> orange who?
You> banana
Bot> LOL! banana! That's funny!
You>

@kirsle  As I can't reproduce the bug, can you please confirm if there is still a bug here?


Canvas on Desktop?

Description of the new feature / enhancement
Is it possible to get Canvas on desktop somehow? I know there's a Spicetify theme that supports it and there's the online API version. Might work well with fullscreen mode or with lyrics
📸 Screenshots
No response

If there is a ready-made solution for Spicetify, then why don't you want to install it? SpotX works great together with Spicetify

The last time I used Spicetify it messed with my username and messed it up a lot (I had to contact support to get it fixed), but maybe that's not an issue anymore. The other thing was the process of installing it

At the moment, the only way is to install this Spicetify theme, I agree that the installation process is not easy there, but there is a detailed description.
If you do not want to risk your account, then create a test one for testing.
And this site canvastify.delitefully.com provides api, but you can't use it, you need to create your own server, it's just shown for informational purposes, well, the restrictions are only for premium users.


Can't build working package on linux

grunt babel produces ./build folder, this folder is npm link-ed to another project after start it produces error
var _ref = _asyncToGenerator( /#PURE/regeneratorRuntime.mark(function _callee(fileName) {
^
ReferenceError: regeneratorRuntime is not defined
In general I'm triyng to get rid of webdriverio , your tool for taking snapshots and comparing them is awesome but webdriverio is a bit clumsy with browser interactions

@ipeshev this project is deprecated and doesn't work with the latest webdriverio version anymore. Please use wdio-screenshot and the wdio-visual-regression-service. Thanks!


How do I force a version and stop automatic upgrades?

Describe the bug
I'm stuck on issue #873 and would like to revert to 3.5.3 until a fix is available. Can you provide previous versions via alternate snap channel or instructions on how to keep lens from auto updating on me?

I had lens installed as a snap package and it updated this morning.
There doesn't appear to be a 3.5 series snap channel, so I can't revert back :(
So I switched to the 3.5.3 appimage, and it auto-upgraded me to 3.6.0 (╯°□°)╯︵ ┻━┻

Environment (please complete the following information):

Lens Version: 3.5.3/3.6.0
OS: Linux - Pop!_OS 20.04 LTS
Installation method (e.g. snap or AppImage in Linux): Both

Fixed by #1941


E2E: go tag main SLSA3 adversarial asset binary

Repo: https://github.com/slsa-framework/example-package/tree/v12.0.33
Run: https://github.com/slsa-framework/example-package/actions/runs/2463092378
Workflow name: go tag main SLSA3 adversarial asset binary
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.go.tag.main.adversarial-asset-binary.slsa3.yml
Trigger: push
Branch: v12.0.33
Date: Wed Jun  8 17:02:48 UTC 2022

Repo: https://github.com/slsa-framework/example-package/tree/v12.0.34
Run: https://github.com/slsa-framework/example-package/actions/runs/2466082802
Workflow name: go tag main SLSA3 adversarial asset binary
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.go.tag.main.adversarial-asset-binary.slsa3.yml
Trigger: push
Branch: v12.0.34
Date: Thu Jun  9 05:11:40 UTC 2022

Repo: https://github.com/slsa-framework/example-package/tree/v12.0.35
Run: https://github.com/slsa-framework/example-package/actions/runs/2470240372
Workflow name: go tag main SLSA3 adversarial asset binary
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.go.tag.main.adversarial-asset-binary.slsa3.yml
Trigger: push
Branch: v12.0.35
Date: Thu Jun  9 18:06:40 UTC 2022

Repo: https://github.com/slsa-framework/example-package/tree/v12.0.36
Run: https://github.com/slsa-framework/example-package/actions/runs/2472846195
Workflow name: go tag main SLSA3 adversarial asset binary
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.go.tag.main.adversarial-asset-binary.slsa3.yml
Trigger: push
Branch: v12.0.36
Date: Fri Jun 10 05:06:28 UTC 2022
Tests are passing now. Closing this issue.


Final Review

Hey Jordan, great job on pushing through some difficult things and figuring it out!
I'm not sure I have much code-specific feedback since we worked together on a bunch of stuff. But I'll try to mention a couple things.
I already mentioned this to you but I think if you can try to get small things working at once and keep your app compiling, it will be easier to stay out of trouble in terms of compiler errors. Sometimes that's hard but try to do things one at a time and fix errors as you go.
On git usage:

it’s great to descriptive commit messages, especially when you’re working with others, but I find it useful for myself too.
try to commit early and often during the development process. Whenever you have something working it’s a good chance to commit, and that can get you out of trouble sometimes if things get messy. You can just reset your changes and get back to a clean state.
remember to ignore .DS_Store files in the future. There’s a note about that here.

Hang in there and keep at it! 😄
It was great having you in class! I hope you enjoyed it and found it useful. Have a great break!

Right?!?
https://github.com/jordanaevanz/ATLAS-Personal-App/blob/Develop/personal_project/API.swift#L70


Add specs to request variables branch

@jaredbeck
Added specs that I asked you to add in the original PR, let me know what you think and if I can merge it into your branch.

This looks great, Alex, thanks!
I don't normally bother testing delegation of deprecated methods, but I'm happy to have the tests. Later this year, when we remove the deprecated methods, the test will be easy to remove.
I also usually avoid shared_examples, because they are usually in separate files, which makes them harder to follow, but you have used it in the same file and it is relatively self-contained.
However, these are very minor points, and I really appreciate you adding these tests, thanks.

Sure thing!


merge warnings and errors

can we merge warnings and errors for pygrambank check?
value without source and  "comment contains string ""check""" are warnings, the others are errors. Warnings spit out the Feature_ID separate from the message, errors merge them.  For me keeping up with quality-control, these can all be treated the same.
I think we should treat these as the same, both as errors please.
For the CLDF-release, I'd prefer to suppress all data-points that trigger an error, save the general sheet error "Less than 25 datapoints have comments"

@johenglisch said this over here: https://github.com/glottobank/Grambank/issues/2722

I think the WARNING label still makes sense for making some of the lowest-priority checks sound appropriately low-priority (‘comment contains string "check"‘, ‘less than 25 comments‘).
But yeah, we can elevate some of the warnings to errors if you feel people should always address them.
EDIT: Here's a list of all warnings from the check script:

‘value without source’
‘source given, but no value’
‘comment contains string "check"’
‘comment given, but no value’
‘non-empty cell with empty header’
‘less than 25 datapoints have comments’

I think that all of should be resolved before merging new sheets and that the following should mean suppression for cldfbench:

‘value without source’
‘source given, but no value’
‘comment contains string "check"’
‘comment given, but no value’

‘comment contains string "check"’

Sure about that one? Because iirc this was just a hint that a data point might be worth checking (also, that hint is bound to have a few false positives, since there are numerous reasons why one would have the word check in a comment).
The other three were already used to filter data points and I only had to change the prefix in the logging.

I have been thinking about this. I've decided that when a comment with this warning is reviewed we should change the comment so that it reflects the situation and no longer contains the word "check". I think it's not good to have these type of comments, and there are only a handful of them left so I think we should surpress them.

okay, I changed the 'check' thing to an error

okay, I changed the 'check' thing to an error

🙏🙏🙏


langgraph[patch]: ToolNode support for tools outputting msg

After https://github.com/langchain-ai/langchain/pull/24038, tools will be able to take in typed ToolCalls directly and will output a ToolMessage when invoked with a ToolCall
Requires langchain-core==0.2.16 to be released

So tools always return a tool message in this formulation? I must have missed where we guarantee this; it' sin the TOolCall itself?

So tools always return a tool message in this formulation? I must have missed where we guarantee this; it' sin the TOolCall itself?

tools always return tool message if input is a ToolCall with "type" and "id" set


chore(main): release configs 0.1.5

:robot: I have created a release beep boop
0.1.5 (2024-05-08)
Miscellaneous Chores

main: release iso-base 0.1.5 (3849a49)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/hugomrdias/iso-repo/releases/tag/configs-v0.1.5 :sunflower:


Tixati download link does not install anything

Describe the bug
When initiating the installer from Tixati the PowerShell begins the process and claims it succesfully installed said program, but when searching all over my drive for the program nothing show up (even when manually searching in the file explorer).
The manual installer from their website does still work.
To Reproduce

start utility
choose tixati checkmark
click "install/upgrade"
watch powershell claiming a job well done
try and start the alleged installed software you still don't have installed.

Expected behavior
That said installer actually installs said software and being able to use the software after said install process.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
This is a highly LOW PRIORITY issue, mostly wondered if it is an utility issue or winget issue.

Hello @tibrusjer
Thanks for reporting this, and the reason behind not finding the installed program after installing it, in both WinGet (the package manager) and WinUtil.. because it's the portable edition of Tixati,  meaning it's downloaded in %LOCALAPPDATA%\Microsoft\WinGet\Packages folder (either this foldor or somewhere else, can't remember exactly where it installs portable apps), in other words, you won't find Tixati by searching it up.
So this is a WinGet thing, not WinUtil. When I added Tixati package to winget-pkgs repo, I chose the Portable Edition of it, not installer (because the installer would not Silently install. Therefore, it failed WinGet testing pipeline)
You can run it Tixati by typing tixati_Windows64bit or tixati_Windows32bit in your terminal (depending on your hardware.. but it'll most likely be 64 bit).
I hope this answers your question. If you have anything else you're wondering about (and it's related to this issue ticket), you post it and I'll reply as soon as I can.


fs/munmap: export the symbols to avoid build break on C++ syntax

Summary
fs/munmap: export the symbols to avoid build break on C++ syntax
base/metrics/persistent_memory_allocator.cc:1009:18: error: expected id-expression before ‘-’ token
1009 |   int result = ::munmap(memory, size);
|                  ^~~~~~
Change-Id: I24e55c18d09553d3acc341a27063d01489d905ec
Signed-off-by: chao.an<EMAIL_ADDRESS>

Depends on https://github.com/apache/incubator-nuttx-apps/pull/389


fix(macos): persist destination folder access across app launches

closes https://github.com/localsend/localsend/issues/1971

Nice!


can't host on vps

hi there it seems like i can't run this bot on my vps
i followed that what's bot guide provided but replaced what'sbot with this bot instead
it worked but at the end when i try to lonch it using npm start it throughs an syntax error
running on ubuntu22.04 on s390x platform fresh install
i will attach the error log bellow
error.log

I cant see any error in the file you attached! Seems Issue is with your session. Do run npm run genToken to generate a session file.

hey there. sorry, i did'nt capture the log properly here is the proper one
log.txt


feat: add ArtifactHub Metadata

The Following PR will add the necessary information to enrich the look and feel of the Kepler repo on ArtifactHub.
Things Changed:

I have added more metadata to the chart.yaml file which will render in ArtifactHub as per their standard
Bumped the version of chart to reflect these changes in ArtifactHub
Added the Artifact badge to the readme of the repo so you can click it
Added a README.md to the template folder which will render in ArtifactHub for the repo homepage for good experience for users/adopters to see values they can use

May I ask why did you create a README.md inside the templates folder?
You can just append that info the same file in charts/kepler.

Hey @yellowhat yes thanks I had it one directory to low, i have added it back up to charts/kepler where it is meant to be..

thank you @bradmccoydev

No problem, it is looking really good: https://artifacthub.io/packages/helm/kepler/kepler

very cool! @bradmccoydev


unable to use themes in java

I have a PDF generated from an ascii through asciidoctor i need to make the PDF in a presentable way like setting the background color for the code block and changing the fonts for a product.I am not seeing any doc for java.

Please see https://github.com/asciidoctor/asciidoctor-maven-examples/tree/master/asciidoctor-pdf-with-theme-example.

Its not applying can you provide a code snippet to add to options map or how to include in java??

You just need to specify the pdf-style attribute to the file path of a YAML theme file.
import java.io.File;
import org.asciidoctor.Asciidoctor;
import org.asciidoctor.Attributes;
import org.asciidoctor.Options;
import static org.asciidoctor.AttributesBuilder.attributes;
import static org.asciidoctor.OptionsBuilder.options;

public class Test {
    public static void main(String[] args) {
        Asciidoctor asciidoctor = Asciidoctor.Factory.create();
        Attributes attributes = attributes().attribute("pdf-style", "pdf-theme.yml").get();
        Options options = options().backend("pdf").attributes(attributes).get();
        asciidoctor.convertFile(new File(args[0]), options);
    }
}

And is it possible to load a .css theme file instead of .yml??

And when use the attributes am getting attributes() is not defined and am using
	<dependency>
		<groupId>org.asciidoctor</groupId>
		<artifactId>asciidoctorj-pdf</artifactId>
		<version>1.5.0-alpha.9</version>
	</dependency>
	<dependency>
		<groupId>org.asciidoctor</groupId>
		<artifactId>asciidoctorj</artifactId>
		<version>1.5.2</version>
	</dependency>

It is not possible to use CSS with Asciidoctor PDF. That's not the design goal of the library. We are writing directly into the PDF object and CSS is not available there. The yml file is our attempt to provide a familiar environment for a web developer, but specifically designed for styling PDF.

is it possible to print the JSON instead of Table ??

@mojavelinux is there any provision for displaying JSON Schema in its own format inistead of showing them as Tables??

I don't understand the question you are asking. Could you clarify or perhaps provide an example?

https://github.com/rathishkumar89/swaggerAsciiTest this is the example am working i am pulling the JSON over HTTP and converting it into ascii https://github.com/rathishkumar89/swaggerAsciiTest/blob/master/ascii-to-pdf/src/main/java/com/ascii/asciidoc/swaggerstaticdoc/core/FrameJson.java Lno :85 am taking the schema and even without the schema option also the result is ended up with table structure.But, the expected is from ASCII doc if i convert to PDF it should end up with JSON Structure .May i know the syntax or any switch we need to include at time of PDF Creation?

This seems to me like a swagger2markup question. Asciidoctor doesn't do anything with JSON schemas. It just converts the AsciiDoc it is given.

The java examples above seem to be addressing the same need that I'm trying to address in ruby. I'm guessing my syntax is incorrect?
I have the following code that is trying to reference a theme folder,  yaml file within the theme folder and also a fonts folder:
Asciidoctor.convert_file adoc_file, :backend => 'pdf', :in_place => true, :pdf_stylesdir => './public/resources/themes', :pdf_style => './public/enqabile-theme.yml', :fonts_dir => './public/resources/fonts' # creates a pdf file
Any tips will be appreciated

First, you need to use attributes, not options, to specify the location of the PDF theme and resources. Second, the pdf-style is relative to the pdf-stylesdir. In other words, here's what you need to use:
Asciidoctor.convert_file adoc_file,
  backend: 'pdf',
  attributes: {
    'pdf-stylesdir' => 'public/resources/themes',
    'pdf-style' => 'enqabile',
    'pdf-fontsdir' => 'public/resources/fonts'
  }

See https://github.com/asciidoctor/asciidoctor-pdf/blob/master/docs/theming-guide.adoc#applying-your-theme for more details.

Thanks Dan, that did the trick!

I'm glad to hear it!


Can't recognize ISBN barcode

Cannot scan barcode with ISBN format,
It can be scanned with qr_code_scanner, but this library cannot,
Some sample codes I attach below

If I test this with my Android device, I can scan all 3 images properly. What device are you testing this on?

If I test this with my Android device, I can scan all 3 images properly. What device are you testing this on?

@navaronbracke We test on the iPhone device

@hoangduchuu I tested this on mobile_scanner 5.2.3 and it does detect these barcodes also on iPhone.
I do see the following, though:
5060198641183 is detected as BarcodeType.product
9780760378045 is detected as BarcodeType.isbn
9780593076118 is detected as BarcodeType.isbn
So MLKit does not detect that first one as an ISBN barcode.

I was having this same kind of problem after inserting this:
` @override
void initState() {
super.initState();
const scriptUrl = 'https://unpkg.com/@zxing/library@latest';
if (kIsWeb) {
  MobileScannerPlatform.instance.setBarcodeLibraryScriptUrl(scriptUrl);
}

}
}`
it worked perfectly


Adjust logging to have Standard Apache common log output.

I worked on Issue #27 and changed from dev to apache common logging format. Now color is gone and logs shouldn't cause problems.

I see I just thought coloring would be the issue. Will adjust it

Thanks 😄

@pmanlukas would we get timestamp for each log entry with this change?


LHN - RBR appears on the wrong workspace chat for an error occurring on another workspace

Details

Fixed Issues

$ https://github.com/Expensify/App/issues/47874
PROPOSAL: https://github.com/Expensify/App/issues/47874#issuecomment-2325602648
Tests

Navigate to Old Dot and sign up with a new account
Go to settings > Account and validate the account
Navigate to staging.new.expensify.com and sign in with the same account signed in on Old Dot
Create a workspace and name it as "Workspace 1"
Go to workspace editor and enable workflows
Go to workflows and enable delayed submission and set it to manual and enable approvals also
Create another workspace and name it as "Workspace 2"
Navigate to Workspace 1 chat and create a manual expense
Click on submit but do not click on approve
Navigate to Old dot and refresh the page
Go to reports and click on the expense created on ND
Click on detail and change the workspace from workspace 1 to workspace 2
Go to New dot and wait for a few seconds for the change to take effect
After the changes took effect (Workspace 1 will not have GBR on the LHN and instead workspace 2 will have GBR and the expense will be moved to it)
Go to workspace chat 2 and create a scan expense with a normal picture so that it will throw error after it is done scanning
Observe the LHN of workspace chat 1 on the LHN
No RBR will be present on the workspace chat 1

[x] Verify that no errors appear in the JS console

Offline tests

NA
QA Steps

Same as Tests above

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos
https://github.com/user-attachments/assets/078f7eeb-35ae-4069-85f3-67cd6726d572

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari

MacOS: Desktop

I have read the CLA Document and I hereby sign the CLA

recheck

Hi @ntdiary,
Thank you for reviewing my PR. I wanted to ask for some guidance on how to test the changes across all six platforms (Android Native, Android mWeb Chrome, iOS Native, iOS mWeb Safari, MacOS Chrome/Safari, and MacOS Desktop). Currently, I'm using Linux for development, and as this is my first PR, I'm not entirely sure how to test on all these platforms.
Could you please advise on the best approach for this? Any help or resources you can provide would be greatly appreciated.
Thank you!

https://github.com/Expensify/App/blob/main/contributingGuides/TESTING_MACOS_AND_IOS.md Perhaps this document can help you. As for Android mWeb/native, you can use Android Studio and its emulator or your real device. Additionally, we also have a guide to help set up the local environment https://github.com/Expensify/App?tab=readme-ov-file#local-development 😄

@ntdiary Hi
Thank you for providing the resources. I’ve started working on setting up the environment and will let you know if I encounter any issues. Your assistance is much appreciated!

@ntdiary
Working well on the web.
The Android build in Android Studio completes successfully, but after the "Build Successful" message, the app doesn't fully launch on the emulator and stops at that point. It appears to be an issue with loading Scripts.

@ntdiary
Yes, I have executed both npx react-native start and adb reverse tcp:8081 tcp:8081, but the issue persists. Additionally, the web is running correctly on localhost:9000, while attempting to use 8081 or 8082 results in a security error.

@ntdiary
The application is now running successfully on https://<IP_ADDRESS>:8082 within the Android emulator. I've also attached a video in the "Android: mWeb Chrome" section for your reference.
Could you please confirm if this meets the requirements?
video

@SIMalik, LGTM! And if the video is too large, you can also upload screenshots like I did. 😄

@ntdiary

@ntdiary
Yes, I am currently in the process of setting up macOS from scratch.

Hi @ntdiary,
I initially installed macOS 10.15 Catalina, but I encountered several unsupported command issues, which affected its performance. Realizing it was an outdated version, I decided to remove it.
Currently, I am in the process of installing macOS 15 Sequoia. Since this is my first time installing macOS, it’s taking some time, but I'm making progress and am not stuck.
Thank you for your understanding!

Hi @ntdiary
MacOS: Chrome / Safari
After Code change:

Before code change:

Working on other platforms.

Hi @ntdiary
macOS: Ventura 13.7
iOS Setup: I’ve completed the following steps:
Step 1: Install Xcode 14.3.1
Step 2: Install Xcode Command Line Tools
xcode-select --install
Step 3: Set Up CocoaPods
sudo gem install bundler
bundle install
Step 4: Configure Mapbox Token
npm run configure-mapbox
Step 5: Install Project Dependencies
npm install && npm run pod-install
Step 6: Run the iOS Simulator
npm run ios
Issue: When I open Expensify/ios/NewExpensify.xcworkspace in Xcode, I encounter the following error message:
"The target 'NewExpensifyTests' contains source code developed with Swift 3.x. This version of Xcode does not support building or migrating Swift 3.x targets."
Could you please advise on how to proceed or correct any steps to successfully run the iOS app?
Thank you!

Hi @ntdiary,
Yes, I'm currently testing on the other platforms and will ensure everything is completed. If I encounter any issues, I’ll be sure to let you know.
Thank you for your support!

@SIMalik looks like the automated tests found an error that will need to be fixed. We might need to re-evaluate the approach.

@neil-marcellini
You're absolutely right; I’m currently working on addressing the failing tests and will make the necessary adjustments.

@ntdiary, @neil-marcellini
getRBRReportAction
  ✓ returns report action that contains errors (297 ms)
  ○ skipped returns undefined when report has no RBR
  ○ skipped returns parentReportAction if it is a transaction thread, the transaction is missing smart scan fields and the report is not settled
  Report has missing fields, isn't settled and it's owner is the current user
    Report is IOU
      ○ skipped returns correct report action which has missing fields
    Report is expense
      ○ skipped returns correct report action which has missing fields
  There is a report action with smart scan errors
    ○ skipped returns correct report action which is a report preview and has an error
    ○ skipped returns correct report action which is a split bill and has an error
    ○ skipped returns undefined if there's no report action is a report preview or a split bill

Test Suites: 1 passed, 1 total
Tests:       174 skipped, 1 passed, 175 total
Snapshots:   0 total
Time:        8.196 s
Ran all test suites matching /tests/unit/DebugUtilsTest.ts/i with tests matching "returns report action that contains errors".

@SIMalik I would try merging main into this PR. This merge is failing.
HEAD is now at a26cfb45889 Merge 38c543159587474035ee77f06f8b7b34f1202d6e into e4969b22e7dbb1f6a200cc587ced4ab665415c0e

@SIMalik I would try merging main into this PR. This merge is failing.
HEAD is now at a26cfb45889 Merge 38c543159587474035ee77f06f8b7b34f1202d6e into e4969b22e7dbb1f6a200cc587ced4ab665415c0e

Hi @neil-marcellini
I went ahead and merged the main branch into the feature branch (issue-47874). The merge completed successfully with no conflicts. I have also pushed the updated branch. The latest commit hash is 30f43f214da, and the branch is now fully up to date with main.
Please let me know if there’s anything else I should address or if any further steps are needed on this.
Thanks,

@SIMalik looks like there are conflicts again

@neil-marcellini @ntdiary
Conflicts removed.
After code change:

Before code change:

@SIMalik, this message?

Don't worry, just click "OK" to ignore it. I see it every time I open the project in Xcode. :) Have there been any other fatal errors during compilation?

@ntdiary
macOS: 13.7
Xcode: 14.3.1
Swift: 5
I have facing following errors during build.

The changed files check is failing but none of those files were changed in this PR, so they should be handled elsewhere.

@neil-marcellini @ntdiary
After deployment on staging


adds doc for repo publishing

The Problem:
drud/general#33 OP - Add guidelines and preparation steps for publishing repositories to the public org space.
The Fix:
The Test:
Automation Overview:

Related Issue Link(s):
drud/general#33 OP
Release/Deployment notes:

I've added the suggestion under a heading for retiring public repos, with the slight change to say "If a repository needs to be pulled back private or archived"

Two approves. Merged!


Incorrect interpretation of -nagle option

Setting -nagle option to 1 sets the TCP_NODELAY flag.
Shouldn't that be the opposite? In fact, setting TCP_NODELAY disable the Nagle algorithm.

But of course you're right. Will fix in next release.

Fixed in 1.1.0


Dev

This is release of v1.0.2
See ChangeLog
Not sure why yet, but Pester tests in v5.0.2 and v5.0.3 fail on my local machine (win10 w/ PWSH) but pass in AzurePipeline. 🤷♂🤦♂
Closing #1

Well, I forgot that AzurePipelines would auto-test PR's. That's cool. 👍
This version passes Pester in my pipeline. Should research the differences with the pipeline environment and my machine so I can test accurately BEFORE PR's.


Move Style to Synless

https://github.com/justinpombrio/partial-pretty-printer/blob/1e8f7ded83895fa7ebf7d71648afe6dde2405aa0/src/style.rs#L1C4-L1C4

Removed from PPP. Migrated to Synless issue, b.c. Synless still needs to add styles.


ModSecurity: StatusEngine call failed

Hi,
I am getting this kind of error message in my /var/log/apache2/error.log:
...
[Fri Apr 15 18:32:16.151541 2022] [mpm_event:notice] [pid 3265:tid<PHONE_NUMBER>] AH00489: Apache/2.4.53 (Raspbian) OpenSSL/1.1.1n
[Fri Apr 15 18:32:16.151867 2022] [core:notice] [pid 3265:tid<PHONE_NUMBER>] AH00094: Command line: '/usr/sbin/apache2'
[Fri Apr 15 18:37:59.449854 2022] [mpm_event:notice] [pid 3265:tid<PHONE_NUMBER>] AH00492: caught SIGWINCH, shutting down gracefully
[Fri Apr 15 18:38:00.473588 2022] [:notice] [pid 3803:tid<PHONE_NUMBER>] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
[Fri Apr 15 18:38:00.476451 2022] [:notice] [pid 3803:tid<PHONE_NUMBER>] ModSecurity: APR compiled version="1.7.0"; loaded version="1.7.0"
[Fri Apr 15 18:38:00.476542 2022] [:notice] [pid 3803:tid<PHONE_NUMBER>] ModSecurity: PCRE compiled version="8.39 "; loaded version="8.39 2016-06-14"
[Fri Apr 15 18:38:00.476580 2022] [:notice] [pid 3803:tid<PHONE_NUMBER>] ModSecurity: LUA compiled version="Lua 5.1"
[Fri Apr 15 18:38:00.476609 2022] [:notice] [pid 3803:tid<PHONE_NUMBER>] ModSecurity: YAJL compiled version="2.1.0"
[Fri Apr 15 18:38:00.476638 2022] [:notice] [pid 3803:tid<PHONE_NUMBER>] ModSecurity: LIBXML compiled version="2.9.10"
[Fri Apr 15 18:38:00.476668 2022] [:notice] [pid 3803:tid<PHONE_NUMBER>] ModSecurity: Original server signature: Apache
[Fri Apr 15 18:38:00.476935 2022] [:notice] [pid 3803:tid<PHONE_NUMBER>] ModSecurity: StatusEngine call: "2.9.3,Apache,1.7.0/1.7.0,8.39/8.39 2016-06-14,Lua 5.1,2.9.10,90"
[Fri Apr 15 18:38:05.487574 2022] [:notice] [pid 3803:tid<PHONE_NUMBER>] ModSecurity: StatusEngine call failed. Query: GIXDSSDFDSFSDFDSFSDFSDFSDFSDFSDFSDFSDFSDFLZR.FY3S4MBMSDFSDFSDFSDFSDFSDFSDfRN.GA3C2MJsdfsdfsdfsdfsdfsdfsdfsdfLBSFY4S4MJQ.FQ4TA.1650040680.status.modsecurity.org
...
What am I missing here or what needs to be changed so that StatusEngine call does NOT fail?

Hi @un99known99 ,
The status engine sends limited information about the deployment (e.g. which ModSecurity version are you using) to our server.
The name resolution is currently happening correctly; I've raised that with a  colleague.
The current failure should not cause any functionality problems beyond the message in error.log at startup. You could opt to just ignore it until the DNS issue is resolved.
If that is not a palatable option for you, could opt to turn off the status engine: 'SecStatusEngine Off'

The setting has been updated to 'Off' in modsecurity.conf-recommended as of https://github.com/SpiderLabs/ModSecurity/commit/733427197e2fe4fabcbb0f43bd1e636ef923a6b4

@martinhsv many thanks!


TxObjectList Select Do Not Reset between uses

Hi,
It seems that  "TXObjectList.Where(...).Select"  clause doesn`t clean its result array if you try to use it more than once, which causes unexpected results.
I attach a test case as close to the real scenario as I can.
Thanks in advance,
_20210609_TxObjectList_Select_Do_Not_Reset.zip

Thank's for you report. Now it's fixed.

You are wellcome!!!