Issues converting recurrent ML-Agents model to TFLite (Tensorflow 1.15)

Hi
I have been tryingt to convert a frozen graph def created by ML-Agents to TFLite. However, there is an input variable to the model, which is defined by:
self.sequence_length = tf.placeholder(
            shape=None, dtype=tf.int32, name="sequence_length"
        )

which seems to cause problems in the conversion.
If I convert without passing shapes the following happens:
tflite_convert --output_file=model.tflite --graph_def_file=unity/ml-agents/models/ppo-0/RoverLearning/frozen_graph_def.pb --input_arrays=vector_observation,epsilon,recurrent_in,sequence_length --output_arrays=action

2019-11-05 13:59:21.589574: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-05 13:59:21.602134: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe2bf9e7280 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2019-11-05 13:59:21.602151: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
Traceback (most recent call last):
  File "/Users/dwright/anaconda3/envs/mlagents/bin/tflite_convert", line 8, in <module>
    sys.exit(main())
  File "/Users/dwright/anaconda3/envs/mlagents/lib/python3.6/site-packages/tensorflow_core/lite/python/tflite_convert.py", line 515, in main
    app.run(main=run_main, argv=sys.argv[:1])
  File "/Users/dwright/anaconda3/envs/mlagents/lib/python3.6/site-packages/tensorflow_core/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/Users/dwright/anaconda3/envs/mlagents/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/Users/dwright/anaconda3/envs/mlagents/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "/Users/dwright/anaconda3/envs/mlagents/lib/python3.6/site-packages/tensorflow_core/lite/python/tflite_convert.py", line 511, in run_main
    _convert_tf1_model(tflite_flags)
  File "/Users/dwright/anaconda3/envs/mlagents/lib/python3.6/site-packages/tensorflow_core/lite/python/tflite_convert.py", line 199, in _convert_tf1_model
    output_data = converter.convert()
  File "/Users/dwright/anaconda3/envs/mlagents/lib/python3.6/site-packages/tensorflow_core/lite/python/lite.py", line 898, in convert
    self._set_batch_size(batch_size=1)
  File "/Users/dwright/anaconda3/envs/mlagents/lib/python3.6/site-packages/tensorflow_core/lite/python/lite.py", line 1032, in _set_batch_size
    shape = tensor.shape.as_list()
  File "/Users/dwright/anaconda3/envs/mlagents/lib/python3.6/site-packages/tensorflow_core/python/framework/tensor_shape.py", line 1171, in as_list
    raise ValueError("as_list() is not defined on an unknown TensorShape.")
ValueError: as_list() is not defined on an unknown TensorShape.

I added some debut messages to lite.py and found that the issue is with the variable sequence_length
If I pass the shapes to the command like this:
tflite_convert --output_file=model.tflite --graph_def_file=unity/ml-agents/models/ppo-0/RoverLearning/frozen_graph_def.pb --input_arrays=vector_observation,epsilon,recurrent_in,sequence_length --output_arrays=action --input_shapes=1,7:1,2:7,40:1

I get a similar error:
2019-11-05 14:25:44.658772: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-05 14:25:44.677569: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fee79ed3960 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2019-11-05 14:25:44.677591: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2019-11-05 14:25:44.741840: I tensorflow/core/grappler/devices.cc:60] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA support)
2019-11-05 14:25:44.741948: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2019-11-05 14:25:44.819310: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:786] Optimization results for grappler item: graph_to_optimize
2019-11-05 14:25:44.819337: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788]   constant_folding: Graph size after: 304 nodes (-81), 371 edges (-83), time = 56.212ms.
2019-11-05 14:25:44.819342: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788]   constant_folding: Graph size after: 304 nodes (0), 371 edges (0), time = 8.713ms.
 Input "sequence_length" is different.
Fatal Python error: Aborted

Current thread 0x0000000111d5b5c0 (most recent call first):
  File "/Users/dwright/anaconda3/envs/mlagents/lib/python3.6/site-packages/tensorflow_core/lite/toco/python/toco_from_protos.py", line 52 in execute
  File "/Users/dwright/anaconda3/envs/mlagents/lib/python3.6/site-packages/absl/app.py", line 250 in _run_main
  File "/Users/dwright/anaconda3/envs/mlagents/lib/python3.6/site-packages/absl/app.py", line 299 in run
  File "/Users/dwright/anaconda3/envs/mlagents/lib/python3.6/site-packages/tensorflow_core/python/platform/app.py", line 40 in run
  File "/Users/dwright/anaconda3/envs/mlagents/lib/python3.6/site-packages/tensorflow_core/lite/toco/python/toco_from_protos.py", line 89 in main
  File "/Users/dwright/anaconda3/envs/mlagents/bin/toco_from_protos", line 8 in <module>

Any help is highly appreciated!

Hi
I'm exploring the same rabbit hole (ml-agents -> tflite); currently at about the same point.
Here are a few learnings that may help:

try the python api instead of command line util (it's recommended): https://www.tensorflow.org/lite/convert/python_api
if you happen to be on windows, the windows distrib of tensorflow seems to be missing lite or just the converter, but seems to work fine running it in the linux sub system (e.g. ubuntu): https://docs.microsoft.com/en-us/windows/wsl/install-win10.
perhaps try the older version of tf lite converter api, that matches the tf version used by ml-agents (1.7): https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/r1/convert/python_api.md#additional-instructions
maybe try reporting this in the ml-agents repo instead (as ml-agents is well aware of tensorflow, but likely less so vice versa)

If you get any further pls let me know - I'll do the same!
Cheers
Matt

Ha, just found your post in ml-agents repo!

Note https://www.tensorflow.org/lite/convert/rnn#not_currently_supported which mentions RNN usage of sequence_length is not supported, but will be by end of 2019...
Maybe it is supported in previous version of converter.
My case isn't an RNN btw, so sounds like your rabbit hole might be a bit longer.

Wow, thanks @kaploink! I will look into those suggestions and keep you up to date! Could you tell me what your set up and objective is, just for information?

@wrd90
I'm making a low cost quadruped robot running on arduino or raspberry pi (tbd), and using unity's ml-agents to teach it to walk.
I've got the physical prototype robot and the corresponding trained ml-agents model, now at the pointy end of trying to put them together...
Currently at the conversion to tflite stage.
How did you find out what the input arrays/shape should be...?

That sounds cool @mp-airwallex!
I found the shapes by visualizing frozen_graph_def.pb using tensorboard

err, oops, wrong account!  kaploink == mp-airwallex
ah, awesome, thanks

Note https://www.tensorflow.org/lite/convert/rnn#not_currently_supported which mentions RNN usage of sequence_length is not supported, but will be by end of 2019...
Maybe it is supported in previous version of converter.
My case isn't an RNN btw, so sounds like your rabbit hole might be a bit longer.

I have tried to use the drop-in replacements suggested in your link but it looks like that doesn't work with the version used by ml-agents...
  File "/Users/dwright/anaconda3/envs/mlagents/bin/mlagents-learn", line 11, in <module>
    load_entry_point('mlagents', 'console_scripts', 'mlagents-learn')()
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/learn.py", line 417, in main
    run_training(0, run_seed, options, Queue())
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/learn.py", line 233, in run_training
    options.multi_gpu,
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/trainer_util.py", line 91, in initialize_trainers
    multi_gpu,
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/ppo/trainer.py", line 75, in __init__
    seed, brain, trainer_parameters, self.is_training, load
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/ppo/policy.py", line 47, in __init__
    brain, trainer_params, reward_signal_configs, is_training, load, seed
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/ppo/policy.py", line 97, in create_model
    trainer_params.get("vis_encode_type", "simple")
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/ppo/models.py", line 52, in __init__
    self.create_cc_actor_critic(h_size, num_layers, vis_encode_type)
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/ppo/models.py", line 91, in create_cc_actor_critic
    name="lstm_policy",
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/models.py", line 577, in create_recurrent_encoder
    rnn_cell, lstm_input_state, initial_state=lstm_vector_in
  File "/Users/dwright/anaconda3/envs/mlagents/lib/python3.6/site-packages/tensorflow_core/lite/experimental/examples/lstm/rnn.py", line 168, in dynamic_rnn
    raise RuntimeError("OpHint dynamic rnn only supports control flow v2.")
RuntimeError: OpHint dynamic rnn only supports control flow v2.

@kaploink I almost figured ;)
I tried to replace the LSTMCell with the drop-in replacements suggested in your link, but there seems to be a problem which sounds much like the initial error message in the initial conversion:
Sorting model, may take a while... Done!
IGNORED: Cast unknown layer
IGNORED: AddV2 unknown layer
IGNORED: Shape unknown layer
IGNORED: TensorListReserve unknown layer
IGNORED: TensorListFromTensor unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: EmptyTensorList unknown layer
IGNORED: While unknown layer
IGNORED: TensorListStack unknown layer
Traceback (most recent call last):
  File "/Users/dwright/anaconda3/envs/mlagents/bin/mlagents-learn", line 11, in <module>
    load_entry_point('mlagents', 'console_scripts', 'mlagents-learn')()
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/learn.py", line 417, in main
    run_training(0, run_seed, options, Queue())
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/learn.py", line 255, in run_training
    tc.start_learning(env)
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/trainer_controller.py", line 219, in start_learning
    self._export_graph()
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/trainer_controller.py", line 129, in _export_graph
    self.trainers[brain_name].export_model()
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/trainer.py", line 152, in export_model
    self.policy.export_model()
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/tf_policy.py", line 230, in export_model
    tf2bc.convert(frozen_graph_def_path, self.model_path + ".nn")
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/tensorflow_to_barracuda.py", line 1552, in convert
    i_model, args
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/tensorflow_to_barracuda.py", line 1377, in process_model
    process_layer(n, o_context, args)
  File "/Users/dwright/MSE/projects/rainSim/unity/ml-agents/ml-agents/mlagents/trainers/tensorflow_to_barracuda.py", line 1220, in process_layer
    assert all_elements_equal(input_ranks)
AssertionError

@wrd90
Ah okay, well looks like the tf pros have been adding some labels so sounds like they might have some answers for you soon.
The tflite conversion worked fine for me (non-RNN) using a slightly adjusted version of your command (tflite_convert --output_file=model.tflite --graph_def_file=./frozen_graph_def.pb --input_arrays=vector_observation,epsilon --output_arrays=action), so thanks heaps for that.  I've been able to continue onto tflite inference test (worked :) ), and now chucking it on the pi and hooking up to sensors & motors.  Not bothering with microcontroller version yet - will optimise later.
Cheers

@kaploink Glad to hear you've been able to get some inference results! I have decided to move away from LSTMs for now. Instead I am currently buffering my observations and actions, which I then feed into the network as well. I am hoping this will perform well enough, but so far I have not seen any significant differences in the learning process compared to the LSTM version. Due to time restrictions in my project I had to find a workaround for now. Also I expect the support for RNNs only to be added to TF2, to which ML-Agents will need to migrate first.
Cheers

@dav1d-wright
As many issues have been fixed in the latest version could you please try on the latest stable version [2.4/2.5/2.6] and let us know.


Publish APK Files in GitHub again.

As a user I want to download the APK Files directly from Github (especially in devices where Google Play is not installed)
The APKs are no longer available for the newer releases.
Please do release the APK Files for Bayern-FLOSS to Github.
Thanks.

@vthejay currently we have to create the releases manually.
We have already an open issue for creating them automatically
https://github.com/digitalfabrik/entitlementcard/issues/1232

Done in https://github.com/digitalfabrik/entitlementcard/pull/1264


Issue #3597 - Avoid bytestring/str hodgepodge when resolving cookie from Request URL

When resolving the request full URL, the Host cookie is used to override the URL's host, in Python 3 this causes a mix of str and bytestring objects when the Host cookie is set to a bytestring. Patch checks that ParsedResult is str type and host is non-str type before attempting to decode bytestring to str.

Patch should address Issue #3597

All appears to be in order 😃
============================= test session starts =============================
platform win32 -- Python 3.5.2, pytest-2.8.7, py-1.4.31, pluggy-0.3.1
rootdir: C:\Users\Brian\PycharmProjects\requests, inifile:
plugins: httpbin-0.0.7, cov-2.2.1, mock-0.11.0
collected 381 items

tests\test_hooks.py ...
tests\test_lowlevel.py .........
tests\test_requests.py .............................................................................................................................................X.........X....................................................
tests\test_structures.py ....................
tests\test_testserver.py ...........
tests\test_utils.py ..s...................................................................................................................................

============== 378 passed, 1 skipped, 2 xpassed in 55.53 seconds ==============

Went through your comments and reworked the module of internal references to to_native_string. 👍


[$500] Search - No username displayed in search for task message

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: v1.4.13-5
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by:
Slack conversation: @
Action Performed:

Open the app and login with user A
Assign a new task to user B with any title
From any other device, login with user B and send message to task created in step 2
From user A, assign task created in step 2 to user C
From any other device, login with user C and send message to task created in step 2
Check in LHN in user A, for message by user C, user C username will be displayed besides the message
Open search and check the task report, it will not have username besides the message as it displayed in LHN

Expected Result:
App should display username besides message for task with multiple members in search as it does in LHN
Actual Result:
App does not display username besides message for task with multiple members in search
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

Screenshots/Videos
https://github.com/Expensify/App/assets/115492554/1304779c-059b-46da-9e08-a4d4ea69efc4
https://github.com/Expensify/App/assets/115492554/9e4d101e-4318-49ab-8b92-4cbfa28b6409

View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~0101327f233d3a2169
Upwork Job ID:<PHONE_NUMBER>719885312
Last Price Increase: 2023-12-22
Automatic offers:

shubham1206agra | Contributor | 28070795

Proposal
Please re-state the problem that we are trying to solve in this issue.
What is the root cause of that problem?
What changes do you think we should make in order to solve the problem?
Change to:
result.alternateText = showChatPreviewLine && lastMessageText ? lastMessageText : Localize.translate(preferredLocale, 'report.noActivityYet');

https://github.com/Expensify/App/blob/a24f77b27fce2d79302e2076a32461e595684183/src/libs/OptionsListUtils.js#L531
What alternative solutions did you explore? (Optional)

Proposal
Please re-state the problem that we are trying to solve in this issue.
No username displayed in search for task message
What is the root cause of that problem?
we display last message text for the task report and
the OptionsListUtils is not consistent with the SidebarUtils options
https://github.com/Expensify/App/blob/a24f77b27fce2d79302e2076a32461e595684183/src/libs/OptionsListUtils.js#L420-L422
as in the side bar we have the follwoing conditions  https://github.com/Expensify/App/blob/07d64187afe2cd5f26b912ad8e8a84c5d5b2e83c/src/libs/SidebarUtils.ts#L388-L392
What changes do you think we should make in order to solve the problem?
We need to make both consistent so we need to add this:
       } else if (lastAction?.actionName !== CONST.REPORT.ACTIONS.TYPE.REPORTPREVIEW && lastActorDisplayName && lastMessageTextFromReport) {
            result.alternateText = `${lastActorDisplayName}: ${lastMessageText}`;
        } else {
            result.alternateText = lastMessageTextFromReport.length > 0 ? lastMessageText : Localize.translate(preferredLocale, 'report.noActivityYet');
        }

@eVoloshchak proposals for you here!

bump @eVoloshchak

@NicMendonca I tried bumping in Slack as well – no response, unfortunately. Same situation on other issues 😢
https://expensify.slack.com/archives/C01GTK53T8Q/p1703108470058559

@paultsimura's proposal looks good to me.
🎀👀🎀 C+ reviewed

More stuff related to tasks cc @thienlnam
Proposal looks good. But why do we show the username in the subtitle for tasks? Seems inconsistent to me. So, just double checking that the behavior is correct?

why do we show the username in the subtitle for tasks

We already show this username in LHN, so this task is intended to sync task in Search with the task in LHN.
Just to clarify: this is only for tasks that have multiple participants involved (assigned to A, got a message from them, then reassigned to B, and got a message from them).

@thienlnam @marcaaron What is the status here?

More stuff related to tasks cc @thienlnam

Proposal looks good. But why do we show the username in the subtitle for tasks? Seems inconsistent to me. So, just double checking that the behavior is correct?
Yeah that behavior looks correct - as mentioned from Paul that subtitle only shows up with multiple participants so it should have the same behavior as a group chat

Thanks. The PR is ready for review: https://github.com/Expensify/App/pull/33656

@shubham1206agra can you accept the offer please? Thanks!

@paultsimura you've been paid!

@NicMendonca Accepted

@shubham1206agra don't forget the BZ checklist!

@shubham1206agra you've been paid 👍

@shubham1206agra bump on the BZ checklist!

@shubham1206agra bump!

Oops. Sorry for the delay. I will do this right now.

BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

[x]  [@shubham1206agra] The PR that introduced the bug has been identified. Link to the PR: https://github.com/Expensify/App/pull/22734
[x]  [@shubham1206agra] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment: https://github.com/Expensify/App/pull/22734/files#r1463404196
[x]  [@shubham1206agra] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion: Not required
[x]  [@shubham1206agra] Determine if we should create a regression test for this bug. No
[x]  [@shubham1206agra] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.
[ ]  [@NicMendonca] Link the GH issue for creating/updating the regression test once above steps have been agreed upon:


darwin: Remove uiprivMap & all singleton delegates

This fixes #157 for me, closes #165 and should fix #160 too.
It entirely removes all of the uiprivMap code and it's use as singleton delegates. This is a combined commit incorporating the relevant removal in all affected components.
Replace singleton delegate by subclassing

uiButton
uiCheckbox
uiCombobox
uiDateTimePicker
uiEditableCobobox
uiMenu
uiSlider
uiWindow

Replace singleton delegate by assigning a delegate object

uiEntry

Additonal cmocka tests:

uiMenu
uiEntry

I also added some additional tests in the proposed qa/manual testing suite in #133
Consists of/supersedes: #154 #166 #159 #168 #167 #175 #172 #158 #162
Happy for any testers, I hope I didn't break anything along the way.
For anyone wondering why uiEntry is not done via subclassing: The code it interwoven with uiSpinbox and we have 3 different NS-types in there as well. I'm not a fan of the coupling, but IMO decoupling is something TBD in the upcoming darwin refactor (min version, ARC, etc).

First attempt to rebase all the commits left some weird artifacts, so I went with the merge strategy. Sorry for the verbosity, but this is git...

Fixed the issues in uiMenu that had escaped my manual testing.

👍 thank you so much!


sfc playground Tip: add an "import-map.json" file to specify import paths for dependencies.

Vue version
3.x
Link to minimal reproduction
shorturl.at/q2389
Steps to reproduce
i had import relative library cdn
What is expected?
it is ok
What is actually happening?
it can't work
System Info
No response
Any additional comments?
No response

wrong reproduction link.

wrong reproduction link.

shorturl.at/q2389

Just remove the , trailing comma of line 4


[CARBONDATA-1942] Documentation for Concurrent Lock Retries

Add documentation for concurrent lock retires. Add the parameter name, Default value, and description

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

add to whitelist

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/2359/


blade.resourcebundle sample

Just started reviewing :) :octocat: Sent from GH.

This has been pushed to new repo here: https://github.com/gamerson/liferay-blade-samples/pull/6


Chore: Moved tests into own directory

Also changed imports from relative to absolute, and added a real test to its not a completely useless PR :P

Its because of how pytest/python loads modules, when you use relative imports it can cause circular dependencies which makes it break


Hang in sync_wait() when coroutine has co_await() expression.

The following tests hang (when added to test/sync_wait_tests.cpp):
TEST_CASE("sync_wait(task<>) with await")
{
        static bool hit_result;

        auto makeTask = []() -> cppcoro::task<>
        {
                co_await std::experimental::suspend_always{};
                hit_result = true;
                co_return;
        };

        hit_result = false;
        auto task = makeTask();
        cppcoro::sync_wait(task);
        CHECK(hit_result);

        hit_result = false;
        cppcoro::sync_wait(makeTask());
        CHECK(hit_result);
}

TEST_CASE("sync_wait(shared_task<>) with await")
{
        static bool hit_result;

        auto makeTask = []() -> cppcoro::shared_task<>
        {
                co_await std::experimental::suspend_always{};
                hit_result = true;
                co_return;
        };

        hit_result = false;
        auto task = makeTask();
        cppcoro::sync_wait(task);
        CHECK(hit_result);

        hit_result = false;
        cppcoro::sync_wait(makeTask());
        CHECK(hit_result);
}

Compiler information:
Ubuntu clang version 12.0.0-++20200925052620+2a11a197af7-1~exp1~20200925153300.175
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

Build flags for that file:
/usr/bin/clang++-12  -I../include -g -stdlib=libc++ -fcoroutines-ts -std=c++20 -MD -MT test/CMakeFiles/run.dir/sync_wait_tests.cpp.o -MF test/CMakeFiles/run.dir/sync_wait_tests.cpp.o.d -o test/CMakeFiles/run.dir/sync_wait_tests.cpp.o -c ../test/sync_wait_tests.cpp

and link flags for the test file (filenames omitted for brevity):
/usr/bin/clang++-12 -g -stdlib=libc++ ${OBJECT_FILES_OMITTED} -o test/run  lib/libcppcoro.a  -lpthread

I ran the tests in lldb, let it hang a bit, and then stopped it to get the backtrace. It looks as though it's deadlocking due to a mutex:
* thread #1, name = 'run', stop reason = signal SIGSTOP
  * frame #0: 0x00007ffff7c6089d libc.so.6`syscall + 29
    frame #1: 0x00000000006481c2 run`(anonymous namespace)::local::futex(UserAddress=0x00007fffffffd2b8, FutexOperation=128, Value=0, timeout=0x0000000000000000, UserAddress2=0x0000000000000000, Value3=0) at lightweight_manual_reset_event.cpp:131:11
    frame #2: 0x0000000000648251 run`cppcoro::detail::lightweight_manual_reset_event::wait(this=0x00007fffffffd2b8) at lightweight_manual_reset_event.cpp:181:16
    frame #3: 0x00000000005613e5 run`cppcoro::awaitable_traits<cppcoro::task<void>&, void>::await_result_t cppcoro::sync_wait<cppcoro::task<void>&>(awaitable=0x00007fffffffd460) at sync_wait.hpp:32:9
    frame #4: 0x000000000056262e run`_DOCTEST_ANON_FUNC_8() at sync_wait_tests.cpp:81:2
    frame #5: 0x0000000000418685 run`doctest::Context::run(this=0x00007fffffffe3c8) at doctest.h:5651:7
    frame #6: 0x0000000000419056 run`main(argc=1, argv=0x00007fffffffe4d8) at doctest.h:5759:71
    frame #7: 0x00007ffff7b6c0b3 libc.so.6`__libc_start_main + 243
    frame #8: 0x000000000040c0ae run`_start + 46

But upon running strace with just that test enabled, I get the following output prior to the test hanging.
write(1, "sync_wait(task<>) with await\n", 29sync_wait(task<>) with await
) = 29
sigaltstack({ss_sp=0x45a880, ss_flags=0, ss_size=8192}, {ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}) = 0
rt_sigaction(SIGINT, {sa_handler=0x4181a0, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK, sa_restorer=0x7facf8af53c0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7facf8af53c0}, 8) = 0
rt_sigaction(SIGILL, {sa_handler=0x4181a0, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK, sa_restorer=0x7facf8af53c0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7facf8af53c0}, 8) = 0
rt_sigaction(SIGFPE, {sa_handler=0x4181a0, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK, sa_restorer=0x7facf8af53c0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7facf8af53c0}, 8) = 0
rt_sigaction(SIGSEGV, {sa_handler=0x4181a0, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK, sa_restorer=0x7facf8af53c0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7facf8af53c0}, 8) = 0
rt_sigaction(SIGTERM, {sa_handler=0x4181a0, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK, sa_restorer=0x7facf8af53c0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7facf8af53c0}, 8) = 0
rt_sigaction(SIGABRT, {sa_handler=0x4181a0, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK, sa_restorer=0x7facf8af53c0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7facf8af53c0}, 8) = 0
futex(0x7fffb05b0728, FUTEX_WAIT_PRIVATE, 0, NULL

The hang happens for both task and shared_task.
Let me know if there's something I can do to further debug this.

Nevermind, I understand now the problem - using suspend_always is an enticing trap to confuse one about how the standard actually works :)
Disregard.

Hi @Qix-
May I ask how you solved this issue? I'm running into the same thing.

Hey, unfortunately not. Coroutines are convoluted and since I've not worked with them lately I've forgotten almost everything about them.
Best of luck :/

No worries, thanks for the reply :)

For reference, the problem in the original code snippet was that it was awaiting the suspend_always awaitable which suspends the coroutine and never resumes it, meaning that sync_wait() will wait forever as the coroutine will never complete.
As a general rule, you should never await suspend_always in the body of a coroutine. It’s mainly intended for use in the implementation of a coroutine promise type.

Oh right, yes. This issue was filed pretty early on in playing with coroutines and definitely was a misunderstanding of suspend_always.
Thanks again for all the help and prose @lewissbaker :)

Thanks @lewissbaker. I've come to the conclusion that my issue was not cppcoro's fault either. There was a deadlock in another library that I'm using and I was only looking at the backtrace for the main thread in gdb which was just showing sync_wait waiting for my program to complete, as it should.


Brainstorming: how to change the way plugins are resolved

There is one last reason pnpm has issues: the way plugins are resolved. This causes two big issues:

pnpm works well only with top-level plugins or when Node.js is executed with --preserves-symlinks. However, the --preserve-symlinks mode will most probably be deprecated in the future. There is an open discussion about solving the symlinks problem in Node.js, but I don't see big enthusiasm around it. Most probably es6 modules won't even support the --preserve-symlinks option.
many pluggable packages are designed to work with flat dependency trees. Flat dependencies trees are needed for packages like npm or yarn, which copy/paste packages. pnpm does not need flat dependency trees because it solves the problem differently.

I've spent a lot of time trying to solve all the issues by changing pnpm. However, it seems that making a fast package manager is not enough, we have to change the way packages are written as well. Can we come up with a better way to write pluggable packages (I mean like eslint, babel, etc)? Can we come up with a solution that will work with pnpm + Node without the preserve-symlinks option?
So requirements:

should work with pnpm
should work with plain Node.js, no additional configuration should be required
should be better than the existing approaches (otherwise it won't be adopted. Nobody cares about pnpm, nobody will change something just to support another package manager)

After thinking about it a little bit more... I think we can create custom bin files that will add the needed node_modules folders to the search path. (Using the NODE_PATH env variable). Seems to work with eslint.

The solution should work for packages that use require.resolve to find their plugins. There is one exception for packages that are used both globally and locally (like gulp). I suggest to always use packages locally, but I'll open a separate issue for that problem


Release 0.1.14

Have we verified that the current codebase is stable and working? @schilippe had added two commits recently and I'd want to test unpacked before pushing out a new version.

in the short term I don't think we need worry about efficiency---as I
mentioned, I hit the storage limit only after a year and haven't noticed
any performance issues, so that gives us a year to think about handling
performance issues for anyone else.
On 2/10/2017 2:58 PM, Steven Allen wrote:

    You can view, comment on, or merge this pull request online at:

https://github.com/haystack/tipsy/pull/99
    Commit Summary

Release 0.1.14
File Changes

M chrome-extension/manifest.json
https://github.com/haystack/tipsy/pull/99/files#diff-0 (2)
Patch Links:

https://github.com/haystack/tipsy/pull/99.patch

https://github.com/haystack/tipsy/pull/99.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/haystack/tipsy/pull/99, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABFpXopkwLSc3iq-bmSbILQBYEz9im10ks5rbMFYgaJpZM4L9yoN.

I think we're also still hopinh for @schilippe to clarify what was in
those two commits...
On 2/12/2017 12:16 PM, Tim Branyen wrote:

Have we verified that the current codebase is stable and working?
@schilippe https://github.com/schilippe had added two commits
recently and I'd want to test unpacked before pushing out a new version.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/haystack/tipsy/pull/99#issuecomment-279232906, or
mute the thread
https://github.com/notifications/unsubscribe-auth/ABFpXrwSQrxoSBS0WB96xnBuhN0nm-c6ks5rbz5jgaJpZM4L9yoN.


When serving APIs how to get rid of @ts-nocheck

Hi together,
since CAP 8 we have to problems that this typeScript code is criticized before that it worked like this, what would be the solution without nocheck. I've putted a sample code for this here
// @ts-nocheck

import { Service } from '@sap/cds';
import { x } from '@;
import { y } from '@';

type ErrorHandler = (error: any, request: any) => Promise<void>;

export class x extends Service {
    static SERVICE_DESCRIPTOR = {
        operationTypeToRequiredScopes: {
            READ: x
        },
    };

    async init() {
        //---------------------------------------------------------------------
        // Generic error message handler
        this.on('error', this.handleError as ErrorHandler);

        return super.init();
    }

    private async handleError(error: any, request: any) {
        await x(
            error,
            request,
           y
        );
    }
}

When we remove nocheck we get this
Property 'on' does not exist on type 'x'.ts(2339)
any

Hi @eminoklopcic ,
thanks for opening this issue. IN this repository we only cover documentation related issues. Yours is different. Do you use the @cds-typer package?

Hi yes, we use @.***/cds-types": "0.7.0",
From: René Jeglinsky @.>
Date: Wednesday, 27. November 2024 at 14:22
To: cap-js/docs @.>
Cc: Oklopcic, Emin @.>, Mention @.>
Subject: Re: [cap-js/docs] When serving APIs how to get rid of @ts-nocheck (Issue #1473)
You don't often get email from @.*** Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification
Hi @eminoklopcichttps://github.com/eminoklopcic ,
thanks for opening this issue. IN this repository we only cover documentation related issues. Yours is different. Do you use the @cds-typer package?
—
Reply to this email directly, view it on GitHubhttps://github.com/cap-js/docs/issues/1473#issuecomment-2503867038, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BFNAEJNMKETDDFL6GN46P5L2CXBQVAVCNFSM6AAAAABSSKB3FGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBTHA3DOMBTHA.
You are receiving this because you were mentioned.Message ID: @.***>


pure alsa crashes after less than an hour

Debian Buster, pyenv with Python 3.7.4. Pure alsa no pulseaudio.
Runs pretty well for at least 30 minutes, then suddenly crashes.
Full log:
$ ledfx --open-ui
Loading configuration file from /home/dreamer/.ledfx
Started webinterface at http://<IP_ADDRESS>:8888
ALSA lib pcm_dsnoop.c:638:(snd_pcm_dsnoop_open) unable to open slave
ALSA lib pcm_dmix.c:1108:(snd_pcm_dmix_open) unable to open slave
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib pcm_oss.c:377:(_snd_pcm_oss_open) Unknown field port
ALSA lib pcm_oss.c:377:(_snd_pcm_oss_open) Unknown field port
ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Connection refused

ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Connection refused

ALSA lib pcm_a52.c:823:(_snd_pcm_a52_open) a52 is only for playback
ALSA lib pcm_usb_stream.c:486:(_snd_pcm_usb_stream_open) Invalid type for card
ALSA lib pcm_usb_stream.c:486:(_snd_pcm_usb_stream_open) Invalid type for card
ALSA lib pcm_dmix.c:1108:(snd_pcm_dmix_open) unable to open slave
python3.7: src/hostapi/alsa/pa_linux_alsa.c:3641: PaAlsaStreamComponent_BeginPolling: Assertion `ret == self->nfds' failed.
Aborted

Hmm, looking deeper this seems to be a PortAudio issue: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=944509

Hi Dromer,
We have done quite a bit of development and documentation work. Have a look at the readme on: https://github.com/ahodges9/LedFx/tree/dev & https://ledfx.readthedocs.io/en/docs/index.html
Can I ask you to try the LedFx Exe version 0.0.3: https://ledfx.app/download/ - only x64 bit environment
Your welcome to join the LedFx community on Discord and we can help you out from there: https://discord.com/invite/wJ755dY

What do you expect me to do with a .exe?

Yeah still have major issues. With alsa it works for a couple seconds and then fails. Jack doesn't work at all.
There is more output when it fails, so maybe this helps you:
$ ledfx --open-ui
Loading configuration file from /home/dreamer/.ledfx
Started webinterface at http://<IP_ADDRESS>:8888
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'cards.USB-Audio.pcm.hdmi.0:CARD=0,AES0=4,AES1=130,AES2=0,AES3=2'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM hdmi
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'cards.USB-Audio.pcm.hdmi.0:CARD=0,AES0=4,AES1=130,AES2=0,AES3=2'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM hdmi
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'cards.USB-Audio.pcm.modem.0:CARD=0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline:CARD=0,DEV=0
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'cards.USB-Audio.pcm.modem.0:CARD=0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline:CARD=0,DEV=0
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'cards.USB-Audio.pcm.modem.0:CARD=0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM phoneline
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'cards.USB-Audio.pcm.modem.0:CARD=0'
ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM phoneline
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
jack server is not running or cannot be started
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
jack server is not running or cannot be started
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
ALSA lib pcm_oss.c:377:(_snd_pcm_oss_open) Unknown field port
ALSA lib pcm_oss.c:377:(_snd_pcm_oss_open) Unknown field port
ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Connection refused

ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Connection refused

ALSA lib pcm_a52.c:823:(_snd_pcm_a52_open) a52 is only for playback
ALSA lib pcm_a52.c:372:(a52_slave_hw_params_half) Cannot get slave hw_params
ALSA lib pcm_usb_stream.c:486:(_snd_pcm_usb_stream_open) Invalid type for card
ALSA lib pcm_usb_stream.c:486:(_snd_pcm_usb_stream_open) Invalid type for card
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
jack server is not running or cannot be started
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
python3.7: src/hostapi/alsa/pa_linux_alsa.c:3641: PaAlsaStreamComponent_BeginPolling: Assertion `ret == self->nfds' failed.
Aborted


Added a CLI arg to determine the language to generate through TTS

Minimal addition of a CLI option that limits the generation to a particular language.
Probably useless


CI/CD Error: Cannot find module @rollup/pluginutils

Ci/CD pipeline shows
"cannot find module ' @rollup/pluginutils'

Require stack:

<EMAIL_ADDRESS>rollup.config.js
/node_modules/rollup/dist/shared/loadConfigFile.js
/node_modules/rollup/dist/bin/rollup

** To reproduce: **
Rollup and @lopatnov/rollup-plugin-uglify are dependencies in a  Drupal project.
We have the following rollup devdependencies:
    "@lopatnov/rollup-plugin-uglify": "^2.1.2",
    "rollup": "^2.61.1",
    "rollup-plugin-commonjs": "^10.1.0",
    "rollup-plugin-node-builtins": "^2.1.2",
    "rollup-plugin-node-globals": "^1.4.0",
    "rollup-plugin-node-resolve": "^5.2.0",

in our CI/CD pipeline, we execute the following npm script:
    "scripts:rollup": "rollup -c rollup.config.js",

And this is in our rollup config:
import commonjs from "rollup-plugin-commonjs";
import nodeResolve from "rollup-plugin-node-resolve";
import globals from "rollup-plugin-node-globals";
import builtins from "rollup-plugin-node-builtins";
import uglify from "@lopatnov/rollup-plugin-uglify";

export default [
  {
    input: "src/scripts/js/main.js",
    output: {
      file: "dist/js/main.min.js",
      format: "iife",
      name: "Main",
      sourcemap: true
    },
    plugins: [
      nodeResolve({ preferBuiltins: false }), // or `true`
      commonjs(),
      globals(),
      builtins(),
      uglify(),
    ],
  },
];

Expected behavior
We expect Rollup to build without issue
Screenshots
If applicable, add screenshots to help explain your problem.

Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
We're running this in Gitlab.

Hi @paceaux , now rollup-plugin-uglify uses @rollup/pluginutils. I thing you'll have to install it as dev dependency on your project or use the previous version of this plugin. The change was done on #10

Dear @paceaux, did you try to install rollup-pluginutils: npm i --save-dev @rollup/pluginutils?

@lopatnov we were running node >=14.19.1 both locally and in our CD environments. Clearing node_modules and removing package-lock.json did not work.
@danielart Our temporary fix was to rollback and use 2.1.2 (our last known safe version).
This change broke all of our CI builds.
If this plugin now has a dependency on @rollup/pluginutils that we have to install, it might be good to update the readme (but also, why can't it be made a dependency of the plugin?)
I'm sure my employer won't be the only once experiencing broken builds.

@lopatnov I think this issue could be resolved if @rollup/pluginutils were a dependency instead of a devDependency.

Indeed, it can be mark as a dependency, that will force to install the new pluginutils. But have in mind that the  previous versions had the old pluginutils version as a dependency, so you'll have to remove manually the old one or check how to manage both versions if you need the old one for other plugins. In any case, is a good idea to add it to readme/changelog and check which devdeps should be dependencies. @paceaux  thanks for the issue, feel free to open a PR with the changes.
I'll talk to my company and see what we can do to support you. Do you have a "sponsor" page or links I can share?

I'm only a contributor, all merits to @lopatnov . But I think you can support him through this Link

Dear @paceaux , I have a job. I think that a small amount of help will be more significant for Izmail Perinatal Center . It could be small help for small people.
I've merged fix. Can you check? Is it working?

Inactive issue closed. Fixed!


augeas requirement not listed in puppet forge

Hey guys, pretty self explanatory but:
In init.pp and systemd.pp it has augeas methods but augeas isn't listed as a dependency and I see no provider for it.
Easy enough to resolve, but wanted to make sure it wasn't over looked.
Thanks for the outstanding module!

@chad-upton sorry you are running into this issue.
When using the puppetlabs repo's ( which most people do in prod environments ) augeas is automatically installed indeed.
I'm not sure if there is a nice way to manage the dependency.

Hi,
Augeas is now reported under the requirements at https://github.com/elastic/puppet-elasticsearch#requirements
Closing


[µTVM] Use standalone_crt build tree for all µTVM builds

Right now due to historical reasons we sometimes build the CRT from build/standalone_crt and other times from src/runtime/crt. This PR consolidates building to always be from build/standalone_crt and updates several path computations to use that. Ultimately we'll need this if we expect microTVM to be useful when TVM has been pip installed rather than just built from source.
This is also helpful for autotuning, as we expect the autotvm runner to use standalone_crt to build the project. This PR also changes the default compiler options to reference the standalone_crt directory.
Finally, to properly separate the host main() and crt_config.h from the remaining reusable CRT libs, places these in a new directory standalone_crt/template along with the template crt_config.h.

Hi @areusch,
This is great!
Out of curiosity,  the basic purpose of the PR is essentially move the crt sources to build directory right  ? -- so that they get shipped with the wheel ?

Hi @areusch,
This is great!
Out of curiosity,  the basic purpose of the PR is essentially move the crt sources to build directory right  ? -- so that they get shipped with the wheel ?

@manupa-arm yeah correct--it revises all functions in tvm.micro to operate relative to either a default standalone_crt directory, which now resides in build, or to one you specify. for autotuning, this becomes more helpful as i've moved compilation to the runner, so the standalone_crt directory will be different than the default. but, it also will help in future cases where standalone_crt is included as a wheel data file and it is necessary to write it out to disk in a project-specific location.

@manupa-arm yeah correct--it revises all functions in tvm.micro to operate relative to either a default standalone_crt directory, which now resides in build, or to one you specify. for autotuning, this becomes more helpful as i've moved compilation to the runner, so the standalone_crt directory will be different than the default. but, it also will help in future cases where standalone_crt is included as a wheel data file and it is necessary to write it out to disk in a project-specific location.

@tqchen @manupa-arm @u99127 @gromero @tom-gall looks like this is ready for review

@areusch Hi Andrew. The change looks good (I'll differ the review of Jenkinsfile to others). I just suggest to fit docstrings of get_standalone_crt_dir and get_standalone_crt_lib() in 80 columns maximum. Also, a side note on the PR itself, if possible, I'd avoid merges in the middle of the incremental changes pushed to the branch to be reviewed, because it makes a tad difficult to squash the commits into the final change, in that sense I think rebasing on top of master is better. I also tested micro_tflite.py against a disco board and it's all right. So, thumbs up from my side.

@gromero thanks for the review. unfortunately if you force-push (I.e. rebase) it loses the context around the previous comments, and I like that less :/. for line length, we're at 100 char, and pylint's enforcing that, so i'm keeping to that here.

@gromero thanks for the review. unfortunately if you force-push (I.e. rebase) it loses the context around the previous comments, and I like that less :/.

@areusch yeah.. I forgot about the "constraint" on push -f. Maybe the only thing we could do is to try to avoid merging in the middle of the incremental pushes (pushes due to the review process)? On the other hand  that amounts to not doing a rebase on the top of the most fresh code in main, but I wonder if github and the CI is actually already covering what might go wrong if rebase is not done (github will check for merge conflicts on top of the main branch, but I'm not sure about the CI, I guess it only builds and tests the branch ones want to merge into main, i.e. it doesn't build the commits found in github's "Commits" tab on top of the main branch). Yep, tricky thing, merging in the middle of the incremental pushes for complicated changes might get really hard to review. I see.

@gromero easiest way to patch given merges might be something like:
$ git fetch origin main
$ git fetch areusch <pr>
$ git diff areusch/<pr>...origin/main >patch # NOTE 3 periods
$ git checkout $(merge-base areusch/<pr> origin/main)
$ git apply <patch

I think the rest is just limitations of this code-review system. I generally try to push -f until there's a comment on the thread

@gromero easiest way to patch given merges might be something like:
$ git fetch origin main
$ git fetch areusch <pr>
$ git diff areusch/<pr>...origin/main >patch # NOTE 3 periods
$ git checkout $(merge-base areusch/<pr> origin/main)
$ git apply <patch

@areusch I see. I imagine that's the flow the committer will follow when squashing for the final commit to get merged into main? Because if one does a git rebase -i straight to the branch related to the PR with the incremental changes "intertwined" with the commits brought by the merges it would kind of a nightmare to squash afaics?
Now, I've just made a quick test and I see that the review comments (even the comments related to a line of code) haven't disappeared after a git push -f, so it does really keep the review history. What I did on the experiment for a PR under review was simply the following, whilst checked out in the PR branch (pr) and with an incremental change already committed to the branch:
git pull --rebase origin main
git push -f origin pr

The forced push and the new incremental change (plus the changes merged / pushed into main meanwhile) were all correctly displayed below the review comments posted previously to it.
Feel free to ping for a few tests if you want :)

I generally try to push -f until there's a comment on the thread

Yeah, that's what I do too.

@gromero yes, I think that's roughly what GH does when someone hits merge.
the review comments do stick around, but the context is usually obliterated, which is usually pretty crucial to understanding what they meant. maybe see if the old sha is still available after a few hours--GH may do a cleanup task internally

@gromero yes, I think that's roughly what GH does when someone hits merge.

Got it :+1:

the review comments do stick around, but the context is usually obliterated, which is usually pretty crucial to understanding what they meant. maybe see if the old sha is still available after a few hours--GH may do a cleanup task internally

@areusch Amazing, indeed, after some time, when clicking on "View Changes" related to a commit that was rebased (and git pushed -f) I get:
We went looking everywhere, but couldn’t find those commits.
Sometimes commits can disappear after a force-push. Head back to the latest changes here.

The best part is "Sometimes commits can disappear... ". OK, I give up :P

@gromero @manupa-arm can you explicit ack if you're ok with this change?
https://tvm.apache.org/docs/contribute/code_review.html#approve-and-request-changes-explicitly

@tmoreau89 @tqchen to merge

Thank you @areusch @gromero @manupa-arm - the PR has been merged


[BUG] pygwalker bug report: interactive UI doesn't show up

Describe the bug
When trying to load a df with pygwalker inside Jupyter Notebook to display the interactive UI, a HTML message is displayed instead of the interactive UI.
import pandas as pd
import pygwalker as pyg
df = pd.read_csv("toto.csv", sep=',')
gwalker = pyg.walk(df)

Box(children=(HTML(value='<div id="ifr-pyg-1" style="height: auto">\n    <head>\n        <meta http-equiv="Con…

To Reproduce
Steps to reproduce the behavior:

Go to 'jupyter notebook'
Click on 'load a csv into a pandas dataFrame'
type gwalker = pyg.walk(df)
See error

Expected behavior
A graphical and interactive UI shows up !
Screenshots

Versions

pygwalker version: 0.3.17
python version: 3.11
browser: Brave

Additional context
Add any other context about the problem here.

Hi, @PhilippeDataScienc
Could you tell me the version of your notebook or jupyterlab?
I will try to reproduce it.
Also, Could you try running the code below and tell me what the output is?
from IPython.display import display
import ipywidgets as widgets

button = widgets.Button(description="Click Me!")
output = widgets.Output()

display(button, output)

def on_button_clicked(b):
    with output:
        print("Button clicked.")

button.on_click(on_button_clicked)

Thanks.

your code snippet works for me ! I have a click me button displayed!

My jupyterlab version is: 4.0.9

Problem seems to be solved with your code snippet, maybe I had an import missing ?

I guessed that it was your first time using the pygwalker process, and I successfully reproduced the bug.
Since you haven't installed ipywidgets before, when the Jupyter kernel was already running, installing pygwalker (which relies on ipywidgets for rendering) caused the notebook to not establish communication with ipywidgets, resulting in the unsuccessful rendering of pygwalker's UI.
However, when you close the kernel and refresh the page, JupyterLab should establish communication with ipywidgets again (or for some other reason, JupyterLab establishes communication with ipywidgets), so pygwalker could be re-rendered.
This bug will not occur again when you use pygwalker in the future.


Erroneous compiler error when using generic intersection type as generic class parameter

TypeScript Version:  2.1.6 and 2.2.0-dev.20170212
Code
class Foo<T> {
	f(x: Partial<T>) { }
}
class Bar<T> extends Foo<T & { a: number }> {
	constructor() {
		super();
		this.f({ a: 0 });
	}
}

Expected behavior:
Code compiles without error. Worked fine with 2.2.0-dev.20161204 and broke on 2.2.0-dev.20161205.
Actual behavior:
The following error is generated:

main.ts(7,10): error TS2345: Argument of type '{ a: 0; }' is not assignable to parameter of type 'Partial<T & { a: number; }>'.

Broken by #12643. The break may have been intentional; I need to re-read the change.

After looking at #12643, I'm not yet sure whether the example should be legal or not, but I do know that it worked by mistake previous to the change. Before #12643, keyof (T & { a: number }) was incorrectly identical to keyof { a: number }.
The breaking change happens in getIndexType, which creates types for keyof T. Before #12643, if the type was a type parameter, then a generic index type was created (keyof T, basically). Otherwise the index type was created immediately. After #12643, the code checks whether the type is a type parameter or is a union/intersection that contains a type parameter.
I'm not sure whether { a: 0 } should be assignable to Partial<T & { a: number }> or not. Seems like it should on the surface. I'll look into that next.

Appreciate your looking into this!
I am using this structure to create abstract, reusable React components with TypesScript. A simplified example might be something like the following:
abstract class Dialog<P, S> extends React.Component<P, S & { submitButtonText: string }> {
	f() {
		this.setState({ submitButtonText: '' });
	}
}
class SignUpDialog extends Dialog<{}, { userName: string }> {
	g() {
		this.setState({ userName: '' });
	}
	h() {
		this.setState({ submitButtonText: '' });
	}
}

This way the abstract components can still manage the state that they are aware of.
The compiler logic is a bit over my head, but in the above example the error is only thrown on the statement in f() and not h() which doesn't seem right to me.

Hm. mappedTypeRelatedTo has a special case for Partial and friends: ONLY {} is assignable to it. I don't think this special case takes into account that the constraint type of a mapped type might have known properties like Partial<T & { known: number }> does.
@ahejlsberg, can you take a look? It looks like mappedTypeRelatedTo needs to change to account for cases like this.

I think the current behaviour is correct but unexpected. The explanation is given here.
I guess using spread types instead of intersections would yield the expected result.

In this case we do know some properties of T & { a: number}, so {a: number} should be assignable to Partial<T & { a: number }> if it is assignable to Partial<{ a: number }>.

T may have a property a: string. The resulting type would then be a: string & number which makes a: number not assignable to the intersected type. Generally, the values of the properties of type T are not assignable to the corresponding properties in T & U. In the above case U is a: number.

This might be a bit off track, but I'm surprised the result of conflicting properties on an intersection type, where any one conflicted property is a primitive, is an intersection of those properties. In such cases I would expect the result type to instead be a union of the conflicted properties in which case I don't think there would be a problem with the assignment.

@gcnew Oh, I see now. You are right, it is surprising but unfortunately correct that { a: number } is not assignable to T & { a: number } inside of Partial.
@jacamera string & number is impossible to create but string | number is not correct because the resulting T & { a: number}  might end up with a: string instead of a: number, and uses of it inside Bar would incorrectly assume it to be a number.
The usual case for intersections is either no overlap or agreement in types. Intersecting conflicting types is just a fallback that is correct once in a while, and unusable the rest of the time.

Understood. Thanks again for your time.
To anyone else who finds this, the dialog example from above can be rewritten in the following way to avoid any compiler errors. The only downside I see is that all the properties of the abstract component's state type have to be optional so you just have to know which ones are required in the initial state assignment.
interface DialogState { submitButtonText: string }
abstract class Dialog<P, S extends Partial<DialogState>> extends React.Component<P, S> {
	f() {
		this.setState({ submitButtonText: '' });
	}
}
class SignUpDialog extends Dialog<{}, Partial<DialogState> & { userName: string }> {
	g() {
		this.setState({ userName: '' });
	}
	h() {
		this.setState({ submitButtonText: '' });
	}
}


Fix rendering issue in the Developer Getting Started Guide (#3027)

Pull request opened by the merge tool on behalf of #3027

Merged into branches iso4 in e1c49950fdad380c94565b0da06b2f1d63fc5208


ValueError: Object arrays cannot be loaded when allow_pickle=False

ERROR (theano.gpuarray): pygpu was configured but could not be imported or is too old (version 0.7 or higher required)
NoneType: None
Y:\deep_learning\3D_R2N2\3D-R2N2-master\lib\layers.py:349: UserWarning: DEPRECATION: the 'ds' parameter is not going to exist anymore as it is going to be replaced by the parameter 'ws'.
pooled_out = pool.pool_2d(
Y:\deep_learning\3D_R2N2\3D-R2N2-master\lib\layers.py:349: UserWarning: DEPRECATION: the 'padding' parameter is not going to exist anymore as it is going to be replaced by the parameter 'pad'.
pooled_out = pool.pool_2d(
loading network parameters from output/ResidualGRUNet/default_model/weights.npy
Traceback (most recent call last):
File "Y:/deep_learning/3D_R2N2/3D-R2N2-master/demo.py", line 86, in
main()
File "Y:/deep_learning/3D_R2N2/3D-R2N2-master/demo.py", line 64, in main
net.load(DEFAULT_WEIGHTS)                        # load downloaded weights
File "Y:\deep_learning\3D_R2N2\3D-R2N2-master\models\net.py", line 71, in load
params_cpu_file = np.load(filename)
File "Y:\ProgramData\Anaconda3\lib\site-packages\numpy\lib\npyio.py", line 446, in load
return format.read_array(fid, allow_pickle=allow_pickle,
File "Y:\ProgramData\Anaconda3\lib\site-packages\numpy\lib\format.py", line 692, in read_array
raise ValueError("Object arrays cannot be loaded when "
ValueError: Object arrays cannot be loaded when allow_pickle=False
Process finished with exit code 1

ERROR (theano.gpuarray): pygpu was configured but could not be imported or is too old (version 0.7 or higher required)
NoneType: None
Y:\deep_learning\3D_R2N2\3D-R2N2-master\lib\layers.py:349: UserWarning: DEPRECATION: the 'ds' parameter is not going to exist anymore as it is going to be replaced by the parameter 'ws'.
pooled_out = pool.pool_2d(
Y:\deep_learning\3D_R2N2\3D-R2N2-master\lib\layers.py:349: UserWarning: DEPRECATION: the 'padding' parameter is not going to exist anymore as it is going to be replaced by the parameter 'pad'.
pooled_out = pool.pool_2d(
loading network parameters from output/ResidualGRUNet/default_model/weights.npy
Traceback (most recent call last):
File "Y:/deep_learning/3D_R2N2/3D-R2N2-master/demo.py", line 86, in
main()
File "Y:/deep_learning/3D_R2N2/3D-R2N2-master/demo.py", line 64, in main
net.load(DEFAULT_WEIGHTS) # load downloaded weights
File "Y:\deep_learning\3D_R2N2\3D-R2N2-master\models\net.py", line 71, in load
params_cpu_file = np.load(filename)
File "Y:\ProgramData\Anaconda3\lib\site-packages\numpy\lib\npyio.py", line 446, in load
return format.read_array(fid, allow_pickle=allow_pickle,
File "Y:\ProgramData\Anaconda3\lib\site-packages\numpy\lib\format.py", line 692, in read_array
raise ValueError("Object arrays cannot be loaded when "
ValueError: Object arrays cannot be loaded when allow_pickle=False
Process finished with exit code 1

hello, I met the same question with you, have changed this question？

ERROR (theano.gpuarray): pygpu was configured but could not be imported or is too old (version 0.7 or higher required)
NoneType: None
Y:\deep_learning\3D_R2N2\3D-R2N2-master\lib\layers.py:349: UserWarning: DEPRECATION: the 'ds' parameter is not going to exist anymore as it is going to be replaced by the parameter 'ws'.
pooled_out = pool.pool_2d(
Y:\deep_learning\3D_R2N2\3D-R2N2-master\lib\layers.py:349: UserWarning: DEPRECATION: the 'padding' parameter is not going to exist anymore as it is going to be replaced by the parameter 'pad'.
pooled_out = pool.pool_2d(
loading network parameters from output/ResidualGRUNet/default_model/weights.npy
Traceback (most recent call last):
File "Y:/deep_learning/3D_R2N2/3D-R2N2-master/demo.py", line 86, in
main()
File "Y:/deep_learning/3D_R2N2/3D-R2N2-master/demo.py", line 64, in main
net.load(DEFAULT_WEIGHTS) # load downloaded weights
File "Y:\deep_learning\3D_R2N2\3D-R2N2-master\models\net.py", line 71, in load
params_cpu_file = np.load(filename)
File "Y:\ProgramData\Anaconda3\lib\site-packages\numpy\lib\npyio.py", line 446, in load
return format.read_array(fid, allow_pickle=allow_pickle,
File "Y:\ProgramData\Anaconda3\lib\site-packages\numpy\lib\format.py", line 692, in read_array
raise ValueError("Object arrays cannot be loaded when "
ValueError: Object arrays cannot be loaded when allow_pickle=False
Process finished with exit code 1

hello, I met the same question with you, have changed this question？

Try to uninstall the numpy and insatll the previous version of numpy,such as numpy 1.16.1. I met the same question like you guys, this will help. Coz the new version of numpy  add some function which are different then this project.

I also meet this question，your method did work,thanks.

Try to uninstall the numpy and insatll the previous version of numpy,such as numpy 1.16.1. I met the same question like you guys, this will help. Coz the new version of numpy add some function which are different then this project.

Thanks. A change of Numpy to Version 1.16.1 worked for me.
Maybe this Version should be fixed in the ''requirements.txt''.


[nrf fromtree] tests: drivers: uart: Add nrf54l15/cpuflpr overlays to test scope

Adding a separate overlay for nrf54l15 flpr is required because running tests on console uart fails.
Upstream PR: https://github.com/zephyrproject-rtos/zephyr/pull/73331

Merged through #1777.


iOS8 Rotation issue

Seem to have a problem with landscape orientation in iOS8. The progress view is always in portrait.

I just updated our Podfile to use 0.3.0. Doing clean installs of Debug builds on iOS8.1 in the simulator and on an iPad Mini Retina still exhibit the issue.
It is definitely happening with the following call:
MMProgressHUD showWithTitle:

+1. This is still an issue. While my PR is a hack, it does seem to work (at least for me).

It would help very much if someone who is experiencing the issue to try and debug what specifically the problem is. As it stands, I currently cannot reproduce issues with rotation like everyone is describing on the latest code from the master branch using the example app in the Demo/ folder. Some questions I have for everyone having issues:

Is this on all versions of iOS 8 (iOS 8, iOS 8.1, etc)?
Is the deployment target for your app iOS 7+, or is it iOS 8-only?
Are you using UIViewController-based status bar methods?
Are you using UIViewController-based rotation methods?
What exactly does it do when it "doesn't work"? Does it not rotate at all, does it rotate, but look broken, etc?

Like I said, I cannot reproduce this issue anymore, which makes it extremely difficult to fix.

Here are some images and a description from our QA team's bug report:
On the current iOS staging build on iPhone 6 and 6+ if you tap to record audio while composing a message in landscape the audio record icon shows up incorrectly and if you rotate to portrait it does not correct itself.
Opening in portrait the audio icon is orientated appropriately but if you switch to landscape it gets out of whack.
See attached images for examples.
http://imgur.com/a/rDR42
Is this on all versions of iOS 8 (iOS 8, iOS 8.1, etc)?
iOS 8+
Is the deployment target for your app iOS 7+, or is it iOS 8-only?
iOS 7+
Are you using UIViewController-based status bar methods?
No
Are you using UIViewController-based rotation methods?
No
What exactly does it do when it "doesn't work"? Does it not rotate at all, does it rotate, but look broken, etc?
See description above

Is this on all versions of iOS 8 (iOS 8, iOS 8.1, etc)?
iPad Mini w/ 8.1.2: YES
iPad simulators with 8.1: YES
Is the deployment target for your app iOS 7+, or is it iOS 8-only?
Deployment Target of 7.0.
Are you using UIViewController-based status bar methods?
No
Are you using UIViewController-based rotation methods?
No
What exactly does it do when it "doesn't work"? Does it not rotate at all, does it rotate, but look broken, etc?
We initiate the display of it and it remains in the incorrect orientation through out the duration of it being displayed.
When the device is in any orientation besides the default portrait mode, it is always aligned with the default portrait mode. I've attached to images. One in landscape and one in 'upside-down' or reverse portrait mode.

If it helps, at the time this was presented from LoginContainerViewController, this is the hierarchy:
<SplashViewController 0x7ae3d240>, state: disappeared, view: <UIView 0x7be91a70> not in the window
   + <LoginContainerViewController 0x7bea67a0>, state: appeared, view: <UIView 0x7bea7f90>, presented with: <_UIFullscreenPresentationController 0x7bea6d80>
   |    | <LoginViewController 0x7b86b850>, state: appeared, view: <UIView 0x7bead6b0>
   |    | <SelectMerchantViewController 0x7beaa360>, state: appeared, view: <UITableView 0x7c2dee00>

Hi,
I've just updated to version 3.0.0, and I'm hitting this issue too.
My application uses UIInterfaceOrientationLandscapeRight and UIInterfaceOrientationLandscapeLeft orientations only, however I've found that the MMProgressHUDWindow is being oriented incorrectly.
Looking at - (void)orientRootViewControllerForOrientation:(UIInterfaceOrientation)interfaceOrientation on MMProgressHUDWindow I've checked that the interfaceOrientation being passed is correct however, the applied transform is orienting the window incorrectly.
I've fixed my code for the moment by simply using a transform of 0, but I suspect this is not the complete solution.
--J

We are having this issue also.  iOS 8 with a deployment target of 7.1.  We managed to pull down the source for 2.3, fix the rotation issue, but now we are seeing a problem with the progress spinniner not dismissing properly sometimes and locking up the app.  v 0.30 is still broken.

Is anyone on this thread having issues with rotation on i_Phone_, or is it only on iPad that this issue manifests itself?

And as a follow-up can anyone reproduce this issue using the sample project included in this repo?

I just pushed up 0.3.1 to CocoaPods. This has some other fixes, but there was one commit related to rotation that might actually fix what everyone is talking about in here and might be why I have not been able to reproduce this issue. If this is true, this is my bad for not pushing a release with that one commit in it prior to this.
Please try 0.3.1 out, let me know how it works out, and have an awesome weekend, everyone! :beers:


Add an additional subtype check in Types::lub for AppliedType

Motivation

Currently, if a generic A includes the module M, the type T.any(A[Integer], M) doesn't reduce to M. This change adds an additional subtype check in in the case of Types::lub that deals with AppliedType that fixes this behavior.
Test plan

See included automated tests.

Pay-server build failure at http://go/3dqC8JiL

Pay-server build http://go/MxgdkWur

does GLB have the same issue?

does GLB have the same issue?

Good question. For the same case in glb, it checks whether or not the AppliedType derives from the  second argument. I'll see if I can come up with an example where it doesn't reduce, and run it by you on Tuesday.

I think that the same problem isn't present in glb. Starting from the case on subtyping.cc:797:

We know that it's not an AppliedType, and if it's not a ClassType, the result will be an AndType again.
If it is a ClassType, it won't have derived from the LHS, as that would require it to also be an AppliedType.


Fixed #31255 - Fixed AutoDetector from creating redundant RemoveFields.

DeleteModel and RemoveField weren't aware of what they were holding a reference to.  @charettes wrote some logic and refactored the reduce() and references_field() method of DeleteModel and RemoveField to solve the issue. Building on top of that, I fixed the ManyToManyField optimizations that were broken because of the changes.

This patch doesn't apply cleanly to the master branch. Please feel-free to send a new PR.


Add warning(s) to response header

Need to add a warning (or more, depending on the wording) if the user provided time is outside a reasonable range (tbd) of the TLE epoch, or if the range of times requested puts a result outside the reasonable range.

Also might be related - if the date requested is before the TLE date by an unknown amount, the values will all be null. Not how much off will trigger this issue (before launch date?) but I don't know if that should be part of this warning/error message or a separate one yet.

We could also still add the warning for a smaller date range, like 2 weeks each side, if we want to remind about ephemeris accuracy.

Since it's using the TLE closest to the date this shouldn't happen often until archival mode is working, then it won't happen at all (since it will always pick a contemporary TLE)


AT-3.5: Assign stock-keepers to audits

Test Case ID: AT-3.5
User Story: ALTA-3.5 (#53)
As an inventory manager, I want to be able to assign stock-keepers to perform audits so that I can manage the execution of multiple tasks simultaneously.
Acceptance Criteria
Given that the user is logged in as an inventory manager,

The user should be redirected to the Dashboard page.

The user may initiate an audit using 2 methods:
a. The user selects individual items through the Inventory Items page:
i. The user selects the Inventory Items side menu option.
ii. The user is presented with a table containing the organization's inventory items and a column of checkboxes associated with each listed item.
iii. The user marks the checkboxes of the items to be audited and selects the Start Audit button.
b. The user selects a template from the Templates page:
i. The user selects the Template side menu option.
ii. The user is presented with the organization's predefined templates.
iii. The user selects the desired audit template to begin the audit process.

Once the audit is initiated, the user is redirected to the Select Stock-Keepers page, which lists the availability of each stock-keeper per location and a column of checkboxes associated with each employee.

The user marks the checkboxes of the assigned stock-keepers and selects the Assign button.

The user is presented with the Designate Bins to Stock-Keepers page, which contains the locations of the selected items in separate expansion cards. Each expansion card presents the list of bins affiliated with the items to be audited as well as the assigned stock-keepers.

The user drags and drops the bins to the desired stock-keeper to assign the workload and selects the Assign button.

The user is redirected to the Audit Review page to ensure the selected criteria are correct before initiating the audit.

Process is well executed. ACCEPTED

Process is well executed. ACCEPTED


Is there any way that i could show dimensions of 3d Model

I wanted to show dimensions of 3d models, Is there any method where i can access size of the model.

You mean a bounding box size? We calculate that internally, so it wouldn't be hard to expose, but it's not currently part of our API.

@elalish yes i mean bounding box size, it will be very helpful if we get it via api.

so it wouldn't be hard to expose

Can you say how?

@buddy90210 It's a member of the Model.ts class. You could put a getter on loading.ts to return it as a Vector3D, which we use for the getCameraTarget() method, for instance. Would you like to make a PR?

@elalish yes please, can you make a PR??


can't successfully run kfp_v2  uats behind proxy

Bug Description
running kfp_v2 integration test from https://github.com/canonical/charmed-kubeflow-uats/tree/main/tests, commit [0] experiment fails on
$ kubectl logs -n dash pod/condition-v2-bcjmm-2542164255
I0703 15:00:08.666715      61 launcher_v2.go:90] input ComponentSpec:{
  "inputDefinitions": {
    "parameters": {
      "force_flip_result": {
        "parameterType": "STRING",
        "defaultValue": "",
        "isOptional": true
      }
    }
  },
  "outputDefinitions": {
    "parameters": {
      "Output": {
        "parameterType": "STRING"
      }
    }
  },
  "executorLabel": "exec-flip-coin"
}
I0703 15:00:08.667506      61 cache.go:139] Cannot detect ml-pipeline in the same namespace, default to ml-pipeline.kubeflow:8887 as KFP endpoint.
I0703 15:00:08.667522      61 cache.go:116] Connecting to cache endpoint ml-pipeline.kubeflow:8887
I0703 15:00:08.710062      61 object_store.go:306] Cannot detect minio-service in the same namespace, default to minio-service.kubeflow:9000 as MinIO endpoint.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fe8fe8f9a50>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /simple/kfp/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fe8fea17110>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /simple/kfp/

To Reproduce
In a kf 1.8 env behind proxy, run the kfp_v2 uats

git clone https://github.com/canonical/charmed-kubeflow-uats.git
pip install -r charmed-kubeflow-uats/tests/notebooks/kfp_v2/requirements.txt
run kfp-v2-integration.ipynb

Environment
Model     Controller            Cloud/Region  Version  SLA          Timestamp
kubeflow  openstack-<REDACTED>  k8s/default   3.4.3    unsupported  16:12:37Z

SAAS                             Status   Store                 URL
alertmanager-karma-dashboard     active   openstack-<REDACTED>  admin/cos.alertmanager-karma-dashboard
grafana-dashboards               active   openstack-<REDACTED>  admin/cos.grafana-dashboards
loki-logging                     active   openstack-<REDACTED>  admin/cos.loki-logging
prometheus-receive-remote-write  active   openstack-<REDACTED>  admin/cos.prometheus-receive-remote-write
prometheus-scrape                active   openstack-<REDACTED>  admin/cos.prometheus-scrape
scrape-interval-config-metrics   blocked  openstack-<REDACTED>  admin/cos.scrape-interval-config-metrics
scrape-interval-config-monitors  blocked  openstack-<REDACTED>  admin/cos.scrape-interval-config-monitors

App                        Version                  Status  Scale  Charm                    Channel          Rev  Address        Exposed  Message
admission-webhook                                   active      1  admission-webhook        1.8/stable       301  a.b.c.d  no
argo-controller                                     active      1  argo-controller          3.3.10/stable    424  a.b.c.d  no
dex-auth                                            active      1  dex-auth                 2.36/stable      422  a.b.c.d   no
envoy                      res:oci-image@cc06b3e    active      1  envoy                    2.0/stable       194  a.b.c.d   no
grafana-agent-kubeflow     0.40.4                   active      1  grafana-agent-k8s        latest/edge       80  a.b.c.d   no
istio-ingressgateway                                active      1  istio-gateway            1.17/stable     1000  a.b.c.d   no
istio-pilot                                         active      1  istio-pilot              1.17/stable     1011  a.b.c.d  no
jupyter-controller                                  active      1  jupyter-controller       1.8/stable       849  a.b.c.d   no
jupyter-ui                                          active      1  jupyter-ui               1.8/stable       858  a.b.c.d     no
katib-controller           res:oci-image@31ccd70    active      1  katib-controller         0.16/stable      576  a.b.c.d   no
katib-db                   8.0.36-0ubuntu<IP_ADDRESS>  active      1  mysql-k8s                8.0/stable       153  a.b.c.d  no
katib-db-manager                                    active      1  katib-db-manager         0.16/stable      539  a.b.c.d   no
katib-ui                                            active      1  katib-ui                 0.16/stable      422  a.b.c.d    no
kfp-api                                             active      1  kfp-api                  2.0/stable      1283  a.b.c.d   no
kfp-db                     8.0.36-0ubuntu<IP_ADDRESS>  active      1  mysql-k8s                8.0/stable       153  a.b.c.d  no
kfp-metadata-writer                                 active      1  kfp-metadata-writer      2.0/stable       334  a.b.c.d   no
kfp-persistence                                     active      1  kfp-persistence          2.0/stable      1291  a.b.c.d    no
kfp-profile-controller                              active      1  kfp-profile-controller   2.0/stable      1315  a.b.c.d  no
kfp-schedwf                                         active      1  kfp-schedwf              2.0/stable      1302  a.b.c.d   no
kfp-ui                                              active      1  kfp-ui                   2.0/stable      1285  a.b.c.d   no
kfp-viewer                                          active      1  kfp-viewer               2.0/stable      1317  a.b.c.d  no
kfp-viz                                             active      1  kfp-viz                  2.0/stable      1235  a.b.c.d   no
knative-eventing                                    active      1  knative-eventing         1.10/stable      353  a.b.c.d   no
knative-operator                                    active      1  knative-operator         1.10/stable      328  a.b.c.d    no
knative-serving                                     active      1  knative-serving          1.10/stable      409  a.b.c.d   no
kserve-controller                                   active      1  kserve-controller        0.11/stable      523  a.b.c.d  no
kubeflow-dashboard                                  active      1  kubeflow-dashboard       1.8/stable       454  a.b.c.d    no
kubeflow-profiles                                   active      1  kubeflow-profiles        1.8/stable       355  a.b.c.d  no
kubeflow-roles                                      active      1  kubeflow-roles           1.8/stable       187  a.b.c.d    no
kubeflow-volumes           res:oci-image@2261827    active      1  kubeflow-volumes         1.8/stable       260  a.b.c.d   no
metacontroller-operator                             active      1  metacontroller-operator  3.0/stable       252  a.b.c.d    no
minio                      res:oci-image@1755999    active      1  minio                    ckf-1.8/stable   278  a.b.c.d   no
mlflow-minio               res:oci-image@1755999    active      1  minio                    ckf-1.7/stable   214  a.b.c.d  no
mlflow-mysql               8.0.36-0ubuntu<IP_ADDRESS>  active      1  mysql-k8s                8.0/stable       153  a.b.c.d  no
mlflow-server                                       active      1  mlflow-server            2.1/stable       466  a.b.c.d   no
mlmd                       res:oci-image@44abc5d    active      1  mlmd                     1.14/stable      127  a.b.c.d  no
oidc-gatekeeper                                     active      1  oidc-gatekeeper          ckf-1.8/stable   350  a.b.c.d    no
pvcviewer-operator                                  active      1  pvcviewer-operator       1.8/stable        30  a.b.c.d  no
resource-dispatcher                                 active      1  resource-dispatcher      1.0/stable        93  a.b.c.d   no
seldon-controller-manager                           active      1  seldon-core              1.17/stable      664  a.b.c.d    no
tensorboard-controller                              active      1  tensorboard-controller   1.8/stable       257  a.b.c.d   no
tensorboards-web-app                                active      1  tensorboards-web-app     1.8/stable       245  a.b.c.d    no
training-operator                                   active      1  training-operator        1.7/stable       347  a.b.c.d   no

Relevant Log Output
$ kubectl get all -n dash
NAME                                                  READY   STATUS      RESTARTS   AGE
pod/condition-v2-bcjmm-1791838033                     0/2     Completed   0          3m40s
pod/condition-v2-bcjmm-2085347550                     0/2     Completed   0          4m
pod/condition-v2-bcjmm-2542164255                     2/2     Running     0          3m30s
pod/ml-pipeline-ui-artifact-6b89ccc469-djz6v          2/2     Running     0          5d5h
pod/ml-pipeline-visualizationserver-955b54775-l9v7p   2/2     Running     0          4d20h
pod/test-dash-2-0                                     2/2     Running     0          9m54s

NAME                                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/ml-pipeline-ui-artifact           ClusterIP   <IP_ADDRESS>    <none>        80/TCP     5d5h
service/ml-pipeline-visualizationserver   ClusterIP   <IP_ADDRESS>   <none>        8888/TCP   5d5h
service/test-dash-2                       ClusterIP   <IP_ADDRESS>     <none>        80/TCP     9m54s

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ml-pipeline-ui-artifact           1/1     1            1           5d5h
deployment.apps/ml-pipeline-visualizationserver   1/1     1            1           5d5h

NAME                                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/ml-pipeline-ui-artifact-6b89ccc469          1         1         1       5d5h
replicaset.apps/ml-pipeline-visualizationserver-955b54775   1         1         1       5d5h

NAME                           READY   AGE
statefulset.apps/test-dash-2   1/1     9m54s

Additional Context
No response

Hi @nishant-dash ,

Can you provide the logs you get from the notebook execution? It'd be useful to see in which step the notebook is failing to see what part of it is trying to reach the internet.
Have you tried configuring the pipeline as linked in the example notebook in our how to guide? If you have a different configuration please provide us with that as well.

In the meantime, we are prioritizing this issue and will try to reproduce it.

@nishant-dash also for reproducing the issue, could you please tell us which method for running the UATs are you using? It was not clear from the issue description.
a) From inside a notebook
b) Using the driver

I got access from @nishant-dash to the env in question and was able to investigate
The error from the pipeline runner pod after it went to error state was as follows:
I0709 12:16:46.169293      69 cache.go:139] Cannot detect ml-pipeline in the same namespace, default to ml-pipeline.kubeflow:8887 as KFP endpoint.
I0709 12:16:46.169310      69 cache.go:116] Connecting to cache endpoint ml-pipeline.kubeflow:8887
I0709 12:16:46.207929      69 object_store.go:306] Cannot detect minio-service in the same namespace, default to minio-service.kubeflow:9000 as MinIO endpoint.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5db14d3b50>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /simple/kfp/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5db14e2310>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /simple/kfp/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5db14e2810>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /simple/kfp/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5db14e2bd0>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /simple/kfp/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5db14e2f10>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /simple/kfp/
ERROR: Could not find a version that satisfies the requirement kfp==2.4.0 (from versions: none)
ERROR: No matching distribution found for kfp==2.4.0
I0709 12:22:55.223155      69 launcher_v2.go:151] publish success.
F0709 12:22:55.223210      69 main.go:49] failed to execute component: exit status 1
time="2024-07-09T12:22:55.240Z" level=info msg="sub-process exited" argo=true error="<nil>"
Error: exit status 1
time="2024-07-09T12:22:56.202Z" level=info msg="sub-process exited" argo=true error="<nil>"
Error: exit status 1

It is clear in the logs that the container was not able to install kfp package, specifically :
ERROR: Could not find a version that satisfies the requirement kfp==2.4.0 (from versions: none)
ERROR: No matching distribution found for kfp==2.4.0

The Retrying logs above are from pip trying to install the package and failing. This is because the container is not able to reach the internet.
The default base image for pipeline runners does not have kfp package installed, it will try to install it every time. We have previously seen this in airgapped testing (see this comment).

Discussed with @nishant-dash
I have tested the fix in the environment in which it was failing for them, so we can close the issue.
If you hit this again, please report it and re-open the issue.


your train.txt is the train.txt in the VOCdevkit/ImageSets/Segmentation/ ?

I use the train.txt in Segmentation and main. I have an error
File "train.py", line 243, in
main()
File "train.py", line 130, in main
coord)
File "/data/tensorflow-deeplab-resnet-master/deeplab_resnet/image_reader.py", line 166, in init
self.image, self.label = read_images_from_disk(self.queue, self.input_size, random_scale, random_mirror)
File "/data/tensorflow-deeplab-resnet-master/deeplab_resnet/image_reader.py", line 116, in read_images_from_disk
img_r, img_g, img_b = tf.split(split_dim=2, num_split=3, value=img)

Train.txt is the one provided by the DeepLab team with augmented dataset. The links to the dataset are provided in README.


mybatis doesn't support two same column names

This is the sql(The database is MySQL) : select c.id ,s.id  from class c, student s  where c.id = 102.
The expected result should be:
id         id
102    4035
102    4132
102    4057
But in fact the result is:
id
102
102
102
In the XML file, the 'resultType' is 'HashMap'.
Then I set  the 'resultType' as 'IdentityHashMap'.
so the result becomes like this:
id          id
102     102
102     102
102     102
This problem may be caused by the Class 'DefaultResultSetHandler',on the line 416.

That's how JDBC works.
Here's a quote from ResultSet documentation.

When a getter method is called with a column name and several columns have the same name, the value of the first matching column will be returned.

Alright.  Thanks for commenting ! @harawata .

You are welcome :smile:


Normalization in train_dt

Hello,
I was checking the train_dt function since my results are not good.
input_type = config.get('input_type', 'coord')
    states = np.concatenate(states, axis=0)
    if input_type == 'grayscale':
        # no normalization needed for cnn
        state_mean, state_std = np.array(50), np.array(100)
    else:
        state_mean, state_std = np.mean(states, axis=0), np.std(states, axis=0) + 1e-6
        state_mean[[0,5,10,15,20]] = 0
        state_std[[0,5,10,15,20]] = 1
    num_timesteps = sum(traj_lens)

what's the meaning of the normalization from 0 to 20 with a step of 5?
Thank you

Hello, are you using images as data input? Can i ask you how much data volume are you using?

Thanks for your reply, I also think the image data works better(i run mlp_decision_transformer-expert-mcts.py), but I still encountered a problem when saving the model there is no save_models folder, so checkpoint can't be saved, how do you deal with the problem of no check_point in the initial training?

Hello, may I ask which experimental script you are using？according to your description, you should be using the PPO algorithm as the expert data provider, I use the mcts method to provide expert data, the amount of this data is small.So I wonder how much data the authors use to train the Transformer, as far as I know, the Transformer needs a huge dataset to support, I don't know if there is any relationship between the poor training results and small dataset.

Hello, I actually created my dataset with a DQN agent that I trained. Collisions may happen. I am aware of the size of data that transformers need, indeed I am curious about how many data did they gather to reach those results. But I have two questions:

Which machine did they use to train such a large amount of data? It took 22h to train on 15500 episodes, with bad results.
How many days of training were necessary to train?

By the way, I have no problem saving the last checkpoint. Have you checked the code in pipeline.train_dt? Are you able to save the intermediate checkpoints?

Hi, I think the number of expert training steps as well as the amount of saved data should be increased for better results, also maybe the PPO algorithm will give better results if it is an image input.

The code only saves 5 episodes’ data, which is not nearly enough.
`    for episode in range(5):
obs, info = env.reset()
done = truncated = False
total_reward = 0
episode_data = []
    while not (done or truncated):
        action, _ = model.predict(obs)
        obs, reward, done, truncated, info = env.step(action)
        total_reward += reward
        episode_data.append([obs, action, reward])
        env.render()
    env.close()
    all_data.append(episode_data)
    print('Episode: ', episode, ', Crashed?: ',
            info['crashed'], 'Total Reward:', total_reward)

    np.save('ppo_transformer_pc.npy', np.array(all_data, dtype=object), allow_pickle=True)

env.close()`

The code only saves 5 episodes’ data, which is not nearly enough.
`    for episode in range(5):
obs, info = env.reset()
done = truncated = False
total_reward = 0
episode_data = []
    while not (done or truncated):
        action, _ = model.predict(obs)
        obs, reward, done, truncated, info = env.step(action)
        total_reward += reward
        episode_data.append([obs, action, reward])
        env.render()
    env.close()
    all_data.append(episode_data)
    print('Episode: ', episode, ', Crashed?: ',
            info['crashed'], 'Total Reward:', total_reward)

    np.save('ppo_transformer_pc.npy', np.array(all_data, dtype=object), allow_pickle=True)

env.close()`

The code only saves 5 episodes’ data, which is not nearly enough.
`    for episode in range(5):  # 只有5组数据
obs, info = env.reset()
done = truncated = False
total_reward = 0
episode_data = []
while not (done or truncated):
action, _ = model.predict(obs)
obs, reward, done, truncated, info = env.step(action)
total_reward += reward
episode_data.append([obs, action, reward])
env.render()
env.close()
all_data.append(episode_data)
print('Episode: ', episode, ', Crashed?: ',
info['crashed'], 'Total Reward:', total_reward)
    np.save('ppo_transformer_pc.npy', np.array(all_data, dtype=object), allow_pickle=True)

env.close()`

The code only saves 5 episodes’ data, which is not nearly enough.
`    for episode in range(5):
obs, info = env.reset()
done = truncated = False
total_reward = 0
episode_data = []
    while not (done or truncated):
        action, _ = model.predict(obs)
        obs, reward, done, truncated, info = env.step(action)
        total_reward += reward
        episode_data.append([obs, action, reward])
        env.render()
    env.close()
    all_data.append(episode_data)
    print('Episode: ', episode, ', Crashed?: ',
          info['crashed'], 'Total Reward:', total_reward)

    np.save('ppo_transformer_pc.npy', np.array(all_data, dtype=object), allow_pickle=True)

env.close()`

Hello,I also experimented using kinematic data but the results were equally unsatisfactory, the first time I achieved a return of 41.3 in only 1000 rounds but the second time the return was only 28.1, also I would like to ask what is the use of                            'env_targets': [0.8, 1.0, 1.3],why should return_to_go be set to a constant value when testing?

Hello, I have had some improvement using an increased kinematic observation. Now the loss is decreasing instead of increasing, but in inference the ego just go straight. I noticed that it often tries to go to the left even when in the leftmost lane.
I also found this paper that gives some more technical insights about DT parameters: https://openreview.net/forum?id=vpV7fOFQy4
Regarding the train, I have launched a 100k steps training. 5000 steps per iteration. I am still not sure about the env targets, I would like the authors to answer about that!

Hello,thank you for your reply.I would like to ask How did you change the 5*5 observation? I've been using grayscale image inputs lately, but have found that the trained agents generally go straight and don't avoid obstacles, and I'm guessing that maybe it's because of your expert data can't avoid obstacles.