TabBar enhancement

I have 2 items in TabBar and for both of the items there is a different button at the footer, I want to get the selectedIndex property from the TabBar to conditionally render each button in the footer, right now selectedIndex always stays 0
See these buttons (Preview Post, Post), I want to show them in the footer, and for that I have to conditionally render each button with TabBar selectedIndex

@sharjeelyunus Now I tested it and it looks like it's working fine. Try the following code
View:
  styles:
    useSafeArea: true
  body:
    Column:
      children:
        - TabBar:
            id: tabbarId
            styles:
              tabPadding: 10 20 0 20
              indicatorThickness: 4
              indicatorColor: green
              activeTabColor: blue
              tabBackgroundColor: white
              dividerColor: white
            items:
              - label: First Tab
                widget:
                  Column:
                    styles: { padding: 30, crossAxis: center }
                    children:
                      - Text:
                          text: First Tab content
                      - ContentView:
                          inputs:
                            selectedIndex: ${tabbarId.selectedIndex}
              - label: Second Tab
                widget:
                  Column:
                    styles: { padding: 30, crossAxis: center }
                    children:
                      - Text:
                          text: Second Tab content
                      - ContentView:
                          inputs:
                            selectedIndex: ${tabbarId.selectedIndex}

ContentView:
  inputs:
    - selectedIndex
  body:
    Column:
      styles: { gap: 30, crossAxis: center }
      children:
        - Text:
            text: Tab content ${selectedIndex}
        - Conditional:
            conditions:
              - if: ${selectedIndex == 0}
                Button:
                  label: Preview Post
                  onTap: |
                    //@code
                    console.log(tabbarId.selectedIndex);
              - else:
                Button:
                  label: Post
                  onTap: |
                    //@code
                    console.log(tabbarId.selectedIndex);

I need to use these buttons/condition inside the footer (outside TabBar), inside the tab bar you don't even need conditions, you can just dynamically render 2 widgets for each item
see this here, even though I'm changing the item, the selectedIndex stays 0 in footer and always shows the Preview Post button:
View:
  styles:
    useSafeArea: true

  footer:
    styles:
      height: 100
      padding: 24
    children:
      - Column:
          styles:
            crossAxis: stretch
          children:
            - Conditional:
                conditions:
                  - if: ${tabbarId.selectedIndex == 0}
                    Button:
                      label: Preview Post
                  - else:
                    Button:
                      label: Post
  body:
    Column:
      children:
        - TabBar:
            id: tabbarId
            styles:
              tabPadding: 10 20 0 20
              indicatorThickness: 4
              indicatorColor: green
              activeTabColor: blue
              tabBackgroundColor: white
              dividerColor: white
            items:
              - label: First Tab
                widget:
                  Column:
                    styles: { padding: 30, crossAxis: center }
                    children:
                      - Text:
                          text: First Tab content
                      - ContentView:
                          inputs:
                            selectedIndex: ${tabbarId.selectedIndex}
              - label: Second Tab
                widget:
                  Column:
                    styles: { padding: 30, crossAxis: center }
                    children:
                      - Text:
                          text: Second Tab content
                      - ContentView:
                          inputs:
                            selectedIndex: ${tabbarId.selectedIndex}

ContentView:
  inputs:
    - selectedIndex
  body:
    Column:
      styles: { gap: 30, crossAxis: center }
      children:
        - Text:
            text: Tab content ${selectedIndex}
        - Conditional:
            conditions:
              - if: ${selectedIndex == 0}
                Button:
                  label: Preview Post
                  onTap: |
                    //@code
                    console.log(tabbarId.selectedIndex);
              - else:
                Button:
                  label: Post
                  onTap: |
                    //@code
                    console.log(tabbarId.selectedIndex);

@vusters
Currently, we can't able to access any of the widget's properties with the widget id from the footer right?
View:
  styles:
    useSafeArea: true
  body:
    Column:
      children:
        - Text:
            id: myTextId
            text: Second

  footer:
    styles:
      height: 100
      padding: 24
    children:
      - Text:
          text: ${myTextId.text}

Do we need to create footer as an invokable widget in order to get other widget's properties?
OR
Can we create the whole page or page group widget as a invokable one?

@vusters Vinoth is waiting for your comment for 6 weeks now

@kmahmood74 Sneh is working on this similar issue I guess. Something related to the scopeManager issue.


Basic travis CI integration

I'm working on C bindings for Hazelcast (wrapping the C++ client) and for my project I created a travis CI integration (https://travis-ci.org). As there are no .deb packages available for the Hazelcast C++ client, I do a build of a selected Hazelcast C++ client anyway during my project.
I decided to split that part out and contribute it back to you guys, are you interested?

Currently I run the build against two compilers, clang and g++. We can easily add more versions of the two.
I think it also makes sense to run the unit tests on travis, but I haven't looked into that yet, if you also think that's a good idea, I'm happy to add that to this PR.
In order for this to work, you need to enable travis CI integration in this Github project, as described here: https://docs.travis-ci.com/user/getting-started/

How a build looks like can be seen here: https://travis-ci.org/maxbeutel/hazelcast-cpp-client
The build is triggered with every commit to any branch.
NOTE the Build Status Icon shows the status of my forked repository, so you can get an idea of how it will look like. The URL should be changed when merging into master to point to hazelcast/hazelcast-cpp-client repository.
EDIT I haven't signed the contributors agreement yet, can do so on Monday if this PR has a chance of being merged.

Hello Max, thanks for contributing to our C++ client. I am very excited with this effort.
As it regards to the Travis CI, we already have a CI in place and it has a lot of builds for testing the C++ client. It is running on Jenkins. We can integrate your project into our build system by creating a build for your project, hence you will not need the Travis integration at all unless you have some good reason to use it. Does this sound good?
Please go ahead and fill the contributor agreement.
We can also get your C-client become more visible to the community by putting it into our incubator program if you want. The details are here: http://blog.hazelcast.com/incubator/ , https://github.com/hazelcast-incubator

Hello @ihsandemir
thanks a lot for your feedback. I wasn't aware that you already have a build infrastructure in place. As I understand it's also quite a bit more powerful than my travis CI integration.
The upside of the travis CI integration is of course that the build status can be made available to the public and you can also get contributions to improve/change the build.
But I can also understand that you don't want to maintain another build infrastructure besides your jenkins. Feel free to close this PR then. :)
Regardless of what happens with this PR, I can still sign the contributors agreement, just in case.
With regards to the C client, I would probably create a separate issue for collaboration on this issue. Integrating my project into your build would be of course super helpful.
The current state (which is not yet impressive) can be seen at https://github.com/maxbeutel/hazelcast-c-client - I just wrapped a few basic Map operations, which worked quite nicely.
Currently I'm still thinking about how to wrap the C++ API in a nice and user-friendly way. I summed up my points in a Stackoverflow question http://programmers.stackexchange.com/questions/317851/idiomatic-wrapping-of-c-template-type-api-in-c maybe you also got an idea?
As for featuring it in the community, that's very good idea, but I think it's too early, I would like to have a basic, working implementation and a general direction decided first, currently there is not that much to show yet, haha.
Thanks a lot so far!

Hello @maxbeutel
When we setup your build, you will be able to see the build status (people in github system will also see it. You can check hazelcast builds and PR details to see jenkins build status). Therefore, you should be able to get the same contribution from the community since your project will be public.
I have some ideas how to make the C API, i will write as an answer to your stackoverflow question briefly but we can also discuss and come up with a user friendly, easy to use API. API design is really not easy. We spend a lot of time to make it as intuitive and as easy as possible to use. It is a good design if you can make it so that it will be difficult for a user to misuse a method.
Regarding the community, it is OK to have an incomplete version initially, it will just make people aware of your project and you may get very useful feedbacks and may get some contribution. I looked at your project, it is in initial phase but that is no problem at all for putting it in the incubator. We can even come up with a roadmap (such as versions 0.1, 0.2, 0.3, etc.) and you can work on a roadmap which we do for other new client languages. Therefore, I would suggest just to add it to the incubator.

Hello @ihsandemir
Thank you very much for your offers.
I think it makes sense to close this PR and rather integrate any C bindings with the existing build infrastructure.
Milestones/Roadmap would also pretty great, because currently I can't yet oversee which functionality should be implemented for the minimal viable version.
I read your reply on Stackoverflow, providing an API to store binary sounds like an excellent idea, I wouldn't have come up with that. I will try to make this work this week to try it out.
How should we proceed to discuss collaboration? Shall I open an issue here in GitHub? I can also be reached via Skype at max.beutel maybe we ca have a chat whenever time allows for you.
Thanks a lot for taking the time to provide that much feedback! Much appreciated.

@maxbeutel Yes, you can open issue here to discuss but it may be better to discuss directly on the project. This is also why I told you that we can put it to inside the incubator (https://github.com/hazelcast-incubator), if we have the project at incubator, then we can just open the issues directly on your project under incubator and put the roadmap there. We can also have the implementation reviewed directly there. That would be just like any other client that we develop, and other people from team may have very useful feedbacks as well.
I will add you to my Skype. Surely, we can have a chat to get you started.


Does not actually run as non-root on Fly.io

The Dockerfile creates a headscale user and sets up permissions such that the user (should) be able to perform all operations on the file system that it needs to, but I realized the USER command is not sufficient to run as non-root on Fly.io.
Looking at the App configuration (fly.toml) documentation, I don't see an option to run the app as non-root. I suspect this may be a design decision by Fly.io, and running as non-root there would only be to uphold industry best practices but to no real end. 🤔

Duh, I mixed up the user that you get with fly ssh console with the user that your application process actually gets spawned as. All good, it runs as the headscale user.
❯ fly ssh console
Connecting to <IP>... complete
5683947f641018:/etc/headscale# ps aux | grep heads
  327 headscal  0:08 litestream replicate -exec headscale serve
  356 headscal  0:10 headscale serve
  413 root      0:00 grep heads


Use bluebird promises to improve performance

Built in node.js Promises are 4x slower than blue bird, It's better to bring in the package bluebird and add the line js const Promise = require('bluebird');
I can do a PR if you think it's warranted, but it's probably a good idea overall to do it.

I was wondering about Bluebird, but wasn't aware it was actually faster. Good to know. It's already being used by some of Postleaf's deps, so I'm don't mind adding it to package.json. A PR would be great.
We could probably use Promise.mapSeries to address #5 as well.


code is not getting generated on Mac OS Catalina

Describe the bug
I pulled down the code, followed the steps to install. Front end is running but not generating any html code nor anything in the terminal in terms of logs. Help here would be appreciated.
To Reproduce
Steps to reproduce the behavior:

Run backend
Run frontend
navigate to localhost url
Drag and drop a screenshot
Spinners running but nothing generates
Error shown on screen that unable to generated code, plz check logs
but no logs are getting generated
When I navigate to 127.x.x.x url, nothing shows as if backend is not connecting to front end  maybe?

Screenshots of backend AND frontend terminal logs
Below screenshots of terminal windows showing frontend and backend:

Was able to resolve the issue by fixing the pydantic version in the docker file.

Awesome, we've got to fix the pydantic version issue in the main repo. Thanks for sharing the solution!

@abi Alternately, add an additional instruction to fix pydantic version before running backend


State handling

While the state is handled correctly during parsing using pushState and popState, this data is not used for the buildDom function. This leads to properties which are applied within a paragraph but never removed when popState is called.
Consider the following example: {\rtf1\ansi\ansicpg1252 {Some Text }{\i italic text}{ text which shouldn't be italic} }
During parsing the \i keyword is applied, but when calling popState at the curly bracket it is not removed so the rest of the text is also italic.

I think there are also some more fundamental issues, e.g. how \par is handled.  I have started work on adding support for tables and at least the \par issue is causing some major problems, resulting in me having to re-architect quite a bit.  This issue might actually be fixed in my branch, but I haven't verified.

I pushed my table-support branch, but note that it has lots of issues and doesn't quite work yet.

I just tested your branch, the issue still exists in the current version.

popState needs to update the chp used by buildDom.


[Material] When tapping on DropdownButtonFormField, dropdown stays highlighted

https://github.com/flutter/flutter/pull/93716 request on when tapping on Dropdown, since DropdownButtonFormField uses Dropdown,  due to this, the dropdown stays highlighted. which is an unintended behavior.

minimal code sample
import 'package:flutter/material.dart';

void main() => runApp(const MyApp());

class MyApp extends StatelessWidget {
  const MyApp({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: const MyHomePage(),
    );
  }
}

class MyHomePage extends StatefulWidget {
  const MyHomePage({Key? key}) : super(key: key);

  @override
  _State createState() => _State();
}

class _State extends State<MyHomePage> {
  int? dropdownValue = 1;
  final List<int> hugeMenuItems = List<int>.generate(100, (int index) => index);
  late FocusNode focusNode;

  @override
  void initState() {
    super.initState();
    focusNode = FocusNode();
  }

  @override
  void dispose() {
    focusNode.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: const Text('Title'),
      ),
      body: Column(
        children: <Widget>[
          const TextField(),
          DropdownButton(
              focusNode: focusNode,
              value: dropdownValue,
              onTap: () {
                print('FocusNode has focus: ${focusNode.hasPrimaryFocus}');
              },
              items: hugeMenuItems.map<DropdownMenuItem<int>>((int item) {
                return DropdownMenuItem<int>(
                  key: ValueKey<int>(item),
                  value: item,
                  child: Text(item.toString(),
                      key: ValueKey<String>('Text $item')),
                );
              }).toList(),
              onChanged: (int? value) {
                setState(() => dropdownValue = value);
              }),
        ],
      ),
    );
  }
}

Also address https://github.com/flutter/flutter/pull/93716#issuecomment-985471665

Ideally it would support TextInputAction.go or something like that.

Highlight is working as intended as it seems, it should stay highlighted, see
The main issue is the highlight is not smooth or persistent

Looks like focus moves away whenever there is new route (here when popup shows up in a route)
https://github.com/flutter/flutter/blob/1c673b28ad90d9d2f5daf51c5fbd3e9bba063e49/packages/flutter/lib/src/material/dropdown.dart#L1312-L1317
I think it is core framework issue than this widget


feat: add batch operations for elastic

Goals:

...Add batch operations for elastic backend
https://docarray.jina.ai/advanced/document-store/extend/
...
[ ] check and update documentation, if required. See guide

We closed this PR as we are not working on it anymore


install errors in google colab

Collecting pysurvival
Using cached pysurvival-0.1.2.tar.gz (4.7 MB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from pysurvival) (3.7.1)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from pysurvival) (1.25.2)
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from pysurvival) (1.5.3)
Requirement already satisfied: pip in /usr/local/lib/python3.10/dist-packages (from pysurvival) (23.1.2)
Collecting progressbar (from pysurvival)
Using cached progressbar-2.5.tar.gz (10 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: pyarrow in /usr/local/lib/python3.10/dist-packages (from pysurvival) (14.0.2)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from pysurvival) (1.2.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from pysurvival) (1.11.4)
Collecting sklearn (from pysurvival)
Using cached sklearn-0.0.post12.tar.gz (2.6 kB)
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Preparing metadata (setup.py) ... error
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

i tried also at my local at appears
ERROR: Could not install packages due to an OSError: [Errno 21] Is a directory: '/home/usr/anaconda3/envs/proj2/lib/python3.9/site-packages/numpy-1.23.5.dist-info/METADATA'

I got the same issue!


npm is adding a blank space at line 39 of package.json to fix indentation

16:29 $ git status
On branch test
nothing to commit, working tree clean

16:29 $ npm i
audited 2181 packages in 2.008s
found 26 vulnerabilities (18 low, 2 moderate, 6 high)
  run `npm audit fix` to fix them, or `npm audit` for details

16:29 $ git status
On branch test
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   package.json

no changes added to commit (use "git add" and/or "git commit -a")

16:29 $ git diff
diff --git a/package.json b/package.json
index d47383c..237ca3e 100644
--- a/package.json
+++ b/package.json
@@ -36,5 +36,5 @@
     "ignore": [
       "lib/context.js"
     ]
- }
+  }
 }

This seems to no longer be a problem. Closing this - if you still have issues with it @helio-frota please feel free to reopen.


GT06 protocol ignition decode

Hi Guys,
I'm showing below a gt06 0x22 packet that is supposed to include ignition state according to documentation but Traccar is  decoding ignition as "YES". Documentation below shows "00" is low state for ignition and the packet is reporting "00" as shown below. The install is configured to copy ignition attribute but this packet contains ignition so it should not be copying from older packet.

I have another example below which is the device sending a heartbeat packet (0x13) but traccar changes
ignition state from "YES" to "NO" based on that packet; but the heartbeat packet does not contain any ignition information. The previous ignition state was "YES"

Hi
socstur , its Marcio here,at this moment I cant help you, Im trying to start to build a new server to work with GT06, I got the protocol for it, but Im unable to read the data from gps, I already made a server for tk102 tk103 and its working very well,but the protocol for GT06 is binary, Im using VB6 (yes, its old), but I have no id of how to read the data from this device, ive changed the winsock to read the data in byte or byte() but I dont know what Im doing wrong because I cant read the data, I only get values likes "?c???????". can you help me with the first conection of data sent from terminal? any help from you it would be very very useful for me, thanks a lot

@MarcioBRSP, GT06 is a binary protocol, so you can't convert message into readable text.
Please don't spam everywhere with your questions. This is inappropriate place to ask. If you have any further questions please ask on forum.

i am little bit confused to send values on server in hex format..
can you please share how to send the values in hex format to server.


Unit test readme example, remove second function unnecessary docs

https://github.com/ltabis/rhai-autodocs/pull/13


[Source request] Ragnarok scanlation

Why did this got reopened, I closed this for a reason

Why did this got reopened, I closed this for a reason

New URL: https://ragnarokscanlation.org/


I keep getting "HTTP error code 500"

Describe the bug
After using a command to download my 'saved' Instagram posts,. I keep getting this error
To Reproduce
Steps to reproduce the behavior:
instaloader --login USERNAME --fast-update --no-captions --no-metadata-json --no-video-thumbnails --filename-pattern="{profile} - {shortcode}" ":saved" --dirname-pattern={profile}
Expected behavior
To download my Instagram 'saved' posts
Error messages and tracebacks
Fatal error: JSON Query to graphql/query: HTTP error code 500.
Instaloader version
4.2.10

I'm getting the same error over and over again. However, it works fine when I log in with a different account. So it appears to be a server-side issue that affects individual accounts. Obviously, there's no reason to expect that Instagram/Facebook will help to solve it. I hope that it at least will not spread to my other account, because creating new accounts every day to keep using Instaloader is not a viable solution.

Error code 500 means Internal Server Error, a problem on the server side which we cannot fix. If that error appears, Instaloader retries the request a few times, and if it continues to fail, it aborts.

I had the same issue. It seems to be related to the use of session file. After deleting the file, and entering the password as prompted it works. When launched again with the new sessionfile, same error as previous JSON Query to graphql/query: HTTP error code 500.

Why wont this be fixed? It can at least be added an option to re-create the session file if an error occurs. As explained by @SeNeOEH this can be fixed by deleting the session file and re-entering the credentials.

Why wont this be fixed? It can at least be added an option to re-create the session file if an error occurs.

Please, always feel free to create a pull request so that we can discuss your suggestions.


Fix autolayout support for BaseMessageCollrctionViewCell

Codecov Report

Merging #651 into master will increase coverage by 0.08%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #651      +/-   ##
==========================================
+ Coverage   72.84%   72.93%   +0.08%
==========================================
  Files          19       19
  Lines        1197     1197
==========================================
+ Hits          872      873       +1
+ Misses        325      324       -1

Impacted Files
Coverage Δ

...hatController/BaseChatViewController+Changes.swift
88.93% <0.00%> (+0.42%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f7da140...ca23143. Read the comment docs.


[BUG]: Handling of containers (e.g. std::vector) of Eigen::Ref

Problem description
Hello,
I would like to report a bug which happens under specific circumstances:

from-Python conversions to types like std::vector<Eigen::Ref<const MatrixXd>> (with the default, fixed inner stride),
when the Numpy arrays are (non-contiguous) slices of the same array with the same dimensions (see example code),
when using versions of Eigen predating this PR on gitlab (which added the much needed move constructors).

Following @WKarel's work on PR #215, points 1-2 will lead to the creation of a temporary owned object in the Ref::m_object protected member. This happens on this line, because DMap has dynamic strides, and the Eigen::Ref has fixed inner stride hence the temporary is created:
https://github.com/wjakob/nanobind/blob/8e5fd142d9791f3f7ff66485de479578e64d9f37/include/nanobind/eigen/dense.h#L467
I know of two mitigations for now:

using a version of Eigen that has the move constructors for Eigen::Ref
use nb::DRef which was built to avoid copies

Reproducible example code
using MatXdR = Eigen::Matrix<double, Eigen::Dynamic, Eigen::Dynamic, Eigen::RowMajor>;
using MatRefXdR = Eigen::Ref<const MatXdR>;
using Vec_MatRefXdR = std::vector<MatRefXdR>;
m.def("printRefs", [](const Vec_MatRefXdR& ms) {
     std::cout << "Got matrix\n" << ms[0] << std::endl;
     std::cout << "Got matrix\n" << ms[1] << std::endl;
      // passthrough
      std::vector<MatXdR> out(ms.size());
      for (size_t i = 0; i < ms.size(); i++) {
           out[i] = ms[i];
      }
      return out;
});

# in test_eigen.py
@needs_numpy_and_eigen
def test_vec_ref():
    print()
    np.set_printoptions(precision=4, linewidth=200)
    a = np.random.randn(8, 4).T  # .T ensures "wrong" order for slices
    s1 = a[:2]
    s2 = a[2:]
    print(s1)
    print(s2)
    r1, r2 = t.printRefs([s1, s2])
    print(r1)
    print(r2)
    assert np.allclose(s1, r1)
    assert np.allclose(s2, r2)

The code prints the following result (and the asserts fail)
[[-0.7264  0.4645 -1.3106  1.3194 -1.5611 -1.6445 -0.6314  0.6161]
 [ 0.6566 -0.5272  1.5168  1.2926  0.6595  0.6201 -0.8453  1.0318]]
[[-0.6287  0.4879 -0.8519 -0.2426 -0.0305  0.5116 -1.189  -1.3493]
 [-0.4122 -1.2882 -1.1801  1.0621  0.8644  1.1376 -0.7016  0.2041]]
Got matrix
 -0.628672    0.48795  -0.851887  -0.242632 -0.0305066   0.511559     -1.189   -1.34933
 -0.412173   -1.28822   -1.18013    1.06208   0.864423    1.13762  -0.701575   0.204068
Got matrix
 -0.628672    0.48795  -0.851887  -0.242632 -0.0305066   0.511559     -1.189   -1.34933
 -0.412173   -1.28822   -1.18013    1.06208   0.864423    1.13762  -0.701575   0.204068
[[-0.6287  0.4879 -0.8519 -0.2426 -0.0305  0.5116 -1.189  -1.3493]
 [-0.4122 -1.2882 -1.1801  1.0621  0.8644  1.1376 -0.7016  0.2041]]
[[-0.6287  0.4879 -0.8519 -0.2426 -0.0305  0.5116 -1.189  -1.3493]
 [-0.4122 -1.2882 -1.1801  1.0621  0.8644  1.1376 -0.7016  0.2041]]

Is this an issue that nanobind can actually fix? If not, it might be better to include a static_assert that errors out when compiling with an older version of Eigen.

I think there could be a workaround where the temp storage is managed by Eigen::Ref's type caster itself, which should be kept alive (we can gate this at compile-time behind a check for the version of Eigen, the next version of Eigen doesn't need that).
This means other casters (e.g. optional_caster or the list_caster) shouldn't create the inner caster on the fly like here but always store them as data members.
This would also propagate to collections - for instance the list caster would have to store a list of inner casters I guess - not sure about the memory footprint for this, either. (I tried this on my local checkout of nanobind and it works).

The change you propose to make casters store their member by default are likely impossible because this breaks creation of containers that store references (and it would in any case be too intrusive of a change, if "only" justified by the Eigen type caster). Perhaps a good stop-gap solution would be to raise a compile-time error message and ask users to upgrade their version of Eigen? If you agree, could you create a PR that adds such an error message?

The change you propose to make casters store their member by default are likely impossible because this breaks creation of containers that store references (and it would in any case be too intrusive of a change, if "only" justified by the Eigen type caster).

Hm, you are right it would be intrusive and create tons of problems, and looking back to how pybind11 works, it didn't do that, either.

Perhaps a good stop-gap solution would be to raise a compile-time error message and ask users to upgrade their version of Eigen? If you agree, could you create a PR that adds such an error message?

Unfortunately I don't think this could work. Lots of things depend on Eigen and would need consistent versions of it for interop, and the move ctor is still on the master branch (with no release in sight yet, I've asked around on the Discord).
Over the past week, what I've done instead was check how pybind11 did it, which is through managing a copy (a numpy copy created in C++ and managed with pybind's life support mechanism).
I think we can reproduce this in nanobind by creating a copy ndarray with the temp memory and the right strides (fitting the desired column- or row-major order in Eigen), and manage it through a capsule that's added to the cleanup arg in from_python. I'll open a PR so you can take a look.

I see that

the proposed changes are considerable, and only necessary until the next release of Eigen, while
it may take a long time for Eigen to publish its next release.

A more compact and local (but untested) alternative that comes to my mind could be: derive a struct from Ref<const> with a move-Ctor that moves Ref<const>::m_object. This should be possible, because Ref<const>::m_object is not private, but protected. Something along the lines of:
#if EIGEN_VERSION <= 3.4.0

template<class CRef_>
struct MovableCRef : CRef_
{
  using CRef_::CRef_;
  MovableCRef::MovableCRef(CRef_&& that)
  {
    this->m_object = std::move(that.m_object);
  }
  MovableCRef::MovableCRef(MovableCRef&& that)
  {
    this->m_object = std::move(that.m_object);
  }
};

template<class Ref_>
using MovableRef = std::conditional_t<Ref_::is_const, MovableCRef<Ref_>, Ref_>;

#else

template<class Ref_>
using MovableRef = Ref_;

#endif

And use/return MovableRef whereever Ref<const>::m_object must be moved. Since MovableRef does not add data members, slicing (when returning MovableRef) should be no problem. However, I'm unsure whether changes could be limited to dense.h with this alternative, or changes would also need to be introduced into detail/nb_list.h et al. - in which case this alternative would be unacceptable, I guess.

I see that

the proposed changes are considerable, and only necessary until the next release of Eigen, while
it may take a long time for Eigen to publish its next release.

I agree, the changes are considerable because they aim to reproduce the workaround that pybind11 uses, but I'm sure an alternative is possible.
The last release of Eigen was exactly three years ago (Aug 18, 2021), and it will be the only one available on some package managers (namely Debian, and apt for Ubuntu 22 and 24 LTS) for the foreseeable future. So if we gatekeep this feature to a future Eigen > 3.4 many users would be left out for very long.

A more compact and local (but untested) alternative that comes to my mind could be: derive a struct from Ref<const> with a move-Ctor that moves Ref<const>::m_object. This should be possible, because Ref<const>::m_object is not private, but protected. Something along the lines of:
#if EIGEN_VERSION <= 3.4.0

template<class CRef_>
struct MovableCRef : CRef_
{
  using CRef_::CRef_;
  MovableCRef::MovableCRef(CRef_&& that)
  {
    this->m_object = std::move(that.m_object);
  }
  MovableCRef::MovableCRef(MovableCRef&& that)
  {
    this->m_object = std::move(that.m_object);
  }
};

template<class Ref_>
using MovableRef = std::conditional_t<Ref_::is_const, MovableCRef<Ref_>, Ref_>;

#else

template<class Ref_>
using MovableRef = Ref_;

#endif

And use/return MovableRef whereever Ref<const>::m_object must be moved. Since MovableRef does not add data members, slicing (when returning MovableRef) should be no problem. However, I'm unsure whether changes could be limited to dense.h with this alternative, or changes would also need to be introduced into detail/nb_list.h et al. - in which case this alternative would be unacceptable, I guess.

Actually this is a great suggestion. I'll try this on my fork and see if/how it works.

@WKarel I tried your suggestion. Unfortunately it would require intrusive changes to nb_list.h to work (and I wouldn't know which kind of change to make), or MovableRef would need to be replace Eigen::Ref(...) everywhere in the wrapping code.
The problem here is on this line:
https://github.com/wjakob/nanobind/blob/517f3ee9ec989b818fac17920d70c19753407249/include/nanobind/stl/detail/nb_list.h#L51
In this context, the template parameters are List = std::vector<Eigen::Ref<const T>>, Entry = Eigen::Ref<const T>, and the called function will be vector::push_back(const Entry&) (the copy version since there is no move ctor here and since the list caster is not aware the actual object is a MovableCRef).

FYI, this could perhaps be changed by changing template <typename T_> using Cast = Ref; to template <typename T_> using Cast = movable_cast_t<T_>;

FYI, this could perhaps be fixed by changing template <typename T_> using Cast = Ref; declaration to template <typename T_> using Cast = movable_cast_t<T_>; within the Ref type caster. (And similar, for Map)

I tried this, changing the cast operator to operator MovableRef<Ref>&&(), but this triggered std::bad_alloc memory errors.

I have tried other things but I don't think we can avoid having to do the same trick as pybind11 - creating our own temp memory and tying it to the cleanup list. However I think we can gate this behaviour behind a check for a version of Eigen higher or equal to the current master branch (3.4.90).
What do you think @WKarel ?

I just tried it, and my idea does not work with STL containers using Visual C++, because e.g. std::vector::push_back uses std::allocator::construct, which copy-constructs its argument, since Ref<T const> has no move constructor.
But apart from how to bind std::vector<Ref<T const>> to Python, what is your use-case for it? Ref<T const>s copy constructor is generally not fit for use with STL containers, because it does not copy m_object. And without even a move constructor, what would you (safely) do with a vector of it?
Using Eigen's current release:
using MatXdC = Eigen::Matrix<double, Eigen::Dynamic, Eigen::Dynamic, Eigen::ColMajor>;
using MatXdR = Eigen::Matrix<double, Eigen::Dynamic, Eigen::Dynamic, Eigen::RowMajor>;
std::vector<Eigen::Ref<const MatXdR>> v;
{
  MatXdC m{ { 1., 2. } };
  Eigen::Ref<const MatXdR> r(m);  // r::m_object copy-initialized, and r points to it.
  v.push_back(std::move(r));  // v[0]::m_object not initialized. v[0] points to r::m_object.
}
v[0](0, 0);  // accesses destroyed memory.

I just tried it, and my idea does not work with STL containers using Visual C++, because e.g. std::vector::push_back uses std::allocator::construct, which copy-constructs its argument, since Ref<T const> has no move constructor.
But apart from how to bind std::vector<Ref<T const>> to Python, what is your use-case for it? Ref<T const>s copy constructor is generally not fit for use with STL containers, because it does not copy m_object. And without even a move constructor, what would you (safely) do with a vector of it?

I wasn't aware that Eigen::Ref<const T> was not fit for use with STL containers, because I didn't know about the m_object temporary.
We use STL containers of Refs in some places, namely optional<Eigen::Ref<const T>> in ProxQP for passing QP parameters as optional arguments.

Using Eigen's current release:
using MatXdC = Eigen::Matrix<double, Eigen::Dynamic, Eigen::Dynamic, Eigen::ColMajor>;
using MatXdR = Eigen::Matrix<double, Eigen::Dynamic, Eigen::Dynamic, Eigen::RowMajor>;
std::vector<Eigen::Ref<const MatXdR>> v;
{
  MatXdC m{ { 1., 2. } };
  Eigen::Ref<const MatXdR> r(m);  // r::m_object copy-initialized, and r points to it.
  v.push_back(std::move(r));  // v[0]::m_object not initialized. v[0] points to r::m_object.
}
v[0](0, 0);  // accesses destroyed memory.

I think std::optional behaves the same, causing this kind of undefined behaviour.

std::optional<Eigen::Ref<const T>> makes more sense to me. But indeed, the problem is the same.
As a work-around specifically for std::optional<Eigen::Ref<const T>>, you may define your own partial specialization of type_caster that constructs the Ref from a Map in-place:
NAMESPACE_BEGIN(NB_NAMESPACE)
NAMESPACE_BEGIN(detail)

template<typename T>
struct type_caster<std::optional<Eigen::Ref<const T>>> : optional_caster<std::optional<Eigen::Ref<const T>>>
{
  using Caster = typename type_caster::optional_caster::Caster;
  using Map = typename Caster::Map;
  using DMap = typename Caster::DMap;

  bool from_python(handle src, uint8_t flags, cleanup_list* cleanup) noexcept {
    if (src.is_none())
      return true;
    Caster caster;
    if (!caster.from_python(src, flags_for_local_caster<T>(flags), cleanup) ||
        !caster.template can_cast<T>())
      return false;
    if (caster.dcaster.caster.value.is_valid())
      value.emplace(caster.dcaster.operator DMap());
    else
      value.emplace(caster.caster.operator Map());
    return true;
  }
};

NAMESPACE_END(detail)
NAMESPACE_END(NB_NAMESPACE)

I am not in the position of accepting PRs here. The only way for nanobind itself to support what you need using the current release of Eigen seems to require your rather comprehensive changes, which @wjakob seems to at least dislike. So I've tried to provide a solution outside of it.
Eigen::Ref<const T> is a rather weird citizen, because it can either map its own memory, or some external one (which is hardly possible to tell from the outside), and when copying, the copy may reference the memory of the original, even if the original owns it. Hence, to be safe when making copies, one always needs to keep alive the originals.
That can be fine when creating copies on the stack (possibly as function arguments). But e.g. std::vector::emplace_back may re-allocate its array and move existing elements onto the new one. Without a proper move constructor, this will copy-construct Eigen::Ref<const T>s on the new array and delete the originals on the old one - bang! This may be prevented e.g. within nanobind by pre-allocating the needed capacity. However, I would think twice before broadly using std::vector<Eigen::Ref<const T>> in my own code.
I guess the intent for its unusual copy semantics is to make it cheap to call functions that expect Eigen::Ref<const T> as function argument (which it really is, unless its m_object is an array of large, static size). Otherwise, functions would need to expect const Eigen::Ref<constT>& as arguments - which is a little longer to read, but certainly fits much better into C++. Even the Eigen docs seem to prefer the latter.

I am not in the position of accepting PRs here. The only way for nanobind itself to support what you need using the current release of Eigen seems to require your rather comprehensive changes, which @wjakob seems to at least dislike. So I've tried to provide a solution outside of it.

That's fine then. I'll just apply fixes within our own codebase.

Eigen::Ref<const T> is a rather weird citizen, because it can either map its own memory, or some external one (which is hardly possible to tell from the outside), and when copying, the copy may reference the memory of the original, even if the original owns it. Hence, to be safe when making copies, one always needs to keep alive the originals. That can be fine when creating copies on the stack (possibly as function arguments). But e.g. std::vector::emplace_back may re-allocate its array and move existing elements onto the new one. Without a proper move constructor, this will copy-construct Eigen::Ref<const T>s on the new array and delete the originals on the old one - bang! This may be prevented e.g. within nanobind by pre-allocating the needed capacity. However, I would think twice before broadly using std::vector<Eigen::Ref<const T>> in my own code.

You're right, since vectors can move it would be dangerous. I think I will suggest to my org we outright ban STL vectors (and I guess, sets, and maps?) for Eigen::Ref<const...> outside of optional.

I guess the intent for its unusual copy semantics is to make it cheap to call functions that expect Eigen::Ref<const T> as function argument (which it really is, unless its m_object is an array of large, static size). Otherwise, functions would need to expect const Eigen::Ref<constT>& as arguments - which is a little longer to read, but certainly fits much better into C++. Even the Eigen docs seem to prefer the latter.

This makes sense, I usually do const Eigen::Ref<const T>& in my own code. I guess the copy semantics are a good safeguard when the user has to use something like a container

I think that Eigen::Ref<const T> should have its copy constructor deleted, but it is probably too late now.

Instead, there's an open MR on Gitlab which seeks to remove the temporary m_object instead (as an option in the template parameters).

What's the resolution? Shall I close the issue?

What's the resolution? Shall I close the issue?

We concluded that std::vector<Eigen::Ref> is in fact dangerous due to the temporary and weird copy semantics (even when there is a move ctor).
I had a similar issue downstream with std::optional which I've ended up solving by specializing the type_caster.
Thank you for the discussion, I'll be closing this issue and my PR!


WIP: update to section on ornaments

This PR updates the whole cmnornaments section to match the latest MEI development version.

All corresponding example files have been checked, updated and renamed to *.txt
Example files to other sections haven't been renamed yet to easily keep track of updated files
Added missing ornament glyphs in the text. (see #41)

@rettinghaus please let me know when this PR will be ready for review and merge. Until then I've changed the title to include "WIP" (Work-in-progress).

@ahankinson basically it's ready for review.


Round off issue for currency like ALL, AFN when requesting or sending money with 0.5 decimal

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Open the app
Open any report
Request money/send money in the chat with 0.5 decimal eg. 100.5, 210.5
Observe the amount after clicking next
Observe the amount in message generated by the request

Expected Result:
Both the amount should be equal i.e. round off amount after first step in request/send money and round off amount in message generated after the request should be equal
Actual Result:
Round off amount after first step displays the amount as rounded up and round off amount in message displays the amount as rounded down eg. if requested 100.5, it will display 101 in next step and 100 in message
Workaround:
unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[ ] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: v1.2.65-0
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos:
https://user-images.githubusercontent.com/43996225/216864479-b5c6a58d-fd8f-4158-b9cb-5055b088278b.mp4
https://user-images.githubusercontent.com/43996225/216864496-9eb28f82-3eda-414c-96d3-e6c7fba5f651.mov
Expensify/Expensify Issue URL:
Issue reported by: @dhanashree-sawant
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1675436047423269
View all open jobs on GitHub

I think this might be by design cos the currencies in question very seldom use the .5 decimal cos they are hyperinflated.
Gonna check in #expensif-bugs

Commented with some notes on original thread, I don't think this is a bug.

I am going to close this one, as it's an edge case and seldom encountered.


add new entry for im-operator-v4.0 in size and rules

/retest

issue: https://github.ibm.com/ibmprivatecloud/roadmap/issues/57206


mitigate the number of times we tag an image

According to @lavalamp, Workflow clusters in GKE are triggering this bug. This issue occurs because Workflow will re-tag an image even if no changes were made. We used to do this because we would inject environment metadata into the image when deis config:set was called. Now, we just inject that environment metadata directly into the k8s manifest, so no image modification (and therefore no image re-tagging) is required.
While the upstream bug will eventually get fixed, in the meantime we should investigate how we trigger this bug, and see if there is a way to mitigate this issue.

Doing kubectl get node -o yaml on one of the offending clusters shows a long list of re-tagged app images:
   - names:
      - gcr.io/google_containers/pause:2.0
      sizeBytes: 350164
    - names:
      - localhost:5555/test-57368379:v3
      - localhost:5555/test-225171282:v3
      - localhost:5555/test-349910451:v2
      - localhost:5555/test-765152779:v2
      - localhost:5555/test-531894577:v2
      - localhost:5555/test-531894577:v3
      - localhost:5555/test-957102600:v2
      - localhost:5555/test-957102600:v3
      - localhost:5555/test-101875602:v2
      - localhost:5555/test-736008182:v2
      - localhost:5555/test-312165996:v2
      - localhost:5555/test-312165996:v3
      - localhost:5555/test-966490384:v2
      - localhost:5555/test-92413502:v2
      - localhost:5555/test-953282707:v2
      - localhost:5555/test-578673730:v4
      - localhost:5555/test-846984163:v2
      - localhost:5555/test-19736339:v3
      - localhost:5555/test-539512357:v2
      - localhost:5555/test-316118051:v2
      - localhost:5555/test-457125150:v2
      - localhost:5555/test-270332264:v2
      - localhost:5555/test-270332264:v3
      - localhost:5555/test-84933414:v2
      - localhost:5555/test-84933414:v3
      - localhost:5555/test-241757411:v2
      - localhost:5555/test-241757411:v3
      - localhost:5555/test-778712360:v2
      - localhost:5555/test-778712360:v3
      - localhost:5555/test-778712360:v4
      - localhost:5555/test-848623511:v3
      - localhost:5555/test-159686861:v2
... [couple thousand lines omitted]

So clearly this isn't an immediate problem with Deis Workflow itself; rather it's an artifact of CI running many times and re-tagging similar images.
One approach to try here is garbage-collecting old images and tags when e2e runs. If that approach works out, we should investigate doing something similar within Workflow itself, since it is conceivable a similar problem could happen through normal usage over long periods of time.

We could be smarter in workflow about when we tag a new image - check if the pulled down image matches the last available one, if it does then we'd not increment.  This changes Release::image logic quite a bit tho.
Also only helps reduce the problem, doesn't do any sort of cleanup operation.
If we started doing some GC of images, that means that rolling back to releases where the image has been GC'ed is now out of bounds.

I think we fixed the original issue in deis/charts#352 so this ticket is very low priority at this point.

If we started doing some GC of images, that means that rolling back to releases where the image has been GC'ed is now out of bounds.

Isn't the image pulled into the local registry for the most part, though? Asides for private images using deis registry and deis pull, if we GC the local docker cache on the worker nodes, the image should still be present within the local private registry which the worker nodes can fetch from.
My train of thought is to stop tagging buildpack images as v2/v3/v4 as we continue to perform changes to the image and just use :git-sha as the tag, which is never updated until a new build is pushed to the registry via git push deis master
In either case, still a micro optimization so this is really low priority and no fix is really necessary in the immediate future. The main issue was what @mboersma mentioned, which was that our e2e clusters weren't cleaning up app images from old e2e runs.

Yeah, it should be in the local registry, good point.
I think the bigger issue would be deis pull rather than git push as the latter has sha available as you pointed out. Build Packs already are downloaded as slugs using the slugrunner image and Dockerfile apps use a :git-<sha> tag on images as it stands

just removing this from the milestone since the root issue was fixed in deis/charts#352, which was merged in for v2.7

This issue was moved to teamhephy/controller#50


Bug in plot_maneuver

This cannot be correct:
https://github.com/poliastro/poliastro/blob/80c7de132c42c751d47fbb495dc54fb3e3460f12/src/poliastro/plotting/_base.py#L171-L173
If the len() is zero, then one cannot index it.

Lazy reviewer's fault ✋🏽 Good catch @MLopez-Ibanez


Fix amount of time fix sprint review/release/retro/planning

Allocate time for each of the scrum meeting components
Proposal:
review: 30
release: 5
retro: 25
planning: 30

leaving the ticket open also for sprint 6

closing as time management is efficient (for now)


[python-package] UNKNOWN python package

Description
I have used cmake to precompiled a .so file. But when I use build-python to create python support. It ended up create a unknown-0.0.0 package.
Reproducible example
sudo apt-get install --no-install-recommends git cmake build-essential libboost-dev libboost-system-dev libboost-filesystem-dev
sudo mkdir -p /mnt/workspace
sudo chown $(whoami):$(whoami) /mnt/workspace
cd /mnt/workspace

git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
cmake -B build -S . -DUSE_GPU=1
cmake --build build -j$(nproc)

apt-get install python3-venv
pip install setuptools numpy scipy scikit-learn -U
sudo sh ./build-python.sh install --precompile

Environment info
LightGBM version or commit hash:
Command(s) you used to install LightGBM
root@5567918b09b1:/mnt/workspace/LightGBM# ./build-python.sh install --precompile
building lightgbm
Requirement already satisfied: build>=0.10.0 in /usr/local/lib/python3.10/dist-packages (1.2.2.post1)
Requirement already satisfied: pyproject_hooks in /usr/local/lib/python3.10/dist-packages (from build>=0.10.0) (1.2.0)
Requirement already satisfied: packaging>=19.1 in /usr/local/lib/python3.10/dist-packages (from build>=0.10.0) (24.1)
Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from build>=0.10.0) (2.0.2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
found pre-compiled lib_lightgbm.so
--- building sdist ---
* Creating isolated environment: virtualenv+pip...
* Installing packages in isolated environment:
  - setuptools
* Getting build dependencies for sdist...
running egg_info
creating lightgbm.egg-info
writing lightgbm.egg-info/PKG-INFO
writing dependency_links to lightgbm.egg-info/dependency_links.txt
writing requirements to lightgbm.egg-info/requires.txt
writing top-level names to lightgbm.egg-info/top_level.txt
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.dll' under directory 'lightgbm'
warning: no files found matching '*.dylib' under directory 'lightgbm'
adding license file 'LICENSE'
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing lightgbm.egg-info/PKG-INFO
writing dependency_links to lightgbm.egg-info/dependency_links.txt
writing requirements to lightgbm.egg-info/requires.txt
writing top-level names to lightgbm.egg-info/top_level.txt
reading manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.dll' under directory 'lightgbm'
warning: no files found matching '*.dylib' under directory 'lightgbm'
adding license file 'LICENSE'
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
running check
creating lightgbm-<IP_ADDRESS>
creating lightgbm-<IP_ADDRESS>/lightgbm
creating lightgbm-<IP_ADDRESS>/lightgbm.egg-info
creating lightgbm-<IP_ADDRESS>/lightgbm/lib
copying files to lightgbm-<IP_ADDRESS>...
copying LICENSE -> lightgbm-<IP_ADDRESS>
copying MANIFEST.in -> lightgbm-<IP_ADDRESS>
copying README.rst -> lightgbm-<IP_ADDRESS>
copying pyproject.toml -> lightgbm-<IP_ADDRESS>
copying lightgbm/__init__.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/basic.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/callback.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/compat.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/dask.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/engine.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/libpath.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/plotting.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/py.typed -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/sklearn.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm.egg-info/PKG-INFO -> lightgbm-<IP_ADDRESS>/lightgbm.egg-info
copying lightgbm.egg-info/SOURCES.txt -> lightgbm-<IP_ADDRESS>/lightgbm.egg-info
copying lightgbm.egg-info/dependency_links.txt -> lightgbm-<IP_ADDRESS>/lightgbm.egg-info
copying lightgbm.egg-info/requires.txt -> lightgbm-<IP_ADDRESS>/lightgbm.egg-info
copying lightgbm.egg-info/top_level.txt -> lightgbm-<IP_ADDRESS>/lightgbm.egg-info
copying lightgbm/lib/lib_lightgbm.so -> lightgbm-<IP_ADDRESS>/lightgbm/lib
copying lightgbm.egg-info/SOURCES.txt -> lightgbm-<IP_ADDRESS>/lightgbm.egg-info
Writing lightgbm-<IP_ADDRESS>/setup.cfg
Creating tar archive
removing 'lightgbm-<IP_ADDRESS>' (and everything under it)
Successfully built lightgbm-<IP_ADDRESS>.tar.gz
--- installing lightgbm ---
Looking in links: .
Processing ./lightgbm-<IP_ADDRESS>.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: UNKNOWN
  Building wheel for UNKNOWN (pyproject.toml) ... done
  Created wheel for UNKNOWN: filename=UNKNOWN-0.0.0-py3-none-any.whl size=1794 sha256=31cf85902f1e3ffcb3b79747c9d3629bdf509a4dfe6f2dda9c7bf91c58ce4506
  Stored in directory: /tmp/pip-ephem-wheel-cache-bfyn99ka/wheels/97/48/5c/b2fa3ad0b5ff4da83888744e25e16ce761d05dc33923b317f1
Successfully built UNKNOWN
Installing collected packages: UNKNOWN
  Attempting uninstall: UNKNOWN
    Found existing installation: UNKNOWN 0.0.0
    Uninstalling UNKNOWN-0.0.0:
      Successfully uninstalled UNKNOWN-0.0.0
Successfully installed UNKNOWN-0.0.0

Additional Comments

Ubuntu 22.04 from WSL in windows. I also updated cmake from 3.22 to 3.28
version. I can prepare a docker file as necessary to reproduce the issue if
necessary
On Mon, Oct 7, 2024 at 9:03 AM James Lamb @.***> wrote:

What Linux distribution are you using? That'd help us to replicate this.
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/LightGBM/issues/6665#issuecomment-2396872600,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFPIDSOZWQR64LYKLLDM3XTZ2KBDDAVCNFSM6AAAAABPPA7BNCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJWHA3TENRQGA
.
 -DUSE_GPU=1
cmake --build build -j$(nproc)

ln -sf /usr/bin/python3 /usr/bin/python

python -m pip install \
    -U \
    numpy \
    scipy \
    scikit-learn \
    setuptools

sh ./build-python.sh install --precompile

logs (click me)
building lightgbm
Requirement already satisfied: build>=0.10.0 in /usr/local/lib/python3.10/dist-packages (1.2.2.post1)
Requirement already satisfied: pyproject_hooks in /usr/local/lib/python3.10/dist-packages (from build>=0.10.0) (1.2.0)
Requirement already satisfied: packaging>=19.1 in /usr/local/lib/python3.10/dist-packages (from build>=0.10.0) (24.1)
Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from build>=0.10.0) (2.0.2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
found pre-compiled lib_lightgbm.so
--- building sdist ---
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - setuptools
* Getting build dependencies for sdist...
running egg_info
creating lightgbm.egg-info
writing lightgbm.egg-info/PKG-INFO
writing dependency_links to lightgbm.egg-info/dependency_links.txt
writing requirements to lightgbm.egg-info/requires.txt
writing top-level names to lightgbm.egg-info/top_level.txt
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.dll' under directory 'lightgbm'
warning: no files found matching '*.dylib' under directory 'lightgbm'
adding license file 'LICENSE'
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing lightgbm.egg-info/PKG-INFO
writing dependency_links to lightgbm.egg-info/dependency_links.txt
writing requirements to lightgbm.egg-info/requires.txt
writing top-level names to lightgbm.egg-info/top_level.txt
reading manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.dll' under directory 'lightgbm'
warning: no files found matching '*.dylib' under directory 'lightgbm'
adding license file 'LICENSE'
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
running check
creating lightgbm-<IP_ADDRESS>
creating lightgbm-<IP_ADDRESS>/lightgbm
creating lightgbm-<IP_ADDRESS>/lightgbm.egg-info
creating lightgbm-<IP_ADDRESS>/lightgbm/lib
copying files to lightgbm-<IP_ADDRESS>...
copying LICENSE -> lightgbm-<IP_ADDRESS>
copying MANIFEST.in -> lightgbm-<IP_ADDRESS>
copying README.rst -> lightgbm-<IP_ADDRESS>
copying pyproject.toml -> lightgbm-<IP_ADDRESS>
copying lightgbm/__init__.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/basic.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/callback.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/compat.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/dask.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/engine.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/libpath.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/plotting.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/py.typed -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm/sklearn.py -> lightgbm-<IP_ADDRESS>/lightgbm
copying lightgbm.egg-info/PKG-INFO -> lightgbm-<IP_ADDRESS>/lightgbm.egg-info
copying lightgbm.egg-info/SOURCES.txt -> lightgbm-<IP_ADDRESS>/lightgbm.egg-info
copying lightgbm.egg-info/dependency_links.txt -> lightgbm-<IP_ADDRESS>/lightgbm.egg-info
copying lightgbm.egg-info/requires.txt -> lightgbm-<IP_ADDRESS>/lightgbm.egg-info
copying lightgbm.egg-info/top_level.txt -> lightgbm-<IP_ADDRESS>/lightgbm.egg-info
copying lightgbm/lib/lib_lightgbm.so -> lightgbm-<IP_ADDRESS>/lightgbm/lib
copying lightgbm.egg-info/SOURCES.txt -> lightgbm-<IP_ADDRESS>/lightgbm.egg-info
Writing lightgbm-<IP_ADDRESS>/setup.cfg
Creating tar archive
removing 'lightgbm-<IP_ADDRESS>' (and everything under it)
Successfully built lightgbm-<IP_ADDRESS>.tar.gz
--- installing lightgbm ---
Looking in links: .
Processing ./lightgbm-<IP_ADDRESS>.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: UNKNOWN
  Building wheel for UNKNOWN (pyproject.toml) ... done
  Created wheel for UNKNOWN: filename=UNKNOWN-0.0.0-py3-none-any.whl size=1794 sha256=e88b5b7ae36ef88e896ee2e2e6b4b7964b0f45ae209a1d87efa05cae773a9bff
  Stored in directory: /tmp/pip-ephem-wheel-cache-6jha994c/wheels/1d/09/6b/d983276114c6137d7226d1d09502c590b61e8604505dffb9cf
Successfully built UNKNOWN
Installing collected packages: UNKNOWN
Successfully installed UNKNOWN-0.0.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
cleaning up

Later, I'll try removing this line so I can inspect the build directory:
https://github.com/microsoft/LightGBM/blob/bbeecc09af946c5ff9b84d1ada4749a9f26bca31/build-python.sh#L379
I suspect this MIGHT be related to #6624. Will provide more updates soon.

I suspect this MIGHT be related to https://github.com/microsoft/LightGBM/pull/6624.

Nope, this was not it.
@ZhenyiLin I found that just updating to a more recent pip resolves this. Can you please try that?
pip install --upgrade pip
sh build-python.sh install --precompile

Processing ./lightgbm-<IP_ADDRESS>.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: lightgbm
  Building wheel for lightgbm (pyproject.toml) ... done
  Created wheel for lightgbm: filename=lightgbm-<IP_ADDRESS>-py3-none-any.whl size=3026623 sha256=cab9fd30930a192c76671c15546ee075bcd7f57c58367ffcda33d8e5b7fc8b0b
  Stored in directory: /tmp/pip-ephem-wheel-cache-832doefo/wheels/1d/09/6b/d983276114c6137d7226d1d09502c590b61e8604505dffb9cf
Successfully built lightgbm
Installing collected packages: lightgbm
  Attempting uninstall: lightgbm
    Found existing installation: lightgbm <IP_ADDRESS>
    Uninstalling lightgbm-<IP_ADDRESS>:
      Successfully uninstalled lightgbm-<IP_ADDRESS>
Successfully installed lightgbm-<IP_ADDRESS>

I saw that work with pip 24.2.
pip --version
# pip 24.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)

I'm not sure what range of versions will work.
Root Cause
It looks like python3-pip  / python3-venv on Ubuntu 22.04 ships with pip==22.0.2 and setuptools==59.6.0.
setuptools did not add support for pyproject.toml until v61.0.0 (see the setuptools docs).
Your script is running pip install -U setuptools (where -U is an alias for --upgrade), which installs the latest compatible setuptools at /usr/local/lib/python3.10/dist-packages. But I think the system-installed pip at /usr/lib/python3/dist-packages is ignoring that and loading the setuptools at /usr/lib/python3/dist-packages.
pip install -U pip puts a new pip into /usr/local/lib/python3.10/dist-packages, which then uses the newer setuptools there.
What lightgbm can do
We could have build-python.sh write a setup.cfg file when --precompile is passed, so other users won't face this.
Ubuntu 22.04 is only 2.5 years... it won't reach End of Standard Support until April 2027 (https://ubuntu.com/about/release-cycle). I think this is very worth supporting.

Thanks, updating pip works!


[Suggestion] Dependabot

Dependabot helps to keep all dependencies up-to-date.  It can also be configured to use emojis as commit prefix.  Its suggested updates can be accepted by mailing it the respective instruction which is very useful to handle multiple Dependabot PRs without having to login to GitHub.
Would it be useful to enable it for this repository?

Good suggestion. It would definitively be useful. I'll get into it this weekend.

Here is my suggestion for the setup:
version: 2
updates:
  - package-ecosystem: cargo
    directory: /
    schedule:
      interval: daily
      time: '00:00'
    assignees:
      - tobealive
    commit-message:
      prefix: ':robot:'
      include: scope

  - package-ecosystem: github-actions
    directory: /
    schedule:
      interval: daily
      time: '00:00'
    assignees:
      - tobealive
    commit-message:
      prefix: ':robot:'
      include: scope

Just paste this to a new file named .github/dependabot.yml.  You need to do this because this is your repository; I once configured Dependabot for a project of another maintainer after agreeing with them on the configuration details.  It did not work because I as a contributor were not allowed to edit the repository settings.
Dependabot was a third party app, previously, but then, it became a feature of GitHub itself.  This is why you need "version: 2".  To activate Dependabot, it is sufficient to create its configuration file; this will activate Dependabot for this repository.  I assume it is a security feature that users without writing permission to a repository submitting valid Dependabot configurations are not going to trigger the automatic activation in order to not override the maintainers' privileges.
However, you need to configure the package registries to grab updates from; here, we only have GitHub Actions and Rust.  I copied this basic setup directly from one of my own projects using Dependabot, so I know that this works.  I left the update interval as it was:  Dependabot will search for new updates each day on UTC midnight.  Please note that there might be some delay if a crate publishes a new version which is a feature, again.  Dependabot will then skip intermediate releases, e.g. updates Clap v4.1.11 to v4.1.13 because it did not scan for updates when Clap v4.1.12 was published.  This behaviour is not critical as you can always scan for updates manually from the Insights tab of the repository.  Navigate to the "Dependency graph" section in the left side bar and select "Dependabot" from the three tabs of this section.  You will see a list of package registries found by Dependabot.  Click the link "Last checked ..." and confirm with the "Check for updates" button for each registry.  Dependabot will then scan for updates.
All updates are submitted as PRs.  You can either merge them manually by clicking the merge button or by commenting on the PR with the magic words "squash and merge", "rebase", or "merge" after a mention of Dependabot.  The commands are always described in the initial PR comment, authored by Dependabot.  Expand them and select the appropriate instruction.  This commenting feature allows for Mail Merging, i.e. writing an email to the PR with the magic words such that Dependabot will take care for the update without requiring you to login.  This is extremely useful if you have Dependabot set up for numerous repositories and an essential dependency like actions/checkout published a new release.
You will definitely receive an email to answer because I made you the assignee for each PR.  Please note that sometimes, Dependabot forgets to assign you.  This is very rare but might happen.  It is sufficient to answer the assignment notification.  You might need to experiment how to answer the first times; Thunderbird calls it "Reply to list", GMail apps will automatically reply to the correct message ID.  Some email providers also refuse to send identical emails so make sure that the original message is cited.
I also tried to configure the commit messages to fit this project's requirements.  The plain text emoji name will be rendered to the correct category's emoji on GitHub (according to #11); if you like, you can also insert the emoji directly.  I tried to do so myself but did not succeed because my fonts did not know the emoji.  The "include: scope" is useful to track the updates, Dependabot will name the PR something like "Bump clap from 4.2.0 to 4.2.1" after the configured :robot: prefix.

Thanks a lot for the template @kevinmatthes!
I'll push the config today. One slight change I'll make is to increase the interval. Personally I don't like when repositories are artificially active by Dependabot bumps. Dependabot is a default for security updates anyway, so important notifications are sent anyway and discreetly.

The commit and PR messages differ a bit from what is stated in the docs. E.g. the end of the following section shows pip dev: bump requests from 1.0.0 to 1.0.1 as an example message.
https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#commit-message
I didn't found if it's possible to move from
⬆ (deps): Bump ...
to
⬆ bump ....
So we keep conformity and a clean history with other commit messages.
If you have a clue on how to achieve it directly in dependabot.yml please feel free to share 😊. An actual github action would probably the alternative.


Release checklist

[x] Add configuration singleton with clusters and dataroot (and the other minimal configs as required)

https://github.com/OSC/ood-myjobs/blob/3437bfd2e4be121637890af485a2b2b385053ec4/config/configuration_singleton.rb (the ConfigurationSingleton singleton offers two benefits: 1. we end up with a single global object Configuration with methods to call to return configuration, which can return set values or default values, or have some complex logic to determine at request time what to return (as opposed to setting values on a Rails.configuration.x object at initialization time). The bigger benefit is we can easily write readable automated tests against ConfigurationSingleton without having to worry about the state of some global object
https://github.com/OSC/ood-myjobs/blob/3437bfd2e4be121637890af485a2b2b385053ec4/config/boot.rb#L8-L13 (adding to boot allows using Configuration.xyz in bin/setup in config/application.rb etc.)

[x] Add the bin/setup-production script - FYI this is a poorly named script probably should be bin/per-user-setup but alas.
[x] Update bin/setup to work like the others, handling the desired default RAILS_RELATIVE_URL_ROOT etc.
[x] Add the json attrs fields to models as escape hatch for future work

https://github.com/OSC/ood-myjobs/blob/3437bfd2e4be121637890af485a2b2b385053ec4/app/models/workflow.rb#L8-L11 - lets us add more form elements and attributes to a model without requiring a database migration
https://github.com/OSC/ood-myjobs/blob/3437bfd2e4be121637890af485a2b2b385053ec4/db/schema.rb#L38

[ ] Add type column to script (?) and project (?) as escape hatch for future work?

Basically this commit which arguably should be in all of our Rails apps allowing us to create new models and attributes without the need for any new migrations

[x] RPM (#12)
[x] Test this app's job template vs the original maya app to confirm no significant difference, and revert to original maya job template (using job arrays) as necessary

I'm not sure how Add the json attrs fields and Add type column to script (?) and project (?) are different? So that's why I've left the one item blank, I've not implemented it yet.

json attrs is the new column for json store so you can add arbitrary attributes to existing models
the JsonStore model with a type column so you can add new models using single table inheritance
other two things that need fixed:

[ ] manifest category should be "Gateway Apps" and subcategory should be "Graphics" (well I don't know what makes the most sense for the subcategory)
[ ] puppet needs to be updated for that apps so that the permissions are 750 and the group that owns the app is the maya group

Note: The only things that matter for release right now are the manifest and puppet fix. I think the other two can be delivered in whatever state they are in. It will not be catastrophic if we deploy a new version of the app with a schema change.


Mark all scripts as executable

Simply ran the command from build_release_platform.sh:
find src -name "*.sh" | xargs chmod +x

LGTM
possibly we should add passing along executable bits as part of our open source exporting process

I believe that happens by default, but yeah, I plan to experiment with that.


How to expose nats server  with an external IP with advertiseExternalIP option

hello
how to expose  nats cluster to external IP with advertiseExternalIP option i just tried below settings and it is not worked
used : AKS cluster
created an LB with appropriate selector and port opened for 4222
and configured nats-cluster with below settings
apiVersion: nats.io/v1alpha2
kind: NatsCluster
metadata:
  name: demo-nats-cluster
spec:
  size: 2
  version: "1.4.1"
  tls:
    serverSecret: "nats-clients-tls"
    routesSecret: "nats-routes-tls"
  pod:
    enableClientsHostPort: true
    advertiseExternalIP: true
    bootconfigImage: "connecteverything/nats-boot-config"
    bootconfigImageTag: "0.5.0"
  template:
    spec:
      serviceAccountName: "nats-server"

but  i am getting  the error from init controller
time="2019-03-26T03:04:46Z" level=info msg="Starting NATS Server boot config v0.5.0"
time="2019-03-26T03:04:46Z" level=info msg="Pod running on node \"aks-nodepool1-36780942-0\""
time="2019-03-26T03:04:46Z" level=fatal msg="Could not find external IP address."

FYI . i am able to expose nats cluster without these by  just adding the LB alone(with out init pod) ,but i am not sure whether that is an right approach
please clarify & how to expose nats cluster to outside clients

On AKS I've used, the Azure nodes do not have a public ip (ExternalIP) which is required to be able to connect to the port without using external load balancers. This would be fine on GKE and latest EKS I believe but on AKS you need to use something like https://github.com/dgkanatsios/AksNodePublicIPController so that the external node has a public ip which can be then advertised to the clients.

@wallyqs  thank you for the quick answer using above i am able to create the cluster successfully,but without load balancer how can i expose the nats cluster to the outside (like accessing from another device machine where the client runs)

great to hear that it worked, for connecting to the nodes you could then use the external ip from the kubelet node where it is running using the port 4222 (you may have to change the firewall to expose that port for external access as well)

@wallyqs  thank you am able to  expose the nats server without load balancer
just one quick question
i just enabled authorisation as per the example below
https://github.com/nats-io/nats-operator#using-a-single-secret-with-the-explicit-configuration
and used the C# client   with nats://username:pass@server:port format
but i got an exception saying
[1] 2019/03/26 12:00:23.791575 [ERR] <IP_ADDRESS>:35337 - cid:5 - authorization error - User ""
is there any additional settings needs to be done?

If using Kubernetes v1.12 or above, try enabling config reload:
apiVersion: nats.io/v1alpha2
kind: NatsCluster
metadata:
name: example-nats
spec:
size: 3
version: "1.2.0"
pod:
enableConfigReload: true

@wallyqs  kube version is V1.9 .for my understanding  "enableConfigReload: true" is for on the fly configuration change but for now i just deleted the cluster and created new with authorisation but that also didn't worked
server config is below
apiVersion: nats.io/v1alpha2
kind: NatsCluster
metadata:
name: demo-nats-cluster
spec:
size: 2
version: "1.4.1"
auth:
clientsAuthSecret: "nats-clients-auth"
clientsAuthTimeout: 5
tls:
serverSecret: "nats-clients-tls"
routesSecret: "nats-routes-tls"
pod:
enableClientsHostPort: true
advertiseExternalIP: true
bootconfigImage: "connecteverything/nats-boot-config"
bootconfigImageTag: "0.5.0"
template:
spec:
serviceAccountName: "nats-server"
and client auth secret is created from below json
{
"users": [
{ "username": "user1", "password": "$2a$11$1oJy/wZYNTxr9jNwMNwS3eUGhBpHT3On8CL9o7ey89mpgo88VG6ba","permissions": {
"publish": ["foo."],
"subscribe": ["foo."]
} }
],
"default_permissions": {
"publish": ["SANDBOX.*"],
"subscribe": ["PUBLIC.>"]
}
}
and getting the same error saying like
[1] 2019/03/27 03:17:18.974871 [ERR] <IP_ADDRESS>:51162 - cid:3 - authorization error - User ""
may be this related to client side issue?
client side C#
ConnectionFactory cf = new ConnectionFactory();
Options opts = ConnectionFactory.GetDefaultOptions();
opts.Secure = true;
opts.Url = "nats://username:pass@server:4222";

@wallyqs  sorry  this  one is client side issue .working fine now
thanks

@nithinkd-geon glad to hear everything went well, one issue with Kubernetes v1.9 is that the authorization configuration will be static so the secret has to be ready before the pods are deployed since config reload can't be used.

@wallyqs  got it


Navigation issue in documentation

Contributing occurs twice in the documentation thus making the next button go to the end of the doc insteadof the next chapter.

This is intentional. Maybe it's arguable, but I've added it in both sections intentionally, feeling it fits well on both.


How do I read GL balances using this SDK?

Since there is no tutorial, I was wondering how it was possible to read "account balance by dimension" from https://developer.intacct.com/api/general-ledger/account-balances/ .
Is this currently supported at all?
What would be the right way to approach this?

Currently the SDK doesn't support this. This request would be an enhancement.

I suggest you check out DdsJobCreate for containing all information together for making this request (and ApAdjustmentCreate to see how to write the date in the split format for  and  elements).  DdsJobCreate will show you an approach for how to implement this overall.  Feel free to submit a pull request if you do implement this and we will review it for adding to the SDK.  Thanks!

@intacctcoleenho is it possible to use one of the common functions (IA.Functions.Common) to request the account balance data?

@intacctcoleenho Unless I'm underthinking this, I don't think it would be too difficult to add this functionality.  I'd be interested in creating a PR for this.  I assume it would mainly involve creating a new class in Functions/GeneralLedger? Something like AccountBalanceList?  Let me know if this would be something you'd merge and if there is any documentation for what is needed to merge.


Build failure with ROCm 5.4.3 - due to missing header inclusion dependencies

Issue Type
Bug
Have you reproduced the bug with TF nightly?
Yes
Source
source
Tensorflow Version
git 49d0ea44f5fda22b7529d74341c68147e8c95779
Custom Code
No
OS Platform and Distribution
Linux Ubuntu 22.04
Mobile device
No response
Python version
3.10
Bazel version
6.1.0
GCC/Compiler version
gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0
CUDA/cuDNN version
No response
GPU model and memory
Radeon 7900 XTX
Current Behaviour?
Build failed with errors below.
Standalone code to reproduce the issue
The build fails with the errors pasted above:

# ./configure with ROCM enabled and everything else default.
$ bazel build tensorflow/tools/pip_package:build_pip_package

ROCM 5.4.3 is installed at /opt/rocm-5.4.3/ and symlinked from /opt/rocm. (Installed via apt on Ubuntu 22.04 from official repos https://repo.radeon.com/rocm/apt/5.4.3 jammy/main amd64 Packages)

### Relevant log output

```shell
ERROR: /home/uday/tensorflow-upstream-rocm/tensorflow/compiler/xla/stream_executor/rocm/BUILD:477:11: Compiling tensorflow/compiler/xla/stream_executor/rocm/rocm_helpers.cu.cc failed: undeclared inclusion(s) in rule '//tensorflow/compiler/xla/stream_executor/rocm:rocm_helpers':
this rule is missing dependency declarations for the following files included by 'tensorflow/compiler/xla/stream_executor/rocm/rocm_helpers.cu.cc':
  '/opt/rocm-5.4.3/include/hip/hip_version.h'
  '/opt/rocm-5.4.3/include/hip/hip_runtime.h'
  '/opt/rocm-5.4.3/include/hip/hip_common.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_hip_runtime.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_hip_common.h'
  '/opt/rocm-5.4.3/include/hip/hip_runtime_api.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/host_defines.h'
  '/opt/rocm-5.4.3/include/hip/driver_types.h'
  '/opt/rocm-5.4.3/include/hip/texture_types.h'
  '/opt/rocm-5.4.3/include/hip/channel_descriptor.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_channel_descriptor.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_hip_vector_types.h'
  '/opt/rocm-5.4.3/include/hip/surface_types.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_hip_runtime_pt_api.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/hip_ldg.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_hip_atomic.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_device_functions.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/math_fwd.h'
  '/opt/rocm-5.4.3/include/hip/hip_vector_types.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/device_library_decls.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/llvm_intrinsics.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_warp_functions.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_hip_unsafe_atomics.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_surface_functions.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/ockl_image.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/texture_fetch_functions.h'
  '/opt/rocm-5.4.3/include/hip/hip_texture_types.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/texture_indirect_functions.fh'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_math_functions.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/hip_fp16_math_fwd.h'
  '/opt/rocm-5.4.3/include/hip/library_types.h'

I can reproduce the same issue with the latest Tensorflow upstream git version. (514005d074bba67116ff35abfbba32dafff8ad1e Jun 3, 2023).

Hi there I encountered the exact same problem building TF2.12on Rocm5.6. I was able to solve the problem by following the instructions on https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/develop-upstream/rocm_docs/tensorflow-build-from-source.md
'''
Follow steps at Basic Installation to install ROCm stack. NOTE: ROCm install instructions recommend a purge and reinstall of ROCm rather than upgrading from previous release. For details of the ROCm instructions, please refer to the ROCm QuickStart Installation Guide.
To build with ROCm3.10, set the following environment variables, and add those environment variables at the end of ~/.bashrc
export ROCM_PATH=/opt/rocm-4.0.0
export HCC_HOME=$ROCM_PATH/hcc
export HIP_PATH=$ROCM_PATH/hip
export PATH=$HCC_HOME/bin:$HIP_PATH/bin:$PATH
export ROCM_TOOLKIT_PATH=$ROCM_PATH
''''
Obviously you need to replace the rocm version in the file, which in your case would be /opt/rocm-5.4.3

Are there still people who are waiting for 7900XTX support? Though the performance is still a bit poor, TensorFlow-upstream now runs when built on the latest ROCm release. I was looking into the status of ROCm support for 7900XTX and found a few issues opened by different people and wanted to link all to the issue I opened in MIOpen repo. Though there has not been any confirmation from the developer, I think the performance issues are due to insufficient optimization of MIOpen.
https://github.com/ROCmSoftwarePlatform/MIOpen/issues/2342

@bondhugula Has your issue been resolved? If so, please close the ticket. Thanks!

It's been over a year since this issue was filed. All the associated packages/software have now moved forward significantly. I haven't tried out with recent versions. But if others have reported that the build works fine, please go ahead and close this.

Hi, this issue will be closed now as I could not reproduce it with the newest release. Please feel free to reopen it or ask follow-up questions should further assistance be required. Thanks!


Import - Basic functionality

Add support for the import command
Usage: hp-cli import [Options] --device <DEVICE_PATH> --output <OUTPUT_PATH>
These options should be implemented in this ticket:
  -d, --device <DEVICE_PATH>  Path to USB mass storage or SD card where data will be imported from.
  -o, --output <OUPUT_PATH>   Path to where the directory for imported data will be created and data will be imported.
  -h, --help                  Displays help for available commands, all other arguments are ignored.

The remaining options are taken care of in separate tickets:

https://github.com/nordic-dev-net/hydrophonitor-cli/issues/14
https://github.com/nordic-dev-net/hydrophonitor-cli/issues/6
https://github.com/nordic-dev-net/hydrophonitor-cli/issues/5
https://github.com/nordic-dev-net/hydrophonitor-cli/issues/15

The following topics are also delegated to separate tickets:

https://github.com/nordic-dev-net/hydrophonitor-cli/issues/16

In the current ticket the CLI should just abort immediately in case of an error and print the cause.
Importing data should only be allowed into new output directories.
If the output directory alredy exists, abort immediately and print an appropriate error message.

https://github.com/nordic-dev-net/hydrophonitor-cli/issues/8

This command imports data from the given device, creating a new directory for each imported deployment at the specified output path. This command does not remove files from the device.
At the beginning of the import, the CLI lists the deployments it has detected. For each deployment, the size of the deployment folder is printed. A progress bar shows the progress of the import.
The device is attached to the host computer as USB mass storage. It is assumed that the output data is located in /output directory at the root of the device file system. This directory contains a timestamp-named directory for each deployment (one boot from startup to shut down).
/output/
    <TIMESTAMP>/
        audio/
            <TIMESTAMP>_audio.wav
            ...
        gps/
            <TIMESTAMP>_gps.json
            ...
        depth/
            <TIMESTAMP>_depth.csv
        log/
            <TIMESTAMP>_journalctl.txt
    <TIMESTAMP>/
        ...

All deployments are imported. The idea is that each deployment would be followed by an import and a cleanup that deletes that deployment from the device. However, one outing with the Hydrophonitor could result in several device restart cycles, so we need to support importing several deployments.
Data Formats
Audio
The audio is recorded in batches (this is done to avoid data corruption in case of an ungraceful shutdown) as wav files in /output/<timestamp>/audio directory and are merged upon import into one .wav file.
GPS Data
At the moment of writing, hydrophonitor-gps module records all available data points introduced by gpsd in json files in ``/output//gps` directory. All json files are merged into one json file upon import.
Depth Data
At the moment of writing, depth-recorder module records depth measurements in a csv file in /output/<timestamp>/depth directory. If there are multiple csv files, those are merged into one file upon import.
Logs
During the deployment, journalctl logs are periodically exported to a text file in /output/<timestamp>/log directory.
Imported Data Directory Structure
<TIMESTAMP>/
    metadata.json
    audio-<TIMESTAMP>-<TIMESTAMP>.wav
    gps-<TIMESTAMP>-<TIMESTAMP>.csv
    depth-<TIMESTAMP>-<TIMESTAMP>.csv
    journalctl-<TIMESTAMP>.log

I have 3 questions regarding this ticket:

Should the output parameter be optional and the current directory used as fallback, or should it be mandatory?
Should we validate that the deployments have a valid name, according to the format of the timestamp that you provided in the requirements specification?
In that case, should we assume that all directories with an invalid name are not deployments, but rather contain some other data and therefore skip them?
And I guess if there are some independent files where we expect to find the deployment directories,  those will also have to be skipped.
Unless we produce an error in either of those two cases, but that would seem like overkill, as it's just a minor edge case that isn't expected to occur anyways.
Should the location of the deployments (i.e. /output) be hardcoded, or should we offer a way to overwrite it from the cli?

Let's make output folder explicit (required) for now. Easy to change later if we want the convenience.
I think the more validation checks the better. That being said we can start small and add more checks later. I think such edge cases are not a priority.
Yes this should be hardcoded.

Alright, thanks for the feedback.
And should the output folder already have to exist?
Otherwise, I would ask the user for confirmation, in case the output folder has to be created.

I think if /output folder doesn't exist, it's a hard fail. Most probably an indication that user chose the wrong device.

Output folder will be created with hp-cli flash. If this hasn't been done on the device it's not considered a Hydrophonitor.

Sorry, I meant the destination to which the data will be imported, which the user provides with --output
Does that directory already have to exist, and therefore should we produce an error if it doesn't?
Or should we warn the user if it doesn't exist, ask for confirmation to create it, and then import the different deployments to that newly created directory?

Let's have it as hard fail for now. Let's keep prompting to a minimum an add if needed. If the folder didn't exist user "probably" input the wrong path.


Work recovery from failed jobs

This has been discussed offline before.  Basically the idea is that, a failed job might have already accomplished a portion of the work, so when we restart the job (speaking of which, there currently isn't a way to restart failed jobs), we'd expect it to pick up from where it left off, instead of doing the entire thing again.
The most intuitive idea is to monitor how much input data has already been consumed, and only feed the restarted job the input data that hasn't been consumed.  The challenge here is that it's unclear how we'd figure out if a piece of data has actually been "consumed", as opposed to simply having been read.  For instance, it's easy to know that a job has read file foo.  But how do we know if the job has acted on file foo?  Might have to introduce some new API.

I think we're likely going to have to limit ourselves to shard boundaries for work recovery for the reasons you describe here about not really being able to know if a file has been processed or not. That can probably be ok though because we can scale up the sharding to get whatever recovery granularity makes sense.
We've talked about supporting other computation paradigms like classic Map/Reduce. In those tracking completion on a per file basis would totally be doable so maybe it's worth thinking about what we'd need to implement to get that use case working.

Ran into a similar use case when running through @dwhitena's go compilation pipeline. Basically ... I had parallelism of 64, and doing the rough math I expected the job to complete in about 80 minutes.
However, the input data (list of repos) size is not proportional to computation time. Compiling is CPU bound and some repos seem to take a lot longer. (So the processing is really lumpy)
90 minutes into the job, I have 22 pods left. I'd love to be able to re-distribute the work to the other 42 'idling' VMs, so that we recover some of the total processing time. To do that, we'd have to detect what files have been read as you describe above.

@sjezewski without knowing how long each line/file/block will take to process ahead-of-time, there isn't really any way to know what is left to process at a given time.
Without enforcing some requirements on users regarding filesystem access, we can't just add more data to a running pod since it's entirely possible their code will enumerate files only once and never check for changes. I don't think there is any meaningful way to map file-access info to work-done, since again we don't know the significance of any given read.
So I believe that limits your options to either:

Actively scheduling data to containers as they free up,
Killing running containers, discarding their outputs entirely and redistributing work.

The first option sounds a lot like it's kubernetes' job which makes me hesitant to try it, but I don't see a reason the second couldn't work while being inefficient.

I agree its tricky. This is a type of optimization we wouldn't consider for a while, because it probably only yields benefits in niche cases (like the one I listed above).
I do think its possible if the users have the right expectations. Those expectations are basically to re-query to see what files are available between reads.
This prevents the pod that has files 'taken away' from it, from erring when it tries to read a file it may have 'seen' earlier when listing files in a directory.
It also allows pods to handle new files that might appear after initially listing files in a directory.

A more broad way to frame this problem is: What recourse we have for 'runaway jobs' ? ... jobs whose processing times are huge outliers.
One bit of recourse - w the PFS refactor, we'll treat each pod's changes as a separate commit, so if some subset of the pods are long running (as they were above), you should be at least able to stop them, and see which inputs were causing the problem.

I think we can still do a lot here with job recovery on commit granularity by finding stragglers and restarting them with a higher sharding factor. We'll lose some amount of work along the way but I think the approach can still make things dramatically faster. Especially if we're aggressive about when we call a job a straggler / runaway. For example suppose we have a job with 4 pods running. After a few minutes we notice that 3 of these pods have read 75% of their input data while the fourth has read only 15%. We deem that pod a runaway and so we further shard its input into 8 more containers, those have to start from scratch but they only have 1/32 of the total data so they'll likely finish much earlier than previous run would have.

There's a separate issue open for straggler detection: https://github.com/pachyderm/pachyderm/issues/442
Meanwhile, in v1.4 the data is distributed at a much finer grain (via datums). I believe that would have helped mitigate the issue I was seeing. Also - PPS is smart enough not to reprocess any datums.
I don't think there is any other todo's for this issue (since we have a separate straggler thread). @derekchiang - care to verify?

Can confirm, work recovery is as good as it's going to get.