PR switching from qform to sform causing many downstream support issues, particularly for small voxel sizes

The PR here, https://github.com/InsightSoftwareConsortium/ITK/pull/1868
Is regularly triggering issues in various downstream projects, with users with (wrong? malformed? other?) NIFTI files which used to work with ITK-based tools and now do not.
https://github.com/Project-MONAI/MONAILabel/issues/35
https://github.com/CoBrALab/RABIES/issues/160
https://github.com/ANTsX/ANTs/issues/1213
https://github.com/SimpleITK/SimpleITK/issues/1433
I'm not sure what the solution to this is, but I'd like to propose a couple:

Change code to be a warning and fallthrough to qform reading
Provide a more informative error for the user to correct the generation of the invalid NIFTI file

People who use NIFTI a lot should weigh in. Candidates: @hjmjohnson @cookpa @vfonov @seanm @lassoan.

The code does attempt to fall back to the qform if the sform cannot be used. The ANTs bug you link to was triggered by a combination of small voxels and missing qform (qform_code == 0). If I increased the voxel size, or entered a qform into the header, it worked.
I haven't had time to chase it down but I think there is some numerical test of whether to use sform that is getting confused by small voxel sizes. The data doesn't have to be oblique at all, if you have an sform of Iβ, where β is a sufficiently small number (< 0.2 seemed to trigger it for me), The sform is rejected, and the fallback to qform fails when qform_code is 0.
I think if that can be addressed, we'll be good.

Thanks for the detail @cookpa. I'm going to follow up with the report I got to see what the actual state of the NIFTI header was for reference here.

Details on the problematic scan we had:
https://github.com/CoBrALab/RABIES/issues/160#issuecomment-892471093

I have no preference on how to change the NIFTI code or error messages to deal with this issue.
My hope is that we can build up support for OME-Zarr (or similar modern file format) in the wider image computing community and that will offer enough incentives for researchers to leave behind NIFTI and its sform/qform mess for good.

I think I found it:
https://github.com/InsightSoftwareConsortium/ITK/blob/02e7c448b4e3b77f0e46afb66f92bb3be8995dd7/Modules/IO/NIFTI/src/itkNiftiImageIO.cxx#L1762-L1767
This fails if the voxels are small because the determinant of the sform matrix is below the threshold (1e-5). Perhaps this could safely be made smaller.
But without eliminating the test entirely, there's always going to be a minimum spacing before this error occurs. If the entries in sform matrix are sufficiently small, qform might be the better way to get the rotation anyway. But even if the qform is used, I think very small spacing could become problematic for computing the rotation / scale matrix inverses.
Maybe users should be warned to use a more appropriate unit (eg, NIFTI_UNITS_MICRON) if the matrix determinant is below the threshold?

Oh, actually other units won't help because they will get scaled to mm units

@cookpa good sleuthing!  The 1e-5 threshold is quite demanding; anyone know why it is there?  For example, there are accurate methods for computing a matrix inverse that can handle determinants much, much smaller in absolute value than 1e-5.

PR #1868 introduced it:
https://github.com/InsightSoftwareConsortium/ITK/commit/34231b57021418fdb6afd4fcf5082a73b12969ed#diff-fa33a46a6dc41cde5d18ee8f63cb941794512e03c6377340b8d110fe584cfdfaR1756-R1760
I guess it is fine for MRIs, which have voxels ~ 1.0 mm. @hjmjohnson is there a problem reducing this to a much lower value?

https://github.com/SimpleITK/SimpleITK/issues/1440

Gabriel, can you make a PR with your proposed changes (turn error into warning and more informative message)?

Also, can a mathematically-inclined person figure out a better threshold for determinant:
https://github.com/InsightSoftwareConsortium/ITK/blob/02e7c448b4e3b77f0e46afb66f92bb3be8995dd7/Modules/IO/NIFTI/src/itkNiftiImageIO.cxx#L1764

Also, can a mathematically-inclined person figure out a better threshold for determinant:
https://github.com/InsightSoftwareConsortium/ITK/blob/02e7c448b4e3b77f0e46afb66f92bb3be8995dd7/Modules/IO/NIFTI/src/itkNiftiImageIO.cxx#L1764

I think solution might be to test the condition of the matrix and compare to epsilon, as suggested here, https://stackoverflow.com/a/13270760/4130016

I think solution might be to test the condition of the matrix and compare to epsilon

It would be good if this found its way into your PR 😄

To clarify here, after working through the logic, it looks like we have two things happening together

https://github.com/InsightSoftwareConsortium/ITK/blob/02e7c448b4e3b77f0e46afb66f92bb3be8995dd7/Modules/IO/NIFTI/src/itkNiftiImageIO.cxx#L1764 which is resulting in some kinds of sforms being rejected

combined with
2. https://github.com/InsightSoftwareConsortium/ITK/blob/02e7c448b4e3b77f0e46afb66f92bb3be8995dd7/Modules/IO/NIFTI/src/itkNiftiImageIO.cxx#L1971-L1980
Which, without an else condition, is falling through (and not, at least according to my reading, following the NIFTI spec, which says for #define NIFTI_XFORM_UNKNOWN      0 that this should be followed:
   METHOD 1 (the "old" way, used only when qform_code = 0):
   -------------------------------------------------------
   The coordinate mapping from (i,j,k) to (x,y,z) is the ANALYZE
   7.5 way.  This is a simple scaling relationship:

     x = pixdim[1] * i
     y = pixdim[2] * j
     z = pixdim[3] * k

   No particular spatial orientation is attached to these (x,y,z)
   coordinates.  (NIFTI-1 does not have the ANALYZE 7.5 orient field,
   which is not general and is often not set properly.)  This method
   is not recommended, and is present mainly for compatibility with
   ANALYZE 7.5 files.

For a PR for this, I will add an else fallthrough, and print a warning, but the question is how should the final mapping be handled

Use the analyze method (qform = 0 above)
or
Use the qform anyways

Following up on this again:
https://github.com/InsightSoftwareConsortium/ITK/blob/02e7c448b4e3b77f0e46afb66f92bb3be8995dd7/Modules/IO/NIFTI/src/itkNiftiImageIO.cxx#L1764
Something didn't sit right with me, so I went back to my linalg textbook and a matrix A is invertible if det(A) != 0. I wonder if this test was meant to be fabs(vnl_determinant(mat)) >= 1e-5

If the header has nonzero sform or qform, and they cannot be used for reasons, I'd prefer to throw an error rather than fall back to method 1. Better no data than accidentally flipped data, IMO.
I think the logic is good as is:

Attempt to use "method 1", Analyze backwards compatibility, only if qform_code == sform_code == 0.
Else use sform if it's defined and orthonormal direction cosines can be recovered from it
Else try to use qform
Else throw an error

The only thing I would add is that after attempting and failing to use sform, the user should be warned that the attempt to use sform failed, and qform will be used as a fallback. Currently, that's a silent fallback. Arguably it should be an error too as it suggests something wrong with sform, but at least we are falling back to another transform that's actually in the header, rather than assuming ANALYZE.

Regarding the threshold for the determinant, I found this:
  //: find weights below threshold tol, zero them out, and update W_ and Winverse_
  void            zero_out_absolute(double tol = 1e-8); //sqrt(machine epsilon)

https://github.com/vxl/vxl/blob/d771c601a2072afe46af8a80b8e480d0529d8f2f/core/vnl/algo/vnl_svd.h#L89-L90
So inversion will run into problems with voxels smaller than 1e-8, because the singular values will be zeroed out. This behavior can be changed in vnl_svd, but I'm not sure it's accessible from the wrapper vnl_matrix_inverse

Regarding the threshold for the determinant, I found this in the matrix inversion code:
  //: find weights below threshold tol, zero them out, and update W_ and Winverse_
  void            zero_out_absolute(double tol = 1e-8); //sqrt(machine epsilon)

https://github.com/vxl/vxl/blob/d771c601a2072afe46af8a80b8e480d0529d8f2f/core/vnl/algo/vnl_svd.h#L89-L90
They suggest 1e-8 or sqrt(machine epsilon) as a threshold for the singular values to be zeroed out, but the class looks like it defaults to a threshold of zero. If we had spacing of 1e-8, the determinant would be 1e-24.
I'll look more at the code, I'm thinking it might be better to just get the SVD decomposition directly and check it has the correct rank.

Singular values (essentially eigenvalues) get multiplied together to determine the determinant, so even when 1e-8 is the right threshold for a singular value, it is probably not the correct threshold for a determinant.  Also, the smallest positive single-precision floating point number that can be represented (without resorting to subnormals representations) is 1.175494e-38, and its square root is not 1e-8.
More importantly, rigorously it is the matrix conditioning number rather than its determinant that should be checked.  Is that check a possibility?

More importantly, rigorously it is the matrix conditioning number rather than its determinant that should be checked. Is that check a possibility?

Yes, if we just compute SVD directly its the ratio of the largest to smallest eigenvalues.

Using the matrix condition :
https://github.com/cookpa/ITK/blob/2a6a365b8a6ee1a715254c1c3478666c4e8b8e6f/Modules/IO/NIFTI/src/itkNiftiImageIO.cxx#L1762-L1767
This reads small (18 micron) voxels that previously failed. I'm not sure on the best way to formally test this, as any image created with this class will contain both sform and qform.

Using the matrix condition :
Awesome, I couldn't find something that just outputted the condition number.

This fix will solve the immediate issue, I think we can leave the logic adjustment to a seperate PR?

I put in a PR for the condition number stuff. I could not find a good place to insert a warning without doing more refactoring, so I left that out. IMO the only time to warn is if we can't use sform AND sform_code == NIFTI_XFORM_SCANNER_ANAT.


Clarification Needed on Thread Context Switch in writer() Function (example streaming-server.cpp)

Hello,
I'm currently reviewing the server-side code for handling gRPC requests, specifically the use of co_await rpc.write(response); in the writer() coroutine function. I observed that there is a comment stating "// Now we are back on the main thread." after this line of code.
From my understanding, in Asio and C++ coroutines, the execution context or thread does not automatically switch back to the original context (such as the main thread of grpc_context) after an await operation, unless explicitly specified. However, in this instance, I don't see an explicit switch back to the grpc_context after the write operation in a multi-threaded environment (using asio::thread_pool).
Could you please clarify if the comment accurately reflects the behavior of the code? If there is indeed an automatic switch back to the main grpc_context thread happening, could you point out where and how this context switch is being handled in the code?
This clarification will greatly help in understanding the correct thread behavior and context management in asynchronous operations within the application.
Thank you for your time and assistance.

Hi,
the comment in the code is in fact accurate. Asio describes the thread-switching behavior in their documentation. Although I find it rather difficult to understand, let me try to explain it in my own words.
Take the following example:
    asio::io_context io_context1;
    asio::steady_timer timer{io_context1};
    asio::io_context io_context2;
    asio::co_spawn(
        io_context2,
        [&]() -> asio::awaitable<void>
        {
            // on io_context2 thread

            // Timer is handled on io_context1 thread
            // But after completion it will:
            // `asio::dispatch(asio::get_associated_executor(use_awaitable_completion_handler,
            // timer.get_executor()), use_awaitable_completion_handler)`
            // which causes the switch back to io_context2 thread
            co_await timer.async_wait(asio::use_awaitable);

            // on io_context2 thread
        },
        asio::detached);
    auto g = asio::make_work_guard(io_context1);
    std::thread t{[&]
                  {
                      io_context1.run();
                  }};
    io_context2.run();
    g.reset();
    t.join();

Asio performs the following steps on this line co_await timer.async_wait(asio::use_awaitable);:

Convert the completion token (asio::use_awaitable) into a completion handler using asio::async_result machinery. The details do not matter much, but the result is a callable object (called completion handler) that is to be invoked when the timer completes.
Get the I/O executor by calling timer.get_executor() and use that to do whatever is necessary to initiate and wait for the timer (system calls, epoll, etc.)
Meanwhile create the completion handler executor by calling asio::get_associated_executor(completion_handler, timer.get_executor()) and establish work-tracking (call asio::make_work_guard()) until the timer completes. use_awaitable's completion handler actually has an associated executor, it is the one used in the call to co_spawn (io_context2).
When the timer completes, invoke the completion handler as if by calling asio::dispatch(completion_handler_executor, completion_handler). This is where the thread switching is happening, going from timer.get_executor() (io_context1) to the completion handler executor (io_context2).
The completion handler will then resume the coroutine.

Since asio-grpc tries to mimic Asio's behavior as much as possible, replacing asio::steady_timer with agrpc::ServerRPC and io_context with GrpcContext yields the same thread-switching result.
I hope that helps, let me know if something is still unclear.

Hi,
Thank you for the clear explanation. I tested it and everything worked just as you described. Your guidance made the thread-switching behavior in Asio much clearer.
Also, I want to say that asio-grpc is an amazing project. Really appreciate all the work you've put into it.
Best

Thanks, glad I could help.
Btw, you can disable the thread-switching this way:
inline void use_inline_awaitable() { return asio::bind_executor(asio::system_executor(), asio::use_awaitable); }

co_await timer.async_wait(use_inline_awaitable());

This will cause async_wait to resume the coroutine on whatever thread the timer's I/O executor is running on.

Hi again,
I revisited your example and noticed that after co_await timer.async_wait(), the execution returns to the io_context2 thread, the same thread as before the co_await. So, in the writer code, before co_await rpc.write(), it's in the thread_pool, right? By the same logic, does this mean that after co_await rpc.write(), the execution also returns to the thread_pool?

It doesn't matter what thread it was on before the co_await, what matters is the first argument passed to asio::co_spawn. By default any co_await ... asio::use_awaitable asynchronous operation started within the co_spawned coroutine will switch back to that first argument (provided the asynchronous operation follows Asio's completion handler executor rules explained earlier). I know this can be a bit unfortunate because the co_spawn call can be far away in some other source file.

Great, it's finally clear now. Thank you so much.
Have a good day!

writer is (just a little far away) from the co_spawn that first argument is grpc_context or grpc_context's executor, right?

Correct, the co_spawn call is actually hidden inside agrpc::register_awaitable_rpc_handler. It actually invokes co_spawn like so:
asio::co_spawn(asio::get_associated_executor(completion_handler, first_argument_passed_to_register_awaitable_rpc_handler())

Asio internally always works with executors. There is a co_spawn overload that takes an execution context, like GrpcContext, and essentially forwards the call to co_spawn(grpc_context.get_executor(), ...).


Add Firebase Messaging Examples (Notifications, Data Messages, ...)

Currently there is a lack of examples for the library. I am presenting a Data Message in the README, which doesn't show what most users want to do: Notifications. I have already added the simple Android quickstart-messaging to the Examples folder.
The README should be updated and Examples with the Android example should be shown, especially for Notifications.

I'd love to help out, but struggling to find sufficient documentation on the semantics of the fields. If the Firebase docs were better this would be easier. It's also confusing for me how the different app states (foreground, background, killed) play into it, so I'm not 100% comfortable I'm getting all the messages. I'll come back and share what I learn. In the meantime, if you by chance have any working notifications that could be included in the read.me quickly I would really appreciate it!
Here's what I'm going to try first:
                var pushMessage = new FcmMessage
                {
                    ValidateOnly = false,
                    Message = new Message
                    {
                        Token = tokenParm.pvalue,
                        Topic = DEFAULT_TOPIC,
                        Notification = new Notification
                        {
                            Title = $"{appName}: tasks due",
                            Body = $"You have {tasksDue} tasks due, please open the app now to catch up!",
                        },
                        AndroidConfig = new AndroidConfig
                        {
                            Notification = new AndroidNotification
                            {
                                Title = $"{appName}: tasks due",
                                Body = $"You have {tasksDue} tasks due, please open the app now to catch up!",
                                Sound = "default"
                            }
                        },
                        ApnsConfig = new ApnsConfig
                        {
                            Payload = new ApnsConfigPayload
                            {
                                Aps = new Aps
                                {
                                    Sound = "default",
                                    Badge = tasksDue
                                }
                            }
                        }
                    }
                };

@kehollin Thanks for your efforts! The Firebase Docs are indeed a bit sparse on information, making it a little hard to build a rich set of samples. I will add an example for notifications later today!

Thank you! I've updated the message initializer in my comment. I haven't
testing rigorously (i.e. app in foreground, background and killed states)
but will be doing so this coming week and will update the thread on GitHub
if I find any problems. One thing to note is that I'm sending to a specific
device ID, but including both AndroidConfig and ApnsConfig primarily
because I'm too lazy to code for different platforms, but it seems to be
allowed and harmless.
Kevin Hollingshead
Research Software Engineer
Arizona State University
College of Health Solutions
School of Nutrition and Health Promotion
602-615-9972 (cell)
On Sat, Nov 17, 2018 at 10:01 PM Philipp Wagner<EMAIL_ADDRESS>wrote:

@kehollin https://github.com/kehollin Thanks for your efforts! The
Firebase Docs are indeed a bit sparse on information, making it a little
hard to build a rich set of samples. I will add an example for
notifications later today!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bytefish/FcmSharp/issues/34#issuecomment-439668073,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AExGMenkjbGAbuPGf9KyQHOCloVn6xGEks5uwOm0gaJpZM4W1Hu7
.

@kehollin Good job! 👍I would like to add (a slightly modified) example of your sample to the README, if you are OK with it? 😎

Sure, maybe mask the device id too, no?
On Sat, Nov 17, 2018, 23:04 Philipp Wagner<EMAIL_ADDRESS>wrote:

@kehollin https://github.com/kehollin Good job! 👍I would like to add
(a slightly modified) example of your sample to the README, if you are OK
with it? 😎
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bytefish/FcmSharp/issues/34#issuecomment-439670217,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AExGMYuNM7C4l5fFoKCbTanp5r6y5XIAks5uwPhVgaJpZM4W1Hu7
.

It's early here, so maybe I am a bit slow. What do you mean by Masking the Device ID?

@kehollin I have added an example for Notifications now, along with an Android sample project. The old tutorial was much too complicated, this one is a better Quickstart. Along the way I have also simplified reading the Service Account Key in the library, so one doesn't need to hardcode the project key in Code anymore.
I have not integrated your example yet, but plan to do so. But I would need to write a longer documentation on it, because people will have questions I probably cannot answer... especially for the Apple iOS part, that I have not a lot of experience with.
If you find the time it would be great, if you could make a Pull Request to the README and put an example in the "Advanced" section. Maybe explaining all it in a Quickstart is too heave on Android / C# starters.

Examples have been added for most of the Functions. The Android application makes a good start for Newcomers to Firebase and it is easy to get started with it. Everything else must be extracted from the docs, because I cannot make up examples for the entire API.


Initialize custom backends in open_dataset

The backend classes are initialized in the build_engine function:
https://github.com/pydata/xarray/blob/8b95da8e21a9d31de9f79cb0506720595f49e1dd/xarray/backends/plugins.py#L93
This wasn't the case for custom backends:
https://github.com/pydata/xarray/blob/8b95da8e21a9d31de9f79cb0506720595f49e1dd/xarray/backends/plugins.py#L161
This PR initializes the engine, fixes the incorrect signature in the test (#5033) and reverts the doc changes done in #5532.

[x] Tests added
[x] Passes pre-commit run --all-files
[x] User visible changes (including notable bug fixes) are documented in whats-new.rst

I noticed this issue because my local mypy version was complaining about:
xrtools.py:750: error: Signature of "open_dataset" incompatible with supertype "BackendEntrypoint"
Anyone have any ideas why the CI version doesn't complain about this?

For mypy to do any signature checking either of the inputs or outputs needs to be typed. This will throw the supertype error:
import numpy as np
import xarray as xr
from xarray.backends import BackendEntrypoint

expected = xr.Dataset(
    dict(a=2 * np.arange(5)), coords=dict(x=("x", np.arange(5), dict(units="s")))
)

class CustomBackend(BackendEntrypoint):
    def open_dataset(
        # self,
        filename_or_obj,
        drop_variables=None,
        **kwargs,
    ) -> xr.Dataset:
        return expected.copy(deep=True)

This looks ready to me, and I think we should merge soon.


LGTM


import/extensions is confused by "keen-js"

error  Unexpected use of file extension "" for "keen-js"  import/extensions

import Keen from "keen-js"

I think this should be solved in v1.7.0, if not let me know and we can debug.

(fwiw, I think #296 would have solved it)

Thanks!
On Fri, May 6, 2016 at 5:36 AM, Ben Mosher<EMAIL_ADDRESS>wrote:

(fwiw, I think #296
https://github.com/benmosher/eslint-plugin-import/pull/296 would have
solved it)
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/benmosher/eslint-plugin-import/issues/309#issuecomment-217428155

--
Kurtis Rainbolt-Greene, Software Developer
Founder of Difference Engineers
2985 San Marino St.
Los Angeles, CA 90006
202-643-2263


Anvil tsv #162

Closes #162
Reviewers
Changes

adding tsv reader
adding fetcher layer
adding workspace transformers for anvil

Definition of Done (from ticket)
QA steps (optional)
Known Issues

The changes requested for this PR were done on https://github.com/clevercanary/data-browser/pull/257.
This PR will be closed.


ActionSheet 'show' method says undefined is not an object

react-native, react and native-base version
RN 0.42, R 15.4.2, NB 2.0.13
Expected behaviour
ActionSheet should show just like it does in the docs: http://nativebase.io/docs/v2.0.0/components#actionsheet
Actual behaviour
Error. undefined is not an object (evaluating '_nativeBase.ActionSheet.show')
Steps to reproduce (code snippet or screenshot)
import React, { Component } from 'react';
import { Image } from 'react-native';
import { Button, ActionSheet } from 'native-base';

const placeholder = require('../images/placeholder_person.png');

var BUTTONS = [
  'Option 0',
  'Option 1',
  'Option 2',
  'Delete',
  'Cancel',
];
var DESTRUCTIVE_INDEX = 3;
var CANCEL_INDEX = 4;

export default class ProfileImage extends React.Component {

  render() {
    return (
      <Button transparent onPress={() => ActionSheet.show(
            {
              options: BUTTONS,
              cancelButtonIndex: CANCEL_INDEX,
              destructiveButtonIndex: DESTRUCTIVE_INDEX,
              title: 'Testing ActionSheet'
            },
            (buttonIndex) => {
              this.setState({ clicked: BUTTONS[buttonIndex] });
            }
            )}>
            <Image source={placeholder}  />
      </Button>
    );
  }
}

Screenshot of emulator/device

Is the bug present in both ios and android or in any one of them?
Both

This got skipped in the docs, but you need to add actionSheet inside a <Container /> for it to show properly.

OK - thanks for the info, Sankha. Does the Container need to be the root element on the page? I'm having trouble using ActionSheet inside a Container where it is a sub-element of the root.

Okay, we haven't tested in a nested container scenario. Can you test with Container as the root element? This will help us narrow down the possibilities.

I just tried:
<Container>
  <Content>
    <Button transparent onPress={() => ActionSheet.show(......

as the root level element, but I'm still getting the same error.

Upgrade to RN 0.42.2 and check please.

I've tried upgrading to the latest release (0.42.3), but still no change.

The error is same to me, i have to change the functionality to Fab component.
Thanks.

Unfortunately, I can't use Fab for my case. Sankha, is there any progress on this bug?

I had the same issue and after I checked the node_modules/native-base/src/basic folder I saw that the file Actionsheet.js is not there.
rm -rf node_modules
npm install
solved the problem.

Thanks @predescu. That worked too!

The error is still persisting.

I created a sample app to reproduce.
https://github.com/shikya/NativeBaseTest
Install dependencies and run react-native run-ios to start the app.

@shikya Yes, i have got still the same issue, RN 0.44.0

I am getting the same error, anyone has found a solution?

I am getting this error too. RN 0.48.1
I have created a component that invokes the AS component and then insert the new component in my scene. Look this gist: https://gist.github.com/marcusvbp/67cf22925cb5fc38aa11f85efc7c7e83

Hi @marcusvbp , you should import
import {Root} from 'native-base'

and then
render() {
    return (
      <Root>
      <View style={styles.container}>
        <Button transparent style={styles.btn} onPress={ () => this.showAS() }>
          <Icon name="person" style={{ color: '#666', fontSize: 50 }} />
        </Button>
      </View>
      </Root>
    );
  }

@luiskhernandez Thanks it worked!

The solutions from @luiskhernandez  and @predescu didn't work for me.
The ActionSheet file exists and the  didn't change anything.
In my case I have 2 scenes with the AS component.
The crash occurs when I render the AS in the scene 1 and then go to scene 2 and try to access it.
If I do the opposite, scence 2 first then go to scene 1, it doesn't happen.
The codes are slightly different:
On scene 1 it's used like the docs, but on scene 2 I choose between two sets of BUTTON that I render, with different cancelButtonIndex. The scene one uses a const, but on scence 2 both set a different number. Both scenes use the AS on a  with a .
It crashes before showing the menu or anything. The first output of the red screen says the issue is on the show() method of AS.
Any other solutions? The code is pretty straight forward, but if you need to see it, just let me know.
Thanks in advance.

Adding the root tag worked for me also thanks


JDP Changes from 8.0 (Version)

1. Added Success/Failure Report for every components
Added logic to create a Success/Failure Report for every operation. Earlier we were writing the operation status in the log file itself.
2. Added Exception File  –CSV Report for exception/error
This file gives the information gives the information about exception during the operations.
3. PreMT/Discovery Report Compatible for below commands:

Missing Feature Clean Up:
Now we can use Discovery File (Features_Usage.csv) as well as an input to execute this operation. Earlier it was compatible with PreMT file(PreMT_MissingFeature.csv) only
Missing Event Receivers Clean Up:
Now we can use Discovery File (EventReceivers_Usage.csv) as well as an input to execute this operation. Earlier it was compatible with PreMT file(PreMT_MissingEventReceiver.csv) only

Hi @VirendraKumarSharePoint, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes. I promise there's no faxing. https://cla.microsoft.com.

TTYL, MSBOT;


provide better error message for broken imports

The current error message for a broken import is cryptic and hard to figure out where the error comes from:
 - the error shows as an exception, when I see it, I start scanning the stack trace trying to find something related to my program, but turns out all fo the stack frames are from the system.
 - the message doesn't say where the broken import comes from either. If you have the error somewhere deep in your code, you have to grep all your files to find out where that came from.
It would be great to improve this. Here is an example:
  a.dart:
      import 'b.dart';
      main() {}
Suppose I have not created b.dart. Running dart a.dart today produces this error:
Unhandled exception:
Uncaught Error: FileSystemException: Cannot open file, path = '/usr/local/path/to/current/working/directory/b.dart' (OS Error: No such file or directory, errno = 2)
Stack Trace:
#0      _File.open.<anonymous closure> (dart:io/file_impl.dart:349)
#1      _RootZone.runUnary (dart:async/zone.dart:1155)
#2      _Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:484)
#3      _Future._propagateToListeners (dart:async/future_impl.dart:567)
#4      _Future._completeWithValue (dart:async/future_impl.dart:358)
#5      _Future._asyncComplete.<anonymous closure> (dart:async/future_impl.dart:412)
#6      _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:41)
#7      _asyncRunCallback (dart:async/schedule_microtask.dart:48)
#8      _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:84)
#9      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:131)
#0      _rootHandleUncaughtError.<anonymous closure> (dart:async/zone.dart:886)
#1      _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:41)
#2      _asyncRunCallback (dart:async/schedule_microtask.dart:48)
#3      _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:84)
#4      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:131)
It would be great to produce something like this instead:
  file:///usr/local/path/to/current/working/directory/a.dart: error: line 1 pos 9:
  imported file not found. The import URL was resolved to
  file:///usr/local/path/to/current/working/directory/b.dart, but it was not
  found in your file-system:
    import 'b.dart';
            ^^^^^^

Any updates on this?

@sigmundch's example now gives:
Unhandled exception:
Could not import "file:///C:/Workspace/dart/b.dart" from "file:///C:/Workspace/dart/a.dart": FileSystemException: Cannot open file, errno = 2

which is better.


Improve compiler mismatch experience

Currently we offer rebuild suggestion on module mismatch error message, but if we can get some compiler info that there is an externs mismatch we can be more proactive

This is I guess #134


Strange tab/navbar behavior when pulling from github?

Version
Tell us which versions you are using:
"react": "15.4.2",
"react-native": "^0.42.0",
"react-native-router-flux": "^3.38.0",

Expected behaviour
switching between two tabs does not reload the entire scene's component. I would expect to switch.
Actual behaviour
This only seems to happen with I clone a project form github. I don't really know how else to explain it. The computer I wrote the initial project on still works fine but when I clone it to another tab "switching" instead reloads each tab's scene. Furthermore, the backButton images in the navbar disappear. Has anybody else experienced this?
Again, This only happens on a version I pull from github. the original copy still works fine.

well i'm a dope... i had a different version of RNRF on my original computer. really brain dead today sorry. now to figure out what the difference is...


Maybe for Linux?

Sorry,
The issue not on Linux just Linux based things like replit or other bot Hosting Services

Sorry, The issue not on Linux just Linux based things like replit or other bot Hosting Services

you can just create a new repl and import with the github url, then use the command to start main.py (python main.py)

Get this error everytime

If you want help me better
Discord:  !Schinken#9256

Get this error everytime

reinstall hcaptcha-challanger


fix coredump issue when client not successfully connected but need disconnect

issue:
Core dump will happen when trying to client disconnect which actually not successfully created.

@jikunshang Please take a review. Thanks.


POCOs domain classes and anemic domain model

[Enter feedback here]
Following the link to POCO's definition we see that they should include only props/fields and no methods. In this article we see that domain model classes should be POCOs. Isn't that the definition of an anemic domain model?
Thank you in advance.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 95f3dcd7-3220-e9cf-3ca6-90b1282410ec
Version Independent ID: 5a30bea5-326b-ea72-59ae-dfef1e190e4a
Content: Designing a DDD-oriented microservice
Content Source: docs/architecture/microservices/microservice-ddd-cqrs-patterns/ddd-oriented-microservice.md
Product: dotnet-architecture
Technology: microservices
GitHub Login: @nishanil
Microsoft Alias: nanil

Hi @kosletr, thanks for posting here, for more information on the anemic domain model please refer to the link: https://learn.microsoft.com/en-us/dotnet/architecture/microservices/microservice-ddd-cqrs-patterns/microservice-domain-model. I hope it will answer your question.

Closed per @erjain


Refuse startup with no server admin set up

In the context of https://github.com/apache/couchdb/issues/2191, we wanted to forbid starting CouchDB without an admin user. This patch does this.
Please bike shed on the error message and whatnot. I’ve picked a format that is different from all other error messages so that this particular thing very visible before the boot error garbage is thrown. If somebody knows how make this even nicer, do let me know.
Here is an example of how CouchDB exits without a server admin configured:
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
[info] 2020-01-03T18:25:20.832839Z<EMAIL_ADDRESS><0.281.0> -------- Preflight check: Asserting Admin Account

[info] 2020-01-03T18:25:20.835798Z<EMAIL_ADDRESS><0.281.0> --------
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  No Admin Account Found, aborting startup.
  Please configure an admin account in your local.ini file.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

{"init terminating in do_boot",{{error,{bad_return,{{couch_app,start,[normal,[]]},admin_account_required}}},[{boot_node,start_app,3,[{file,"dev/boot_node.erl"},{line,146}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{boot_node,start_app,3,[{file,"dev/boot_node.erl"},{line,136}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{boot_node,start_app,3,[{file,"dev/boot_node.erl"},{line,136}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ({{error,{bad_return,{{_},admin_account_required}}},[{boot_node,start_app,3,[{_},{_}]},{lists,foldl,3,[{_},{_}]},{boot_node,start_app,3,[{_},{_}]},{lists,foldl,3,[{_},{_}]}

Crash dump is being written to: erl_crash.dump...done
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed

ah needs some work to run with the test suite.
we probably need an override. How about setting an environment variable in make eunit, something like COUCH_TEST_ADMIN_PARTY_OVERRIDE that, if set, disables the mechanisms that refuses to start without an admin pass?

@janl Had this dance with @rnewson recently who tried to special case a setting in the .ini file. He was able to work around it using ./rel/files/eunit.ini for eunit, and modifying dev/run for the latter.
We already have a command line way to pass a static admin:password to dev/run so perhaps just pass that in and bake admin:password as the auth into the tests? I bet we can do it in one place for each of Elixir and JS, except maybe for any admin level tests.
Personally I'd rather not have an escape hatch on this one, otherwise people will find it and abuse it, just as they did when we rushed 2.0 out the door and left 5986 open.

@wohali the problem is that we have tests that rely on setting/changing admin-party, so it’s not that we could just set a admin/pass, we’d have to rework many gnarly tests which might exceed our time budget for this.
I hear you on the escape hatch, my idea was make this a COUCH_TEST prefix. One additional thing we could do is add a little process that logs an error message once a minute or hour if that variable is set and no admin is configured, just to make sure folks get the message.

One thing we’d not do is document this escape hatch, which sets it apart from 5986 which we didn’t leave open by accident, but advertised its use for administrative purposes.

@janl ok, if you don't want to do the heavy lifting around fixing all the tests as a blocker, I understand. We at least have an easy way to get something into the .ini file, that might be better than an env var...unless you want to do something sneaky like put it in the app context (though IIRC that requires a recompile, doesn't it?)
Good idea on periodic logging. How about every 5 minutes? :) We should really be noisy about this.

FYI Jan and I are having a sidebar on IRC; he's reducing the sleep to 500ms to deal with some couch_log latency problems.
When I get back from lunch, I will do a test package of this PR and see if systemd correctly throttles restarting when no admin is configured.

squashed it all into one commit, includes the reduced timeout, escape hatch and annoyance reporter.

and the Makefile fix

Tests still failing, will need to look more closely

Aha, of course, my advice was wrong. I am working on the fix now.

This should have fixed it, tests are passing locally.
The unrelated Python change was to make make python-black succeed, the test was actually broken by the addition of $(TEST_OPTS) and was missing the formatting problem in the new file build-aux/show-test-results.py. (fyi @davisp )
Setting to approved, I'll merge tomorrow if no one else notices it first.

@janl tag. Forgot to update Makefile.win, would you please? thanks :) zzzz.

ported to Makefile.win as best I could, and simplified the annoyance process to be only spawned when the env var is set, instead of checking the env var on each iteration, which won’t ever change.

This is a breaking change for many users. I understand the implications of Admin Party so this change just breaks a bunch of working setups.
Wasn't the fact that it broke your own tests a bit of a clue? Why isn't there an easy way to disable the check?
Would you accept a PR to make it possible to disable this check? I've tried setting COUCHDB_TEST_ADMIN_PARTY_OVERRIDE=1 but that didn't seem to work for me. Will investigate further but using options intended for testing doesn't seem like a good idea long term.
Thanks.

@AndyA That's why this became 3.0, not 2.4. We introduced many breaking changes in 3.0, all of which are documented at http://docs.couchdb.org/en/stable/whatsnew/3.0.html#deprecated-feature-removal .
The lack of solid out-of-the-box security in CouchDB has been a concern for a very long time, expressed by many, and amplified by some high profile leaks of CouchDB-hosted data on the web. CouchDB 0.x was written in the spirit of an open web, open data, but as people started to rely on CouchDB for more secure needs, pressure increased on the project to move in this direction.
We wouldn't accept the proposed patch, sorry.

Thanks @wohali. I understand the desire for this to be the default but don't see the downside to being able to disable it (with suitably dire warnings in the documentation).
As far as I can see the other breaking changes are mainly a concern for server admins. This potentially has far wider effects. We have many scripts that assume password free admin access all of which have to be fixed before we can move to 3.0.
we_understand_the_risks_and_we_still_want_to_party = "I PROMISE NOT TO SUE"

You're welcome to stay on 2.x or maintain a custom patch. The project is not going to change course here.

It seems that anyone who's installing from https://apache.bintray.com/couchdb-deb will have automatically received the breaking change.
Is there a repo address that's pinned to 2.x?

@AndyA Standard deploy practice should be:
apt install couchdb=2.3.1
If you're not pinning your production environment to specific versions, you're going to get unexpected surprises :wink:

Thanks.
This isn't production and I'm aware of how to pin a particular version :)

@AndyA Sorry about that. We're appreciative of your support of CouchDB over the years :)
We're not planning to separate out into different release streams for 3.x, but we probably will for CouchDB 4, since it's a complete replacement of the storage and clustering code.

Thanks @wohali :)


When the exception was thrown, this was the stack:

Steps to Reproduce

...
...
...

Logs

Please provide the error log and your code.

Closing due to lack of content. Please reopen if you have more information to add! Thanks.

══╡ EXCEPTION CAUGHT BY GESTURE ╞═══════════════════════════════════════════════════════════════════
2019-07-11 12:18:40.535 2945-3011/com.example.login_flutter I/flutter: The following assertion was thrown while handling a gesture:
2019-07-11 12:18:40.535 2945-3011/com.example.login_flutter I/flutter: Navigator operation requested with a context that does not include a Navigator.
2019-07-11 12:18:40.535 2945-3011/com.example.login_flutter I/flutter: The context used to push or pop routes from the Navigator must be that of a widget that is a
2019-07-11 12:18:40.535 2945-3011/com.example.login_flutter I/flutter: descendant of a Navigator widget.
2019-07-11 12:18:40.537 2945-3011/com.example.login_flutter I/flutter: When the exception was thrown, this was the stack:
2019-07-11 12:18:40.547 2945-3011/com.example.login_flutter I/flutter: #0      Navigator.of. (package:flutter/src/widgets/navigator.dart:1449:9)
2019-07-11 12:18:40.547 2945-3011/com.example.login_flutter I/flutter: #1      Navigator.of (package:flutter/src/widgets/navigator.dart:1456:6)
2019-07-11 12:18:40.547 2945-3011/com.example.login_flutter I/flutter: #2      Navigator.pushNamed (package:flutter/src/widgets/navigator.dart:877:22)
2019-07-11 12:18:40.548 2945-3011/com.example.login_flutter I/flutter: #3      TabLayoutDemo.build. (package:login_flutter/tabbarbottom.dart:109:49)
2019-07-11 12:18:40.548 2945-3011/com.example.login_flutter I/flutter: #4      _InkResponseState._handleTap (package:flutter/src/material/ink_well.dart:511:14)
2019-07-11 12:18:40.548 2945-3011/com.example.login_flutter I/flutter: #5      _InkResponseState.build. (package:flutter/src/material/ink_well.dart:566:30)
2019-07-11 12:18:40.548 2945-3011/com.example.login_flutter I/flutter: #6      GestureRecognizer.invokeCallback (package:flutter/src/gestures/recognizer.dart:166:24)
2019-07-11 12:18:40.549 2945-3011/com.example.login_flutter I/flutter: #7      TapGestureRecognizer._checkUp (package:flutter/src/gestures/tap.dart:240:9)
2019-07-11 12:18:40.549 2945-3011/com.example.login_flutter I/flutter: #8      TapGestureRecognizer.handlePrimaryPointer (package:flutter/src/gestures/tap.dart:177:9)
2019-07-11 12:18:40.549 2945-3011/com.example.login_flutter I/flutter: #9      PrimaryPointerGestureRecognizer.handleEvent (package:flutter/src/gestures/recognizer.dart:436:9)
2019-07-11 12:18:40.549 2945-3011/com.example.login_flutter I/flutter: #10     PointerRouter._dispatch (package:flutter/src/gestures/pointer_router.dart:73:12)
2019-07-11 12:18:40.550 2945-3011/com.example.login_flutter I/flutter: #11     PointerRouter.route (package:flutter/src/gestures/pointer_router.dart:101:11)
2019-07-11 12:18:40.550 2945-3011/com.example.login_flutter I/flutter: #12     _WidgetsFlutterBinding&BindingBase&GestureBinding.handleEvent (package:flutter/src/gestures/binding.dart:221:19)
2019-07-11 12:18:40.550 2945-3011/com.example.login_flutter I/flutter: #13     _WidgetsFlutterBinding&BindingBase&GestureBinding.dispatchEvent (package:flutter/src/gestures/binding.dart:199:22)
2019-07-11 12:18:40.550 2945-3011/com.example.login_flutter I/flutter: #14     _WidgetsFlutterBinding&BindingBase&GestureBinding._handlePointerEvent (package:flutter/src/gestures/binding.dart:156:7)
2019-07-11 12:18:40.551 2945-3011/com.example.login_flutter I/flutter: #15     _WidgetsFlutterBinding&BindingBase&GestureBinding._flushPointerEventQueue (package:flutter/src/gestures/binding.dart:102:7)
2019-07-11 12:18:40.551 2945-3011/com.example.login_flutter I/flutter: #16     _WidgetsFlutterBinding&BindingBase&GestureBinding._handlePointerDataPacket (package:flutter/src/gestures/binding.dart:86:7)
2019-07-11 12:18:40.551 2945-3011/com.example.login_flutter I/flutter: #20     _invoke1 (dart:ui/hooks.dart:233:10)
2019-07-11 12:18:40.551 2945-3011/com.example.login_flutter I/flutter: #21     _dispatchPointerDataPacket (dart:ui/hooks.dart:154:5)
2019-07-11 12:18:40.552 2945-3011/com.example.login_flutter I/flutter: (elided 3 frames from package dart:async)
2019-07-11 12:18:40.556 2945-3011/com.example.login_flutter I/flutter: Handler: onTap
2019-07-11 12:18:40.556 2945-3011/com.example.login_flutter I/flutter: Recognizer:
2019-07-11 12:18:40.556 2945-3011/com.example.login_flutter I/flutter:   TapGestureRecognizer#88d7e(debugOwner: GestureDetector, state: possible, won arena, finalPosition:
2019-07-11 12:18:40.556 2945-3011/com.example.login_flutter I/flutter:   Offset(110.3, 625.0), sent tap down)

Thanks. This is operating by design, however. As the earlier lines show, you need a Navigator object to send routing commands to. You might want to review: https://flutter.dev/docs/cookbook/navigation/navigation-basics


Support exact filer matching

I'd like to be able to specify exact files such as
rec {
  artifact_a = nix-filter {
     root = ../../;
     include = [
        ./relative/path/to/a.ext
        ./another/path/to/b.ext
     ];
  };

  artifact_b = nix-filter {
     root = ../../;
     include = [
        artifact_a
        ./third/path/to/c.ext
     ];
  };
}

And i'd expect the output for artifact_b to contain
relative/path/to/a.ext,
another/path/to/b.ext,
third/path/to/c.ext,
keeping the directory heirarchy.
Why?
It would allow the user to composable build src lists. Artifact A needs files x, y, artifact B needs A + file z, whilst maintaining their relative paths. Unsure how to achieve file level granularity currently

I'm currently working on a file set combinator library for Nixpkgs which allows you to do essentially this!
let
  commonFiles = lib.fileset.unions [
    ./relative/path/to/a.ext
    ./another/path/to/b.ext
  ];
in {
  a = lib.fileset.toSource {
    root = ../..;
    fileset = commonFiles;
  };
  b = lib.fileset.toSource {
    root = ../..;
    fileset = lib.fileset.union
      commonFiles
      ./third/path/to/c.ext;
  };
}


Could docker network configure the IPAM with already configured bridge addressing ?

Hello !
TL;DR:
When I create a docker network with a bridge that is already configured, Docker overrides the addressing I've set up. Could it be possible that when I create a docker network with an already configured bridge, if I do not supply any additionnal parameters, it configures the IPAM with the in place bridge configuration ?
More details
What I have :
root@machine # ip address show br0
27: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0c:c4:7a:07:6a:7f brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/24 brd <IP_ADDRESS> scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:fe07:6a7f/64 scope link
       valid_lft forever preferred_lft forever

Then I create my network
root@machine # docker network create --driver bridge -o com.docker.network.bridge.name=br0 mynetwork
c228be0db663a5add797899688d9feb8c2699fc4b0a309edd0476a88526e576b
root@machine # docker network inspect c228be0db663a5add797899688d9feb8c2699fc4b0a309edd0476a88526e576b
[
    {
        "Name": "mynetwork",
        "Id": "c228be0db663a5add797899688d9feb8c2699fc4b0a309edd0476a88526e576b",
        "Scope": "local",
        "Driver": "bridge",
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "<IP_ADDRESS>/16",
                    "Gateway": "<IP_ADDRESS>/16"
                }
            ]
        },
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.name": "br0"
        }
    }
]
root@machine # ip address show br0
27: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0c:c4:7a:07:6a:7f brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/16 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:fe07:6a7f/64 scope link
       valid_lft forever preferred_lft forever

The bridge addressing went from <IP_ADDRESS>/24 to <IP_ADDRESS>/16. Could it be relevant to use already in place configuration ?
Also, side remark, could the bridge not be deleted if it has not been created with Docker networks ?
Thanks a lot !

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
For more information about reporting issues, see CONTRIBUTING.md.
You don't have to include this information if this is a feature request
(This is an automated, informational response)

BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
#ENEEDMOREINFO

I believe this is relevant to #20123

AFAIK @blazedd this is unrelated to #20123
@thomas-maurice regarding your question, custom network doesn't provide the -b like behavior to retain and reuse an already created bridge. Since you are using the option com.docker.network.bridge.name, bridge driver creates a bridge with the given name (and it seems it is greedy). You can use the --subnet & --gateway option to setup the preferred subnet and bridge-ip. But if you wish to do any manual plumbing of the created bridge, it must be done after it is created via docker network create command.

@mavenugo yes but that is the "issue" I am pointing out. If I need to specify somehow the bridge I want docker to use, it is because I need a custom one, and probably I need one because I have a custom network configuration attached to it, that I want to manage and have full control over it.
So would it be possible to implement a -b libe behavior for custom network ? So that all my already configured bridges can be reused, and also not be deleted if I decide to destroy the network ?
Also this issue could be retagged "feature request" couldn't it ?
Regards :)

I think its useful because OS networking is configured before docker daemon starts. At that earlier time docker daemon isnt running yet and we cant run some cmds like docker network create ..<bridge params>... And it is best place to setup persistent bridges (that stay after reebots) from the file /etc/network/interfaces. Like have to bridge to eth0 then, setting up parameters. It needs to re-route everything through the bridge. So there is a brief lost network connectivity on the host. That is best kept at boot time, and not after docker-daemon has started up. Because then if restart=always some containers may have already been auto-started by then, etc.

@thomas-maurice @dreamcat4 There are a few complexities with such a requirement. Especially with IPAM. As per the design, IPAM is managed independently of the network driver. So, if the user wishes the network to use the subnet as specified in the user-created bridge, the user must use the --subnet and --gateway options to match that of the bridge. If the user doesn't specify these IPAM options while creating a network, then the IPAM driver will choose any available network subnet that may not match the user-managed bridge.
Other than that, I dont see any other issue with using the com.docker.network.bridge.name driver option to not create a bridge if there is an user-managed bridge already present.
@aboch wdyt ?

If the user doesn't specify these IPAM options while creating a network, then the IPAM driver will choose any available network subnet that may not match the user-managed bridge.

Yes indeed. They dicover the behaviour those --subnet and --gateway options now. Although maybe it could be better in some proper documentation / which I cannot seem to find just yet. We may also be concerned for other driver options which we do not fully know. For example if the driver is also overriding other settings of the bridge.

We may also be concerned for other driver options which we do not fully know. For example if the driver is also overriding other settings of the bridge.

@dreamcat4 sure. and hence my response :

Other than that, I don't see any other issue with using the com.docker.network.bridge.name driver option to not create a bridge if there is an user-managed bridge already present.

@thomas-maurice  I have the exact same requirement. Have you got anywhere with this?

@gedl nope. I ended up not using docker networks at all and doing my own networking cooking at the OS level

@thomas-maurice  I am almost tempted to just remove the "if" that prevents users from specifying an IP in a pre-defined network. My opinion is this restriction is intrusive and useless. Docker can't prevent every user-induced mess up, specially not without reducing functionality and options, like this is the case.
I still have some hope @aboch will reconsider and lobby to remove the restriction (that's definitely easier than allowing a -b switch on network creation).

@thomas-maurice were you at least able to delete the intruding default bridge network? even specifying -b=none is still creating me a useless default bridge network, occupying a range of IPs.

Well I used the default bridge with address I forcedly assign, and then built my own networking stack on the top of it.

I just assigned a /30 range (the smaller it accepts). So annoying having to
remember that those address can't be used anywhere else in the network.
I managed to delete the default network once, in another installation, but
I can't remember how I did it. This was great because with -b=none there
was no leftovers of the default networking (which are useless at best in my
use case).
Gonçalo Luiz
On 20 May 2016 at 06:53, Thomas Maurice<EMAIL_ADDRESS>wrote:

Well I used the default bridge with address I forcedly assign, and then
built my own networking stack on the top of it.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/20349#issuecomment-220523194

@thomas-maurice
Regarding

, and also not be deleted if I decide to destroy the network ?

@mountkin 's change at https://github.com/docker/libnetwork/pull/1301 provides the above and will be available in docker 1.13


firefox add-on disapeared

Would be great to have this in firefox. Thanks!

Yes, firefox send me a email which said that I use webpack to uglify source code isn't safe. It's really annoying, but I will fix it anyway.

That's weird 🤔

It's used to happenn in #19. 😢

use webpack to uglify source code isn't safe

Because of eval again?

It should be available now. I just submit a new version add-on with a instruction file which explains how to build the add-on. Hope firefox won't disable this again.

Yeah! It back now.


Killed when saving LoRA weights

Hi All - has anyone come across this error when saving there lora weights? Notice the Killed.
Config
  litgpt finetune lora \
    --precision bf16-true \
    --quantize bnb.nf4 \
    --data JSON \
    --data.json_path /data_dir/data.json \
    --data.val_split_fraction 0.1 \
    --data.mask_prompt False \
    --data.seed 42 \
    --data.num_workers 4 \
    --lora_r 2 \
    --train.epochs 1 \
    --train.micro_batch_size 2 \
    --checkpoint_dir sky_workdir/checkpoints/mistralai/Mistral-7B-Instruct-v0.1 \
    --logger_name wandb

Logs:
Epoch 1 | iter 938 step 234 | loss train: 0.383, val: 0.439 | iter time: 632.50 ms
Epoch 1 | iter 939 step 234 | loss train: 0.310, val: 0.439 | iter time: 543.10 ms
Epoch 1 | iter 940 step 235 | loss train: 0.352, val: 0.439 | iter time: 384.06 ms (step)
Epoch 1 | iter 941 step 235 | loss train: 0.410, val: 0.439 | iter time: 564.47 ms
Epoch 1 | iter 942 step 235 | loss train: 0.473, val: 0.439 | iter time: 382.97 ms
Epoch 1 | iter 943 step 235 | loss train: 0.501, val: 0.439 | iter time: 382.01 ms
Epoch 1 | iter 944 step 236 | loss train: 0.569, val: 0.439 | iter time: 393.79 ms (step)
Epoch 1 | iter 945 step 236 | loss train: 0.622, val: 0.439 | iter time: 544.00 ms
Epoch 1 | iter 946 step 236 | loss train: 0.525, val: 0.439 | iter time: 737.82 ms
Epoch 1 | iter 947 step 236 | loss train: 0.530, val: 0.439 | iter time: 377.37 ms
Epoch 1 | iter 948 step 237 | loss train: 0.516, val: 0.439 | iter time: 540.38 ms (step)
Epoch 1 | iter 949 step 237 | loss train: 0.423, val: 0.439 | iter time: 656.10 ms
Epoch 1 | iter 950 step 237 | loss train: 0.458, val: 0.439 | iter time: 547.91 ms
Epoch 2 | iter 951 step 237 | loss train: 0.413, val: 0.439 | iter time: 856.00 ms
Training time: 683.34s
Memory used: 14.29 GB
Saving LoRA weights to '/sky_workdir/final/lit_model.pth.lora'
Killed

Hardware:
A10G (24GB memory)
CUDA information:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.12             Driver Version: 535.104.12   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A10G                    On  | 00000000:00:1E.0 Off |                    0 |
|  0%   16C    P8              16W / 300W |      2MiB / 23028MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Jun_13_19:16:58_PDT_2023
Cuda compilation tools, release 12.2, V12.2.91
Build cuda_12.2.r12.2/compiler.32965470_0

Haven't seen that yet but it could be due to memory limitations when merging with the original model. Wdyt @awaelchli ?
Btw otherwise it looks like everything was successful. You can try to manually merge the weights via the litgpt merge_lora command.

Yes, that means you probably ran out of CPU memory. Right now, merging LoRA parameters requires the entire checkpoint to fit in memory.

@alistairwgillespie How much RAM does your system have?

I opened https://github.com/Lightning-AI/litgpt/pull/1189 which seems to help for me. It would be interesting if you could try it, Alistair.

I think that the biggest issue is that the training was done with quantization. But merge_lora.py doesn't support it.
We can load the model in quantized form, merge weights. But when it comes to saving the model we have a couple of options:

If we want to save the model in quantized form there shouldn't be any significant problems. Plus the latest BNB should support it.
If we want to save in dequantized form, then we have to add incremental dequantization and saving. Otherwise the whole model has to be dequantized right before saving, which significantly increases memory consumption.
https://github.com/Lightning-AI/pytorch-lightning/issues/19242
Or just stick to the current approach and ~hope~ assume that a user have more (much more in fact) free CPU RAM than GPU VRAM.

I want to also mention that we might have a problem with merging quantized weights: https://github.com/Lightning-AI/litgpt/issues/935

@Andrei-Aksionov For now the easiest thing is to save it dequantized. The LoRA merge already supports this (you added this!). If it's not working well then we should fix it

Yes, I even remember adding it 😆.
You can quantize a model upon loading (thanks to Fabric), merge while keeping the model in quantized form (guess here kudos goes to me), but if you want to save a model in a dequantized form you need to first to dequantied the whole model.

If it's not working well then we should fix it

The fix would be an incremental dequantization/saving:

Take a layer that we want to save
Dequantize it
Save it
Go back to step 1

Apologies for the late reply, All. I updated the hardware notes in the original issue for audit purposes. Boosting the hardware fixed the issue. The additional feedback was helpful too.


Signup link not work!

When I clicked on the signup button I saw that nothing happened.

Hi @dhra!
Thank you for taking the time to report the issue, it will be addressed with the upcoming release : )


Add DSTT link to nds-cards.com

May as well link to the DSTT in the guide, like the R4i and Acekard 2i.
https://3ds.guide/ntrboot

https://github.com/Plailect/Guide/commit/d2ba02a834aeb3521a9be89e5278fb191e30515d


Add unique request id

Q
A

Is bugfix?
❌

New feature?
✔

Breaks BC?
❌

Tests pass?
✔

There's https://github.com/yiisoft/yii-web/blob/master/src/Middleware/TagRequest.php for the purpose.

There's https://github.com/yiisoft/yii-web/blob/master/src/Middleware/TagRequest.php for the purpose.

It is optional. This PR is for another PR.


Bump minimum golang version to 1.13.8

Recent versions contain security-related fixes in the Go runtime.

I would like to note that the release GitHub Actions workflow currently automatically uses the latest Go 1.13.x release:
https://github.com/oasislabs/oasis-core/blob/ab684b462b77869080c8ac6cf4f4c2c0abb3e25c/.github/workflows/release.yml#L21-L24


Please update to react-select 1.2.0

:)

Specifically this PR fixes an issue where multiple dropdowns can be open at once: https://github.com/JedWatson/react-select/pull/2213

We attempted the update, but ran into some tricky merge conflicts that have yet to be worked out. We'll try to get through them.
In the meantime, you might want to check out the alpha of react-select v2, which adds option group support. If that release works well, we'll be discontinuing this fork.


Python 3.13 compatibility

This adds the changes necessary to run changedetection.io with Python 3.13.

thanks again - tests are being a bit flakey, i'll keep pushing and see whats up

https://github.com/microsoft/playwright-python/releases/tag/v1.48.0
any chance you can bump the playwright version to 1.48.0 ?

some other python 3.13 issue i found in the queue that this probably closes https://github.com/dgtlmoon/changedetection.io/issues/2759

Looks like one of the builders got stuck for some reason.

hey cool it passes!!


chore(type): env.d.ts add exports

PR Checklist
Please check if your PR fulfills the following requirements:

[ ] Tests for the changes have been added (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[ ] infrastructure changes
[ ] Other... Please describe:

What is the current behavior?

What is the new behavior?
Relate issue #1505, package.json add env.d.ts export(only type)
Other information

Thanks


self.label?

https://github.com/MIC-DKFZ/basic_unet_example/blob/45dd8b8e6767448ff859203c834d01bd94f68ab9/datasets/three_dim/NumpyDataLoader.py#L164
Should the self.input be replaced by self.label?

Sorry for the late reply. I was not looking into this repo for quite some while. I think you are right, I will have a look and correct it. Thank you!


bug: cortex models list doesnt return previously downloaded models

Cortex version
0.5.0-123
Describe the Bug
Broken after migration from models.list to sqlite db https://github.com/janhq/cortex.cpp/pull/1336

previously downloaded models appear on v118 cortex models list
after updating to v123, returns empty table

Steps to Reproduce
No response
Screenshots / Logs
Linux

Mac

What is your OS?

[X] MacOS
[ ] Windows
[X] Linux

What engine are you running?

[X] cortex.llamacpp (default)
[ ] cortex.tensorrt-llm (Nvidia GPUs)
[ ] cortex.onnx (NPUs, DirectML)

@gabrielle-ong this is not relevant, btw versions 118 and 123, eng rebuilt the model.list into model.db.
Please close this issue and associate it with the issue tracking model.db.

Closing as not a bug
models list after 0.5.0-v123 not backward compatible due to moving from models.list to sqlite db - https://github.com/janhq/cortex.cpp/issues/1241
Workaround (for reference)

To avoid downloading all models: run cortex-nightly pull <model_id / HF model >, then select N to not download the model. The model info will be saved to the DB
Requires redownloading all models: Delete cortex-nightlycpp/models folder, and re-pull all models to save to DB.


BytesMut lacks a direct implementation of std::io::Write

I'm currently using the codec module to implement a framed tansport.
Before the refactoring, the encode-method of the old Codec trait received a Vec<u8> as parameter to serialize the message into. Vec<u8> directly implements std::io::Write which makes it very easy to put any kind of data into it, since most implementations have some way of writing into an instance of Write.
Now the Vec<u8> has been replaced by BytesMut, which lacks a direct implementation of that trait. Write-compatibility is only archievable through into_writer, which consumes the BytesMut. Consuming is not possible, though, because the method only receives the BytesMut by &mut.
Is this desired behavior?

into_writer should work because BufMut is also implemented on &mut BytesMut. So you are passing ownership of the ref, not the value.
That being said, unless there is a coherence problem (which I don't think there is) I don't see why BytesMut couldn't impl Writer directly.

Could you open an issue in the bytes repo and reference this one?

into_writer should work because BufMut is also implemented on &mut BytesMut.

Right now it doesn't, unfortunately.

Could you open an issue in the bytes repo and reference this one?

Sure! Thanks for the quick support!

Can't you use BufMut::writer? This is working fine for me, for the reasons @carllerche gave. I don't even see an into_writer method.

Oh man, sorry. You guys are right. I was mislead.


TestMissingAnyFills::TestMonth random failures

Description

TestMissingAnyFills::test_month tends to fail intermittently, particularly on the macOS Python3.7 build but also on Linux and Windows at a lesser frequency. Caution suggests that if this specific case is failing sometimes, there must either be a bug within xclim or xarray or there is some RNG-dependent call or test setup that we've coded that has some failing edge cases.
I'm not sure what the urgency of this bug is but it's been plaguing TravisCI builds for several months so I'm opening an issue to at least have this on the board. Here's the stack:
________________________ TestMissingAnyFills.test_month ________________________
self = <test_checks.TestMissingAnyFills object at 0x11b253150>
tasmin_series = <function tasmin_series.<locals>._tasmin_series at 0x11a409f80>

    def test_month(self, tasmin_series):
        ts = tasmin_series(np.zeros(36))
        miss = checks.missing_any(ts, freq="YS", month=7)
        np.testing.assert_equal(miss, [False])

        miss = checks.missing_any(ts, freq="YS", month=8)
        np.testing.assert_equal(miss, [True])

        with pytest.raises(ValueError, match=r"No data for selected period."):
            miss = checks.missing_any(ts, freq="YS", month=1)

        miss = checks.missing_any(ts, freq="YS", month=[7, 8])
        np.testing.assert_equal(miss, [True])

        ts = tasmin_series(np.zeros(76))
        miss = checks.missing_any(ts, freq="YS", month=[7, 8])
>       np.testing.assert_equal(miss, [False])

E       AssertionError:
E       Items are not equal:
E        ACTUAL: <xarray.DataArray (time: 1)>
E       array([ True])
E       Coordinates:
E         * time     (time) datetime64[ns] 2000-01-01
E        DESIRED: [False]

Hypothesis
On line 206-8 of checks.py::missing_any(), we do:
sda = xr.DataArray(data=np.empty(len(t)), coords={"time": t}, dims=("time",))
st = generic.select_time(sda, **indexer)
sn = st.notnull().resample(time=freq).sum(dim="time")

AFAIK, np.empty creates an empty array, without initializing its values, so you get what was in the memory before the initialization. There is a non-null chance that a NaN was there. Thus, both in generic.select_time and st.notnull(), this value is dropped and the test fails.
Whatcha think?
I have tried the current tests many time on two different computers and never had a fail, so testing this is difficult...

I think that's a very real possibility. Good thinking. I suggest replacing by something like:
sda = xr.DataArray(data=np.ones(len(t), dtype=bool), coords={"time": t}, dims=("time",))
st = generic.select_time(sda, **indexer)
sn = st.resample(time=freq).sum(dim="time")


Add Swift Package Manager Support

Adopt support for Swift Package Manager (it will integrate directly with Xcode 11 via Github)

Fixed by https://github.com/ProjectDent/ARKit-CoreLocation/pull/194


Custom Login Field

Added the ability to setup a custom login field other than email.
Added loginField to default options with value 'email'.
Updated SignInData interface to accept login instead of email attribute.
Updated Angular2TokenService to use loginField option to send auth info to server.

Having a non-email login option makes sense, but be aware that this breaks compatibility with the default configuration for devise_token_auth, since its User model requires an email attribute.
Since going with devise_token_auth is pretty much the default option, it might be wise to add a link to this devise_token_auth doc page to the angular-token docs.

This functionality has been added to version 6.0.0-rc.1. It defaults to email obviously.
Please read the Configuration Guide for the loginField.

I should've explained more clearly. It breaks compatibility because the registerAccount method does not take into account the value of loginField (the way the signIn method does), and just sends a stringified registerData object, which has a login property, not an email property.

I agree. But I assume that if someone asks for this functionality, they are aware of the implications.

Is it possible to update the registerAccount method to match the signIn method? Something like this:
let body = JSON.stringify({
  password_confirmation:          registerData.passwordConfirmation,
  confirm_success_url:            this.atOptions.registerAccountCallback,
  [this.atOptions.loginField]:    registerData.login,
  password:                       registerData.password,
  name:                           registerData.name,
  userType:                       registerData.userType
});

return this.post(this.getUserPath() + this.atOptions.registerAccountPath, body);


nested batch actions

support nested batch actions
actionCreators.batch(
  actionCreators.batch(
    // ...
  )
  // ...
)

https://github.com/brietsparks/normalized-reducer/commit/101581d258597569faf75ed12068795e224b2aad


what is "unk"?

when i do -v i see:
volume metadata:
  model: DELL PERC S110
  revision: 3.0
  serial number:<PHONE_NUMBER>69
software metadata:
  name: HDD Raw Copy Tool
  version: 1.10
image metadata:
  size: 930.48 GiB<PHONE_NUMBER> sectors * 512 bytes)
  unk1:<PHONE_NUMBER>000008
  unk2:<PHONE_NUMBER>0000fa
  unk3: 00
-0.00%<PHONE_NUMBER>3709551615 /<PHONE_NUMBER>12 bytes)...

what are unk1 unk2 unk3?

These are fields in the image metadata I hadn't figured out yet at that point. It's short for unknown. I should take a closer look again one of these days...

Hah! It seems I had a pending commit in my local copy implementing just this. I'll see if I can clean it up and push it, but for reference:

unk1: Windows bus type (8 = RAID);
unk2: maximum transfer size
unk3: whether it was a removable device


gphotos-sync: init at 2.14.2

Motivation for this change
gphotos-sync is a backup photos and videos from Google Photos. This packages it, along with two dependent libraries.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 116570 at 8c70cff5 run on x86_64-linux 1

5 packages built successfully:

gphotos-sync
python38Packages.exif
python38Packages.plum_py
python39Packages.exif
python39Packages.plum_py

@zseri Could you explain why you closed this? Is there something wrong with the PR?

Oh, sorry. Didn't notice this. Probably clicked the wrong button.

Ah, got it. I saw a few overrides in python-modules and that seemed like the way to go, but now that I look, I see a lot of substituting version bounds also.
I changed it to relax the version bound. It builds and imports fine. I manually ran the tests (from the git repo) with the newer dependency version and they all pass.

(I'm not sure of the etiquette here yet, am I supposed to always "re-request review" after responding to comments?)

@dnr you should mark the review comments which you fixed as "resolved"

(I'm not sure of the etiquette here yet, am I supposed to always "re-request review" after responding to comments?)

I am just having a really high back log for things to take a look at.

@dnr you should mark the review comments which you fixed as "resolved"

That doesn't seem right.. shouldn't the person who raised the comment decide whether it's resolved or not? What if they're unhappy with my answer? (I did "resolve" the trivial ones, btw, but not the ones that are more subjective.)

I am just having a really high back log for things to take a look at.

@SuperSandro2000 No worries! I just want to know what I should do to keep things moving forward and not get lost. Whether that's "just wait" or "request re-review", I'm happy to do either, just let me know.

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 116570 run on x86_64-linux 1

7 packages built:

gphotos-sync
python38Packages.baseline
python38Packages.exif
python38Packages.plum-py
python39Packages.baseline
python39Packages.exif
python39Packages.plum-py


N3IWF test NAS message issues

the binary only library function nasTestpacket.GetUlNasTransport_PduSessionEstablishmentReques does not put the PDU-SessionType IE into the nas message.

hi @bjoern-r ,
In TS 24.501, this IE is defined as optional, so we didn't provide this IE.
Is there any procedure must using this IE?

sorry you are right. the other core which i tested with your network function expected a pdu-sessionType. i was able to add a fallback to ipv4 to make it work.
thankyou. anyway as you are expecting to establish a ipv4 pdu session it would make sense to add this IE.

hi @bjoern-r ,
For now, we have open source all our library. If there is some needed to add new IE, it's welcome to contribute and help us to improve the program. 😃

thanks!
i will have a look however i'm not an go expert..


component governance tools

Description

Tools to automatically execute the component governance
Related Issues

https://github.com/microsoft/recommenders/pull/888
Checklist:

[ ] I have followed the contribution guidelines and code style for this project.
[ ] I have added tests covering my contributions.
[ ] I have updated the documentation accordingly.

hey @hlums @jainr I did a pipeline that automatically kicks off the component governance, parse everything add the information to ADO. After this is merged in Recommenders I'll do a PR to CV and NLP.
Please take a look


CPU offloading with BLIP2 model [BUG]

Describe the bug
Does deepspeed support CPU offloading for BLIP2 model ?
To Reproduce
When I try to enable stage 3 + cpu offloading on the BLIP2 model, I observe that the memory consumption does not drop and stays at around 10 GB.
I tried the following code snippet:
# command: deepspeed --num_gpus 1 blip2_deepspeed_manual_config.py

from PIL import Image
import requests
from transformers import Blip2Processor, Blip2ForConditionalGeneration, Blip2Config
import torch
import deepspeed

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
model = Blip2ForConditionalGeneration.from_pretrained(
    "Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16
)

ds_config = {
    "zero": {
        "stage": 3,
        "overlap_comm": True,
        "contiguous_gradients": True,
        "reduce_bucket_size": 50_000_000,
        "stage3_prefetch_bucket_size": 0.9 * 50_000_000,
        "stage3_param_persistence_threshold": 0,
        "offload_param": {
            "device": "cpu",
            "pin_memory": False
        }
    },
}

ds_engine = deepspeed.init_inference(model, mp_size=1, dtype=torch.float16, replace_with_kernel_inject=True, config=ds_config)

model = ds_engine.module

# model.to(device)
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

prompt = "Question: how many cats are there? Answer:"
inputs = processor(images=image, text=prompt, return_tensors="pt").to(device, torch.float16)

generated_ids = model.generate(**inputs)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
print(generated_text)

However, from the output, I can understand that deepspeed requested 0.75 GB of memory ?
------------------------------------------------------
Free memory : 9.884766 (GigaBytes)
Total memory: 23.699524 (GigaBytes)
Requested memory: 0.750000 (GigaBytes)
Setting maximum total tokens (input + output) to 1024
------------------------------------------------------

Expected behavior
I expect the GPU memory consumption to be the same as the output Requested memory or be less than the GPU memory requested when I load the model normally
ds_report output
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
async_io ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  please install triton==1.0.0 if you want to use sparse attention
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/aime/PycharmProjects/blip_blogpost/venv/lib/python3.8/site-packages/torch']
torch version .................... 1.13.1+cu117
deepspeed install path ........... ['/home/aime/PycharmProjects/blip_blogpost/venv/lib/python3.8/site-packages/deepspeed']
deepspeed info ................... 0.8.1, unknown, unknown
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.2
deepspeed wheel compiled w. ...... torch 1.13, cuda 11.7

Screenshots
If applicable, add screenshots to help explain your problem.
System info (please complete the following information):

OS: Ubuntu 18.04
GPU count and types: 3 NVIDIA GeForce RTX 3090
Interconnects (if applicable) [e.g., two machines connected with 100 Gbps IB]
Python version Python 3.8.10
Any other relevant info about your setup

Launcher context
Are you launching your experiment with the deepspeed launcher, MPI, or something else?
deepspeed --num_gpus 1 blip2_deepspeed_manual_config.py
Docker context
Are you using a specific docker image that you can share?
no docker image used
Additional context
Add any other context about the problem here.

@alaeddine-13, can you clarify if this training or inference? Thanks!

CPU offloading is only applicable to inference.

@lanking520, that is incorrect. CPU offloading is available for both training and inference.

@tjruwase I was trying only inference. This is the code snippet I was using:
from PIL import Image
import requests
from transformers import Blip2Processor, Blip2ForConditionalGeneration, Blip2Config
import torch
import deepspeed

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
model = Blip2ForConditionalGeneration.from_pretrained(
    "Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16
)

ds_config = {
    "zero": {
        "stage": 3,
        "overlap_comm": True,
        "contiguous_gradients": True,
        "reduce_bucket_size": 50_000_000,
        "stage3_prefetch_bucket_size": 0.9 * 50_000_000,
        "stage3_param_persistence_threshold": 0,
        "offload_param": {
            "device": "cpu",
            "pin_memory": False
        }
    },
}

ds_engine = deepspeed.init_inference(model, mp_size=1, dtype=torch.float16, replace_with_kernel_inject=True, config=ds_config)

model = ds_engine.module

# model.to(device)
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

prompt = "Question: how many cats are there? Answer:"
inputs = processor(images=image, text=prompt, return_tensors="pt").to(device, torch.float16)

generated_ids = model.generate(**inputs)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
print(generated_text)

ds_engine = deepspeed.init_inference(model, mp_size=1, dtype=torch.float16, replace_with_kernel_inject=True, config=ds_config)

@alaeddine-13, the above is not the correct API for enabling cpu-offload for inference. This feature is called zero-inference. Here are links of bloom 176b example and correct API.

Have you solved the problem what you mentioned in this issue?

@YongLD, yes, the answer was given https://github.com/microsoft/DeepSpeed/issues/2913#issuecomment-1455920530.
I will close the issue now.


Resolver fails for git dependency

[x] I am on the latest Poetry version.

[x] I have searched the issues of this repo and believe that this is not a duplicate.

[x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: macOS 10.14.3

Poetry version: 0.12.11

pyproject.toml
[tool.poetry]
name = "test-poetry"
version = "0.1.0"
description = ""

[tool.poetry.dependencies]
python = "^3.6"
nornir = { git = "https://github.com/nornir-automation/nornir", branch = "develop" }

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Issue
poetry install fails:
-> % poetry install -vvv
Using virtualenv: /Users/dmfigol/work/tmp/test-poetry/.venv
Updating dependencies
Resolving dependencies...
   1: fact: test-poetry is 0.1.0
   1: derived: test-poetry
   1: fact: test-poetry depends on nornir (*)
   1: selecting test-poetry (0.1.0)
   1: derived: nornir (*)
   1: fact: nornir (2.0.0) requires Python >= 3.6, < 3.8
   1: derived: not nornir (2.0.0)
   1: fact: no versions of nornir match !=2.0.0
   1: conflict: no versions of nornir match !=2.0.0
   1: ! nornir (!=2.0.0) is partially satisfied by not nornir (2.0.0)
   1: ! which is caused by "nornir (2.0.0) requires Python >= 3.6, < 3.8"
   1: ! thus: nornir is forbidden
   1: ! nornir (*) is satisfied by nornir (*)
   1: ! which is caused by "test-poetry depends on nornir (*)"
   1: ! thus: version solving failed
   1: Version solving took 2.356 seconds.
   1: Tried 1 solutions.

It kind of looks like it could be related to #991 but it also looks like it could be due to the version of python not being 3.6 or higher or the lock file not matching the pyproject.toml. I would be interested to see if the output was the same for poetry install -vvv after running poetry lock -vvv since I've noticed that sometimes running poetry lock explicitly, seems to result in more 'correct' behaviour, but i'm not sure why.


test, /sbin not necessarily in the path os casual users.

Thanks, this is probably always valid on Linux...

I had to fix this. not all sysctl are in /sbin. e.g. on systems with busybox ...

The second call passes ?

ah overlooked the second one, thanks


Visualizing sky coverage

Lauren MacArthur pointed us to some nice resources for visualizing sky coverage using the skymaps module.
https://github.com/lsst/skymap
In particular, it is suggested that we look at the examples in
https://github.com/lsst/skymap/tree/master/examples
Another useful resource might be a notebook written by Jim Chiang
https://github.com/LSSTDESC/DC2-analysis/blob/master/tutorials/dm_butler_skymap.ipynb
Additional examples at this DM JIRA ticket:
https://jira.lsstcorp.org/browse/DM-6903
At the coadd level, one can find whether a given (RA, Dec) coordinate is contained within a tract or patch using skymap objects. Apparently, at the moment, it is still not easy to find all the individual CCD images that overlap a given coordinate, but it is recognized that this is needed functionality.
Example visualization:

@rmorgan10 I think we want our stackclub.Taster to be able to make plots like this. I started trying to import Jim Chiang's notebook to the Exploring a Repo tutorial, but I couldn't get the remote import to work. So, I think we need to re-implement the plot, with attribution. I suggest we edit the final section of the Exploring notebook to make a nice plot, and then when that is working, copy the code into the Taster class so that Taster.report() makes a plot as well as a table. Then, our Data Inventory notebook will automatically become better. What do you think? Want to give this a go?

@drphilmarshall Definitely! I actually started on this if you want to checkout my branch, but I'm still trying to get the functions from Jim's notebook to work in this context. I plan on having it up and running by Wednesday though!

Excellent! Can you ping me on Slack as soon as you have submitted a PR to
the project/datainventory/drphilmarshall  branch, please? Looking fwd to
reviewing this one :-)

Will do!

This is still a relevant issue, but I think that the taster is  unsupported and deprecated. We do have a version of dm_butler_skymap.ipynb in the Stack Club repo, but it uses lsst_sims which is not installed on the RSP images and hence does not work (and has never worked?) on the RSP. It'd be great if anyone on this thread wanted to tackle updating this to plot sky coverage using the tools present in the stack.

In case it's helpful to note: lsst_sims will be undergoing some changes,
and hopefully some of the functions you've been using will be easier to
access in its future (conda-installable) version.
https://jira.lsstcorp.org/browse/RFC-773
On Fri, Apr 23, 2021 at 9:25 AM Alex Drlica-Wagner @.***>
wrote:

This is still a relevant issue, but I think that the taster is
unsupported and deprecated. We do have a version of dm_butler_skymap.ipynb
in the Stack Club repo, but it uses lsst_sims which is not installed on
the RSP images and hence does not work (and has never worked?) on the RSP.
It'd be great if anyone on this thread wanted to tackle updating this to
plot sky coverage using the tools present in the stack.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/LSSTScienceCollaborations/StackClub/issues/151#issuecomment-825771409,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOFBXYNEYVMSKMQVDWC7T3TKGNONANCNFSM4F7S7A2A
.

--
Dr R. Lynne Jones
Research Scientist with Rubin Observatory
she/hers

I think DC2-production has a version without lsst_sims. It may not do exactly the same thing though. Else I am pretty sure that what I used for DC2 processing does not use lsst_sims


Gift Entry: Lookup to parent object not populating if importing a child which already exists

Issue
When importing a record that is a child relationship to Opportunity, and that record already exists, we expect the lookup to the Donation/Opportunity to get populated with the ID of the recently created Opportunity. Currently this does not happen.
Steps to Reproduce

Create a scratch org using the qa_org cci flow. This creates some extra custom objects/mappings needed to reproduce.
Edit the default Gift Entry template, ensure that the "CustomObject1Imported" is on the layout
Create a CustomObject1 record using this code in the dev console's Anonymous Apex function
a. insert new npsp__CustomObject1__c(Name='Test Custom 1');
Start a new gift in Gift Entry using the updated default template
Set the CustomObject1Imported field to the recently created CustomObject1 record.
Complete and save the gift, and process it if created in batch mode.
Open the CustomObject1 record you created earlier in the standard UI, or do a SOQL query in the dev console.
The npsp__Opportunity__c field will be empty, we expect this field to have the ID of the recently created Opportunity/Donation

Work Item: W-7954193

This issue is no longer being considered by the product team.  (W-7954193)


fix(qsync): Some improvements to the RD-compatible notification panel

Checklist

[ ] I have filed an issue about this change and discussed potential changes with the maintainers.
[ ] I have received the approval from the maintainers to make this change.
[ ] This is not a stylistic, refactoring, or cleanup change.

Please note that the maintainers will not be reviewing this change until all checkboxes are ticked. See
the Contributions section in the README for more
details.
Discussion thread for this change
Issue number: <please reference the issue number or url here>
Description of this change

https://github.com/user-attachments/assets/b6420525-eba2-4098-81ef-a6db49fb68a8


CFFL

CFFL

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_208dq2m295/clone/recipe/meta.yaml:16: pip_install_args: pip install should be run with --no-deps.
ERROR: /tmp/abs_208dq2m295/clone/recipe/meta.yaml:21: missing_wheel: For pypi packages, wheel should be present in the host section
WARNING: /tmp/abs_208dq2m295/clone/recipe/meta.yaml:40: missing_description: The recipe is missing a description
ERROR: /tmp/abs_208dq2m295/clone/recipe/meta.yaml:40: missing_license_family: The recipe is missing the about/license_family key.
ERROR: /tmp/abs_208dq2m295/clone/recipe/meta.yaml:40: missing_documentation: The recipe is missing a doc_url or doc_source_url
Errors were found


How to Change the resolution of the output picture?

Thank you very much for your code.
Could you tell me how to change the resolution of the output picture? Thanks

Hello, the size of the search area of the current frame in single target tracking is usually fixed. If you change the size of the search area, you generally need to retrain the model.

During the inference phase, the INSTANCE SIZE option under TRACK in the "experiments/siamcar r50/config.yaml" file is used to resize the search area. The C.TRACK.SCORE SIZE option in the "pysot/core/config.py" file corresponds to the final output size of the inference phase model. If the size of the search area is modified during the inference phase, these two parameters must generally be updated.

Thanks a lot, I have already retraining that. By the way, cou you tell me how to change the size of the "Heat map"? Thanks

By modifying the size of the search area and the output score map in the inference phase, the size of the Heatmap can be modified.


usm: process monitor: reuse consumers.NewProcessConsumer

What does this PR do?
Replace USM dedicated event-monitor consumer implementation with a shared utility called consumers.NewProcessConsumer.
Motivation
Removing redundant code.
Describe how you validated your changes

Possible Drawbacks / Trade-offs
Additional Notes

/merge


Report is not generating if test case fails

Reporting goal is not getting executed if test phase fails.
To reproduce-
fail any test case in test phase. this will terminate all maven process and report generation phase(package) will never get executed.
Report should be generate no mater what is the status of test case(pass/fail).

Hello,
Have you tried changing phase settings in POM file?
I have put it to be testand it generates report. (When i run it with mvn clean test)

Hi,
Just tested it with test phase in plugin, for me it didnt work out. no reports generated on failures.
Only one way to avoid is ignore test failures in maven itself but I want to fail build in case of test failures but still want to have reports generated.

Please check this:
https://github.com/trivago/cluecumber-report-plugin/issues/49

Closed due to no reaction.