scipy.stats.rv_continuous.fit throws empty RuntimeError when input has NaNs

When using the scipy.stats.rv_continuous.fit function, a RuntimeError (without a message) is thrown if the array to which the distribution is being fit contains NaNs. It would be nice to have a warning (or error) for NaNs.
Reproducing code example:

import numpy as np
import scipy.stats as stats

x = np.array([1.6483, 1.8223, 2.7169, 2.4667, 1.1791, 3.5433, 2.6561, 1.6631,
       2.412 , 6.3796, 1.7192, 1.438 , 4.5905, 2.832 , 1.86  , 2.1236,
       2.5366, 2.442 , 1.7216, 1.6853, 2.4713, 1.7739, 3.3199, 3.711 ,
       1.821 , 1.6071, 3.9321, 3.4891, 3.0009, 2.0428, 2.6311, 2.218 ,
       2.3838, 2.6662, 2.3853, 2.3079, 3.4989, 4.8993, 8.8305, 3.6952,
       1.9436, 5.7345, 3.1823, 2.8493, 2.2327, 3.2036, 2.3663, 3.0886,
       2.4526, 2.0095, 2.6788, 4.3908, 2.6042, 1.1514, 3.6923, 2.5201,
       2.8638, 1.7267, 6.2628, 2.3748, 2.3494, 4.6349, 2.8303, 2.1384,
       5.0081, 2.3307, 3.543 , 1.8633, 2.3803, 2.0992, 1.8173, 1.6052,
       2.4661, 3.3964, 2.4746, 2.1748, 1.673 , 3.1572, 3.7291, 2.2546,
       1.5754, 2.2669, 1.6383, 1.7696, 1.9079, 1.0955, 1.5637, 2.66  ,
       1.746 , 1.5319, 2.0958,    np.nan])
K = stats.exponnorm.fit(x)

Error message:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-86-fb3d43a0acaa> in <module>
     13        1.5754, 2.2669, 1.6383, 1.7696, 1.9079, 1.0955, 1.5637, 2.66  ,
     14        1.746 , 1.5319, 2.0958,    np.nan])
---> 15 K = stats.exponnorm.fit(x)

/anaconda3/lib/python3.7/site-packages/scipy/stats/_distn_infrastructure.py in fit(self, data, *args, **kwds)
   2195                                          'scale' in kwds):
   2196             # get distribution specific starting locations
-> 2197             start = self._fitstart(data)
   2198             args += start[Narg:-2]
   2199         loc = kwds.pop('loc', start[-2])

/anaconda3/lib/python3.7/site-packages/scipy/stats/_distn_infrastructure.py in _fitstart(self, data, args)
   2034         if args is None:
   2035             args = (1.0,)*self.numargs
-> 2036         loc, scale = self._fit_loc_scale_support(data, *args)
   2037         return args + (loc, scale)
   2038

/anaconda3/lib/python3.7/site-packages/scipy/stats/_distn_infrastructure.py in _fit_loc_scale_support(self, data, *args)
   2287             return (data_b - b) + margin, 1
   2288         else:
-> 2289             raise RuntimeError
   2290
   2291     def fit_loc_scale(self, data, *args):

RuntimeError:

Scipy/Numpy/Python version information:
1.2.1 1.15.4 sys.version_info(major=3, minor=7, micro=1, releaselevel='final', serial=0)

Thanks for reporting this bug. Looks like the generic fit method of rv_continuous ends up at this runtime error if the distribution has support on the real line (a=-inf and b=inf).
Another issue:
x = np.array([1e6, 1e6, 1e6, np.nan])
stats.exponnorm.fit_loc_scale(x, 1) # (0, 1), does not make sense

Reason are the following lines in stats/_distn_infrastructure.py in fit_loc_scale:
if not np.isfinite(Lhat):
    Lhat = 0
if not (np.isfinite(Shat) and (0 < Shat)):
    Shat = 1

I can try to fix this if no one else is working on it already.

Looks like the generic fit method of rv_continuous ends up at this runtime error if the distribution has support on the real line (a=-inf and b=inf).

Not exactly, for example:
import numpy as np
import scipy.stats as stats
x = np.array([1.6483, 1.8223, 2.7169, 2.4667, np.nan])
K = stats.norm.fit(x)
>>> K
(nan, nan)

I'd guess that is b/c scipy.stats.norm provides custom implementation for the fit and fit_start
Which is one output someone might expect. Alternative behavior that might be reasonable would be to ignore the np.nan values and emit a warning.
Unfortunately the behavior really varies across distributions, e.g. (using same x input as above)
>>> stats.betaprime.fit(x)
/Users/rlucas/scipy-dev/scipy/scipy/stats/_continuous_distns.py:667: RuntimeWarning: divide by zero encountered in true_divide
  a/(b-1.0),
/Users/rlucas/scipy-dev/scipy/scipy/stats/_continuous_distns.py:671: RuntimeWarning: divide by zero encountered in true_divide
  a*(a+1.0)/((b-2.0)*(b-1.0)),
/Users/rlucas/scipy-dev/scipy/scipy/stats/_distn_infrastructure.py:1061: RuntimeWarning: invalid value encountered in subtract
  mu2 = mu2p - mu * mu
/Users/rlucas/scipy-dev/scipy/scipy/stats/_distn_infrastructure.py:903: RuntimeWarning: invalid value encountered in greater
  return (a < x) & (x < b)
/Users/rlucas/scipy-dev/scipy/scipy/stats/_distn_infrastructure.py:903: RuntimeWarning: invalid value encountered in less
  return (a < x) & (x < b)
(1.2093829159613558, 1.3349245819681377, nan, 2.38258322474805e-28)

and for scipy.stats.expon on the first time
>>> K = stats.expon.fit(x)
/Users/rlucas/anaconda3/envs/scipy-dev/lib/python3.6/site-packages/numpy/core/_methods.py:32: RuntimeWarning: invalid value encountered in reduce
  return umr_minimum(a, axis, None, out, keepdims, initial)

amd afterwards no error but output is a tuple of np.nan
>>> K = stats.expon.fit(x)
>>> K
(nan, nan)

I'd propose the behavior of emit a runtime warning and return a tuple of np.nan as a solution.
@chrisb83 @WarrenWeckesser @pvanmulbregt thoughts and/or recommendations of alternate solution(s)?

I'd agree that if the input array has np.nans, then a warning should be thrown. If the input array has only np.nan, then an error should be thrown.
I don't see a good use case for which returning a tuple of np.nan is useful, but this might be a shortsighted statement.

I don't see a good use case for which returning a tuple of np.nan is useful, but this might be a shortsighted statement.

@kmundnic My thinking is that if you are getting np.nan that by propogating the warning through the code it is more likely to be surfaced. However, as I'm writing this I'm thinking maybe just throwing an error would be better. If the behavior is runtime warning and continue with a fit then to get something other than np.nan you'd need to ignore the np.nan values. However, once you get into that territory the number of non np.nan observations you need will differ across distributions, (e.g. single location parameter vs 2 parameter location-scale family. While the fit function might have that information available the logic can become fairly complicated. I'd prefer a simpler solution if possible which would be to emit np.nan as estimates. I know @WarrenWeckesser worked for a while on warnings in the SciPy libraries would like to get his recommendation before proceeding in any direction on fixing the defect.

I'd propose the behavior of emit a runtime warning and return a tuple of np.nan as a solution

Returning nan is aligned to the default behaviour nan_policy = 'propagate' in other stats functions, raising an error corresponds to 'raise'.
Any nonfinite value such as Inf probably need to be handled in a similar way.

Thanks for the input @chrisb83 It sounds (mostly) clear what to do except not sure if a PR should also address adding the nan_policy functionality or if that should be a separate PR?

On Jul 3, 2019, at 12:43 AM, Christoph Baumgarten<EMAIL_ADDRESS>wrote:
I'd propose the behavior of emit a runtime warning and return a tuple of np.nan as a solution
Returning nan is aligned to the default behaviour nan_policy = 'propagate' in other stats functions, raising an error corresponds to 'raise'.
Any nonfinite value such as Inf probably need to be handled in a similar way.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

If we were to add the nan_policy argument to the fit method, the behavior when the input contains nan would be:

nan_policy == 'raise': Raise an exception.
nan_policy == 'omit': Ignore any nan values in the input.  Do not generate a warning if the input contains nan.  E.g. fit([1, 2, np.nan, 3]) should behave the same as fit([1, 2, 3]).  If all the values in the input are nan, the function should behave as if the input was empty.  That is, fit([np.nan, np.nan, np.nan]) should behave the same as fit([]).  Unfortunately, the current behavior of the distributions in that case is inconsistent.  I just tried norm and beta.  norm generated a warning and returned (nan, nan).  beta raised an exception.  I think the correct behavior is to raise an exception. The exception message should state explicitly that an empty input is not accepted.  (The error currently generated by beta is incidental, and the error message is not very helpful.)
nan_policy == 'propagate': Return nans as the result.  Do not generate a warning.

Note that nan_policy controls how the floating point value nan is handled.  It does not affect how inf is handled.
If we don't add the argument nan_policy, then the two most reasonable options seem to be (1) generate a warning and return nans, or (2) raise an exception.  My inclination is to raise an exception, but that's what I'll say if you ask me today.  Ask me tomorrow, and I might prefer a warning and returning nans, or maybe even nans with no warning.  (The argument in favor of the latter: many functions in NumPy pass through nans without a warning, e.g. np.sin([1, np.nan, 2]), np.mean([1, np.nan, 2]), etc. Should we be consistent with that behavior?)  Given that, I'm more inclined to say we should implement the nan_policy argument (explicit is better than implicit, refuse the temptation to guess, etc.).  The default value should probably be 'propagate'--that seems most consistent with the current behavior.  We can't expect complete consistency with the current behavior, because the currently behavior is wildly inconsistent.  The fit methods were not written to take nan into account, and it shows.
For example,
In [13]: from scipy.stats import norm, beta, exponnorm, gamma, weibull_min

In [14]: x = np.array([0.1, 1, 2, 3, 3, np.nan, 5])

norm.fit(x) returns (nan, nan) with no warning:
In [15]: norm.fit(x)
Out[15]: (nan, nan)

gamma.fit(x) return a mix of nan and 1.0 (which makes no sense):
In [16]: gamma.fit(x)
Out[16]: (nan, nan, 1.0)

beta.fit(x) generates incidental warnings and returns a different mix of nans and 1.0 :
In [17]: beta.fit(x)
/Users/warren/miniconda36b/lib/python3.6/site-packages/scipy/optimize/minpack.py:162: RuntimeWarning: The iteration is not making good progress, as measured by the
  improvement from the last ten iterations.
  warnings.warn(msg, RuntimeWarning)
/Users/warren/miniconda36b/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:874: RuntimeWarning: invalid value encountered in greater_equal
  return (self.a <= x) & (x <= self.b)
/Users/warren/miniconda36b/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:874: RuntimeWarning: invalid value encountered in less_equal
  return (self.a <= x) & (x <= self.b)
Out[17]: (1.0, 1.0, nan, nan)

Similarly, weibull_min.fit(x) generates incidental warnings, and returns yet another mix of nan and non-nan values:
In [18]: weibull_min.fit(x)
/Users/warren/miniconda36b/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:874: RuntimeWarning: invalid value encountered in greater_equal
  return (self.a <= x) & (x <= self.b)
/Users/warren/miniconda36b/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:874: RuntimeWarning: invalid value encountered in less_equal
  return (self.a <= x) & (x <= self.b)
Out[18]: (1.3053444453258112, nan, 3.902134995832234e-30)

And, as shown above, exponnorm.fit(x) raises a RuntimeError, with no error message in the exception:
In [19]: exponnorm.fit(x)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-18-390b024fc8e6> in <module>
----> 1 exponnorm.fit(x)

~/miniconda36b/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py in fit(self, data, *args, **kwds)
   2195                                          'scale' in kwds):
   2196             # get distribution specific starting locations
-> 2197             start = self._fitstart(data)
   2198             args += start[Narg:-2]
   2199         loc = kwds.pop('loc', start[-2])

~/miniconda36b/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py in _fitstart(self, data, args)
   2034         if args is None:
   2035             args = (1.0,)*self.numargs
-> 2036         loc, scale = self._fit_loc_scale_support(data, *args)
   2037         return args + (loc, scale)
   2038

~/miniconda36b/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py in _fit_loc_scale_support(self, data, *args)
   2287             return (data_b - b) + margin, 1
   2288         else:
-> 2289             raise RuntimeError
   2290
   2291     def fit_loc_scale(self, data, *args):

RuntimeError:

nan_policy has a few issues at the moment, see  #9558
I would favor just raising an error or return nan if there is any nonfinite value.The majority seems to favor raising an error, i think that is fine.

I agree that long term we’d like a nan_policy and I could do that but it would be blocked by the other issue and not clear when that would be resolved. So I’ll break this into 2 PRs, one that implements a default ‘propogate’ policy for nan, which could be merged as soon as approved. The I’ll open a 2nd PR which implements a full nan_policy and may be blocked by other side until resolved. Please chime in if any objections. The default policy I’ll likely have a PR out for by end of this coming weekend.
Sincerely,
-Lucas Roberts

On Jul 6, 2019, at 6:47 AM, Christoph Baumgarten<EMAIL_ADDRESS>wrote:
nan_policy has a few issues at the moment, see #9558
I would favor just raising an error or return nan if there is any nonfinite value.The majority seems to favor raising an error, i think that is fine.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@chrisb83 and @WarrenWeckesser I'm looking at some of the other methods in stats.py to get an idea of what to do. I'm having a bit of difficulty identifying a function that has a correctly implemented version of nan_policy='propagate' for example:
>>> sc.moment([np.nan, np.nan, np.nan, 1, 2, 3,], moment=1, nan_policy='propagate')
0.0

which doesn't seem correct. I would expect the result to be nan I can open an issue for the moment function is this seems incorrect. Can either of you point me to a function which has a functioning propagate nan policy in the stats.py file (or somewhere else in scipy)?


ENH: Enable flexible weights initialization from src_checkpoint

Add flags pretrain_encoder pretrainer_pooling pretrain_classifier to init submodules weights from a pretrained checkpoint in src_checkpoint

Excellent work Kenza! Approved with a minor suggestion


Feature suggestion: Add sample names to TMB plot

Hi!
Thank you for providing this fantastic package  🤩
I am currently using maftools 2.10.0. I have been using the tmb() plot. From reading through the manual it doesn't seem yet to be possible to label the resulting data points with the tumor sample barcode. It would be quite useful to us to have when later evaluating results of individual samples and cross checking how they relate to the TMB directly.

You can just use the data frame produced and plot it using ggplot2.


docs: add fzf-lua and nu_plugin_skim to the README

Checklist
check the box if it is not applicable to your changes

[x] I have updated the README with the necessary documentation
[x] I have added unit tests
[x] I have added end-to-end tests
[x] I have linked all related issues or PRs
[x] I have re-generated the completions and manpage using cargo xtask compgen and cargo xtask mangen

Description of the changes
closes #570
closes #418

@idanarye, @ibhagwan could you please take a look and see if you're all right with this ?

  -- optional for icon support
  dependencies = { "nvim-tree/nvim-web-devicons" },

I'm not sure if we need this part, although nice to have there's also mini.icons (part of  the amazing mini.nvim) which is also used by default with LazyVim - perhaps it's better to direct users to the repo's README for more options instead?

I'll merge this tomorrow if it's good with you two

I'll merge this tomorrow if it's good with you two

Looks great, Ty @LoricAndre!

@LoricAndre it looks like these Readme updates got clobbered -- the README on main no longer has nushell plugin informatrion.

Hmmm you're right, I need to check how that happened


Suggestion to add to the Gameplay Effect section

Also something to note in the Gameplay Effect section is that Gameplay Effects have a property called "UIData". UIData is used for data for the UI representation of this effect(for like buffs and debuffs). UI layer can listen(or playercontroller/state) for when Gameplay Effect deactivate or activate, and use the UGameplayEffectUIData object from that effect to display to the HUD/UI. That way you can design the ui text, icons, etc from blueprints.
Now typically, I will have the ASC expose a much simpler delegate which just takes in the  UGameplayEffectUIData object, and Gameplay Tag that is trigger when a Gameplay Effect activate and has UIData, and also a deactivation delegate that passes a Gameplay Tag. These two delegates are really for the UI layer, and to avoid having to pass around a huge Gameplay Effect Object.

I purposefully held back from including the UIData because I haven't had to use it yet. I feel like that's something that I would want to include a working example with. I suppose I could do a simple case where the stun effect from the meteor in the sample project displays a stun icon above the stunned characters' heads.
However, the minion characters are in minimal replication mode so they wouldn't receive the GE on clients anyways. A separate table mapping tags to UIData-like classes rather than using the GE UIData makes more sense to cover mixed and minimal replication mode.
It's something to think about for the future but I don't have time right now to work on this.

to avoid having to pass around a huge Gameplay Effect Object.

Just to clarify, we would only ever pass around a pointer to the GE class. So this shouldn't be a concern.
Thanks for the suggestion.

I can understand including a working example with the UIData since in the plug there is only single subclass which has a text Description property.  I was looking at it for cases where the Gameplay Effect lives and does it business at. They do say that it does not exists for server only builds. Which would be good to understand how they exclude it from server only builds


moving react and react-dom to dev-dep

@deanshub Can you explain the reason for this change?
I don't think it's correct and we've had problems with code like this before. An angular project for example should be able to import and define an AngularLazyComponent without having to explicitly import react.
We do need to update the version though...

@ronenst that's exactly my point, I moved react and react-dom from dependencies to dev-dependencies.
I have an issue that a project of mine that I'm upgrading to react 16, uses a dependency that uses this project, when I'm installing dependencies it installs react 15 because it's a regular dependency and not dev which causes an error of conflicting react versions (https://github.com/facebook/react/issues/10320#issuecomment-318754280)
once moving these dependencies to dev-dependency they won't be installed and the react will come from the parent's dependency which makes more since since this is a library and not a project

Do you have react and react-dom defined as externals? That may solve it but you'll need to inject them via script tags.
What worries me is that this change could potentially break projects which don't install React themselves. Even if it is more correct that could be a problem.
Our team can try upgrading the version though, I think that should also solve your issue?

I have a workaround, just thought it would be better to move it here as-well

Okay, thanks, we'll discuss internally.


Patch template

First pass notes:

stm32h7xx_hal_conf.h and system_stm32h7xx.c are no longer needed and should be deleted.
STM32H750IB_flash.lds is no longer needed, and should be deleted.

if the vcxproj or vcxproj-filters reference it, the paths(s) should be updated to the libdaisy/core version.

The VS/VGDB files (*.vgdbsettings, *.vcxproj*, stm32.* should be moved into a vs/ folder, though this will require editing all of the relevant paths within those files. So I can take care of that if you don't want to go through it all. (If you do, don't name it the directory .vs as that is an auto-generated hidden directory, and ignored by .gitignore
README is out of date and is for PluckEcho example.

As discussed earlier, there could be an argument made for moving the template projects to utils/templates or somewhere that the copy/new project scripts can reach them without users thinking they can easily copy the entire folder structure themselves (since the vscode and vs/vgdb integration requires quite a bit of path mangling).
Since the counter-argument is having easy access to good starter code -- the program (being only 30 lines) could easily be added to the README, or elsewhere within our documentation (even in the getting started guide).
Open to either, but definitely worth discussing since this will currently be added to the Web programmer as a second 'bypass/passthru' example, which we don't necessarily need there.

 Make template cpp files for each platform

Regarding the VS folder, shouldn't it be called vs/ ? I'm not sure why you mention the hidden .vs directory

Regarding the location / web programmer, I agree. Let's move it to utils/. We can document in the readme / documentation where to find the template. Worth noting not every platform has a passthrough example yet so we should remedy that situation as well.

closing because the work is going to continue in #127 -- your branch has been merged there and should now be safe to delete.


Process to backup data from Mongo

Progress?  Remaining LOE?

Need to determine approach to be used, probably 16 hours
On Wed, Jun 6, 2018 at 9:33 PM texasbrok<EMAIL_ADDRESS>wrote:

Progress? Remaining LOE?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.


[C++] Support reading Hadoop-snappy File Format Directly

Describe the enhancement requested
Hadoop-snappy File Format is widely used for BigData and ML frameworks; We use spark or hadoop mr to preprocess data for furture processing such like Data warehouse and DeepLearning training;
Hadoop-snappy File is a compressed file consists of one or more blocks. A block consists of uncompressed length (big endian 4 byte integer) and one or more subblocks.
Arrow could read a file which is fully compressed with Snappy Codec but fails to read Hadoop-snappy File Format;
If arrow can read the Hadoop-snappy File consisted of a serials of chunks, we can use the output from Spark directly and save the HDFS storage cost. All directly reading Haoop-snappy file will greatly extend the Arrow's using scenarios~
Within bytedance.com, almost 40% files are compressed by Snappy and stored in HDFS filesystem; we use arrow to read Hadoop-snappy File and Hadoop-Zstd file which save computing resources greatly~
Component(s)
C++

Would you mind provide something about "hadoop-snappy"? Do you mean https://github.com/kubo/snzip#hadoop-snappy-format ?
By the way, Arrow already has snappy codec support and HDFS datasource support.

Yes, I think we will need a link or more information.  It is not clear to me if this is a new compression codec (some kind of block-based snappy) or file format or some change in the hdfs filesystem implementation.

some change in the hdfs filesystem implementation.

In Hadoop common it's named as BlockDecompressorStream with SnappyCodeC
Here is the link for hadoop-snappy: https://code.google.com/archive/p/hadoop-snappy/

Seems it's just SnappyCodec? Can you just try the:

https://github.com/apache/arrow/blob/main/cpp/src/arrow/util/compression.h
or https://arrow.apache.org/docs/python/generated/pyarrow.compress.html

Seems it's just SnappyCodec? Can you just try the:

https://github.com/apache/arrow/blob/main/cpp/src/arrow/util/compression.h
or https://arrow.apache.org/docs/python/generated/pyarrow.compress.html

This is not block based compress/unpresss support.  If we read a hadoop-snappy files, we need to uncompress blocks one by one, which is not convinenet enought for high level components;
If we read blocks from one Haddoop-Snappy file,  definitely we do need to uncompress one block with the help of "arrow/util/compression.h"; However we still need to wright extra code to provide a StreamingView for a serial of uncompressed snappy blocks for hight level component.

Are you reading CSV files?  If not, then I am not sure that a streaming view will be very helpful.  We generally rely on random access to metadata.  This allows us to implement things like column skipping (projection pushdown).  This is why formats like the arrow format and the parquet format support buffer-compression (instead of whole-file compression).  It allows the metadata to stay uncompressed.
When you talk about blocks are you talking about the snappy framing format?  https://github.com/google/snappy/blob/main/framing_format.txt

When you talk about blocks are you talking about the snappy framing format? https://github.com/google/snappy/blob/main/framing_format.txt

NOT CSV files.  We are reading some kind of TFRecod file which is commpressed into Hadoop-Snappy format such like training_instance_0001.tfrecord.snappy
There are 3 kind of general snappy-compressed file (not the parquet use snappy internally) , and this issue is about "hadoop-snappy file"
1.snappy whole file
2.snappy framing  (google, rarely used.)
3.hadoop-snappy file  (whichi is whide useed in big-data ecosystem)
2. is different from 3. we can treat 3. as a kind of general blocked based compressed format( the codec for the blocks could able replaced by ZSTD, gzip and so on;)

@shanjixi I'm curious, what does Arrow bring to the table if the file format (TFR) is not supported by Arrow?

(I'm horrified that there is even a Hadoop-ZSTD)

supported

TFRecord is just like a file contains multi-rows；
In practice，we use a converter to transform TFRecord file (compressed) into arrow::Table Type. Then we apply Arrow filter/take/list_flatten functions before we send the Data to Tensorflow worker;
Arrow is used to read data from HDFS, make further processing and avoid memroy copy cost between these two steps;
What's more, within our company both  Hadoop-ZSTD and Hadoop-SNAP are supported to be read with Arrow directly( we call them as zstd_decompress_inputstream and snappy_decompress_inputstream.

Ok, see https://github.com/apache/arrow/blob/main/cpp/src/arrow/io/compressed.h
You could pretty easily add a HadoopCompressedInputStream there.


Java HELICS API Tests

This pull request contains a file that calls the Java HELICS API functions and asserts the results. This file contains tests that will close issue #306. This file can also be used to determine if issue #348 has been fixed.

I've addressed all the warnings in Codacy except the first one. It complains that the main function is too long. I just wrote all the tests inside of the main function as it was the most straight forward approach for testing the api. I think we can just ignore it.

Is this test running on Travis?  @nightlark  if not can you add it to the tests that run regularly?

the review is only showing 1 file changed is that correct?

@afisher1  would it be possible to go through and fix the warnings in the other Java test files?

The tests aren't running - it looks like the CMakeLists.txt file doesn't have any targets added that use the JavaHelicsApiTests.java source file. There's also another source file named ApplicationApiTest.java which is a pretty similar name -- but that one seems to just test a single value federate publishing a value and advancing time; thoughts on renaming that ApplicationApiTest to something else and calling JavaHelicsApiTests ApplicationApiTest instead?
Also, something more general with the Java tests, I noticed that TestValueFederate.java has a large number of methods that don't get called from its main method -- are the tests in there a work in progress (I don't see any open issues related to them, closest is #315), or is there something special needed to run them (I noticed that the Test*.java files all import java.lang.reflect.Method)?

@afisher1  can you work with @nightlark  to make sure all the java tests are running in Travis/appveyor and  see if we can clean up some of the files if they are duplicative or redundent?

@nightlark  JavaHelicsApiTests.java is the new test script I built that actually asserts functionality of the java bindings for helics. All the other ones were incomplete works from Jeff basically. They called very few functions of the api and didn't validate that they did what they were supposed to. We can basically get rid of them and just use my file. How do we get my file to automatically be run on travis.

The CMakeLists.txt file in the java tests folder needs a new test added, and the name of the test gets added to the line at the bottom of the cmake file to give it the Java label.

The gcc 4.9.3 build had a set of failed assertions, but on rerunning the build it passed. On the PR build, clang has a similar set of assertions that failed. I'm going to restart the build -- below is a copy of the log for the failed run.
10: Test command: /usr/lib/jvm/java-8-oracle/bin/java "-Djava.library.path=/home/travis/build/GMLC-TDC/HELICS-src/build/swig/java" "-classpath" "/home/travis/build/GMLC-TDC/HELICS-src/build/swig/java/buildjar/helics-1.2.0.jar:/home/travis/build/GMLC-TDC/HELICS-src/build/tests/java_helics" "JavaHelicsApiTests"
10: Test timeout computed to be: 1500
10: Picked up _JAVA_OPTIONS: -Xmx2048m -Xms512m
10: java.lang.AssertionError: ep2MsgCount != 3
10: 	at JavaHelicsApiTests.helicsAssert(JavaHelicsApiTests.java:11)
10: 	at JavaHelicsApiTests.main(JavaHelicsApiTests.java:382)
10: java.lang.AssertionError: fed1MsgCount != 2
10: 	at JavaHelicsApiTests.helicsAssert(JavaHelicsApiTests.java:11)
10: 	at JavaHelicsApiTests.main(JavaHelicsApiTests.java:419)
10: java.lang.AssertionError: sub1Updated != 1
10: 	at JavaHelicsApiTests.helicsAssert(JavaHelicsApiTests.java:11)
10: 	at JavaHelicsApiTests.main(JavaHelicsApiTests.java:462)
10: java.lang.AssertionError: sub2UpdateTime != 1.0e9
10: 	at JavaHelicsApiTests.helicsAssert(JavaHelicsApiTests.java:11)
10: 	at JavaHelicsApiTests.main(JavaHelicsApiTests.java:466)
10: java.lang.AssertionError: sub5Value[0] != 1
10: 	at JavaHelicsApiTests.helicsAssert(JavaHelicsApiTests.java:11)
10: 	at JavaHelicsApiTests.main(JavaHelicsApiTests.java:471)
10: java.lang.AssertionError: sub2Real[0] != 5.6
10: 	at JavaHelicsApiTests.helicsAssert(JavaHelicsApiTests.java:11)
10: 	at JavaHelicsApiTests.main(JavaHelicsApiTests.java:477)
10: java.lang.AssertionError: sub2Imag[0] != -0.67
10: 	at JavaHelicsApiTests.helicsAssert(JavaHelicsApiTests.java:11)
10: 	at JavaHelicsApiTests.main(JavaHelicsApiTests.java:480)
10: java.lang.AssertionError: sub1Value[0] != 457.234
10: 	at JavaHelicsApiTests.helicsAssert(JavaHelicsApiTests.java:11)
10: 	at JavaHelicsApiTests.main(JavaHelicsApiTests.java:485)
10: java.lang.AssertionError: sub4Value[0] != 1
10: 	at JavaHelicsApiTests.helicsAssert(JavaHelicsApiTests.java:11)
10: 	at JavaHelicsApiTests.main(JavaHelicsApiTests.java:490)
10: java.lang.AssertionError: !sub7PointString.contains("Blah Blah")
10: 	at JavaHelicsApiTests.helicsAssert(JavaHelicsApiTests.java:11)
10: 	at JavaHelicsApiTests.main(JavaHelicsApiTests.java:498)
10: java.lang.AssertionError: sub7PointLength[0] != 9
10: 	at JavaHelicsApiTests.helicsAssert(JavaHelicsApiTests.java:11)
10: 	at JavaHelicsApiTests.main(JavaHelicsApiTests.java:501)
10: java.lang.AssertionError: !sub3ValueString.contains("Mayhem")
10: 	at JavaHelicsApiTests.helicsAssert(JavaHelicsApiTests.java:11)
10: 	at JavaHelicsApiTests.main(JavaHelicsApiTests.java:511)
10: java.lang.AssertionError: sub3Length[0] != 6
10: 	at JavaHelicsApiTests.helicsAssert(JavaHelicsApiTests.java:11)
10: 	at JavaHelicsApiTests.main(JavaHelicsApiTests.java:514)
10: java.lang.AssertionError: sub3ValueSize != 6
10: 	at JavaHelicsApiTests.helicsAssert(JavaHelicsApiTests.java:11)
10: 	at JavaHelicsApiTests.main(JavaHelicsApiTests.java:518)
10: 80 of 94 tests passed.
10: 85% successfull.
10: 14 tests failed.
2/5 Test #10: Java_Helics_Api_tests ............***Failed    0.63 sec

The PR build with clang looks like it is failing consistently.


sudo not found: cypress/browsers

I use the docker image cypress/browsers:node14.15.0-chrome86-ff82 for testing e2e and component tests in Github Actions.
Repro

Add the following job to a new GitHub Action
chrome:
name: Chrome
timeout-minutes: 15
runs-on: ubuntu-latest
container: cypress/browsers:node14.15.0-chrome86-ff82
needs: build
steps:

name: Checkout 🛎
uses: actions/checkout@v2

name: Increase File Watchers 🔭
run: echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
other steps ...

Expected Result

The command works

Actual Result
Run echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
/__w/_temp/84726bfa-c8c8-4827-84e7-1288541db1df.sh: 1: /__w/_temp/84726bfa-c8c8-4827-84e7-1288541db1df.sh: sudo: not found
Error: Process completed with exit code 127.

Background
When using yarn cypress run-ct to run the component tests I got many warnings: ENOSPC: System limit for number of file watchers reached that's why I want to increase the limit with the following command.
With a normal ubuntu job this works fine.

This is exactly what is happening to me, but I don´t think this is an issue with the docker image. If I remember correctly, you cannot change the number of watchers directly on Docker, you have to change it on the host system (in this case Ubuntu). The problem is, I don´t know how (or if it is even possible with Github actions) to run the watchers command before starting the docker image. If anybody knows how to do this it would be very helpful.

@panmona
I'm going to close this old issue since it refers to versions which are no longer supported. You can respond here if you still need help.


Fix LineLength offenses in files thru app/classes

Fix RuboCop LineLength offenses in:

Rakefile
app/classes/...

Coverage increased (+0.02%) to 91.511% when pulling 8914a0b1440188d0c572a5a187d8961348a4f809 on JoeCohen:LineLength-classes into 824dd92e8320b972f3e20b0d0d86ec744bfe5045 on MushroomObserver:master.

I'll deal with the CodeClimate violations later today or tomorrow.


hardcoded etcd snapshot s3 upload timeout of 30 seconds

Environmental Info:
K3s Version:

k3s version v1.21.5+k3s2 (724ef700)
go version go1.16.8
Node(s) CPU architecture, OS, and Version:

Linux pi1 5.10.63-v8+ #1457 SMP PREEMPT Tue Sep 28 11:27:02 BST 2021 aarch64 GNU/Linux
Cluster Configuration:

3 servers, 1 agent
Describe the bug:

Been told there is a hard coded timeout of 30seconds on uploading etcd snapshots to s3, On a slower internet connection it may take longer than 30 seconds to push the snapshot to an s3 storage.
Steps To Reproduce:

Installed K3s:
enabled s3 snapshots
no s3 snapshots

Expected behavior:

Should expect snapshots on s3 bucket
Actual behavior:

snapshots only stored locally
Additional context / logs:

Oct 14 00:00:38 pi3 k3s[1928]: time="2021-10-14T00:00:38.258458746+11:00" level=error msg="Error received in attempt to upload snapshot to S3: Put \"https://BUCKETNAME.s3.dualstack.ap-southeast-2.amazonaws.com/etcd-snapshot-pi3-1634130000\": context deadline exceeded"
Oct 14 00:00:38 pi3 k3s[1928]: time="2021-10-14T00:00:38.258554116+11:00" level=info msg="S3 upload complete for etcd-snapshot-pi3-1634130000"
Oct 14 00:00:38 pi3 k3s[1928]: time="2021-10-14T00:00:38.706423059+11:00" level=info msg="Saving current etcd snapshot set to k3s-etcd-snapshots ConfigMap"

config.yaml
etcd-s3: true
etcd-s3-access-key: KEY
etcd-s3-bucket: BUCKETNAME
etcd-s3-region: ap-southeast-2
etcd-s3-secret-key: SECRET

Backporting

[ ] Needs backporting to older releases

Verified in k3s on Master with commit id ef263bd2b0ecb29c3404cd4b80fcfcf8b4301912
Steps

Create a single node cluster (1 server 1 agent, Ubuntu 20.04)
Install k3s on server and agent nodes

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--write-kubeconfig-mode 644 --token secret --cluster-init" INSTALL_K3S_COMMIT=ef263bd2b0ecb29c3404cd4b80fcfcf8b4301912 sh -

Verified the cluster is created and running
Take snapshot of the cluster and store in s3

k3s etcd-snapshot \
  --s3 \
  --s3-bucket=<BUCKET_NAME> \
  --s3-access-key=<ACCESS_KEY> \
  --s3-secret-key=<SECRET_KEY> \
  --s3-region=<REGION> \
  --s3-timeout=50s

Validate the snapshot is created locally and also stored in s3 bucket respectively

INFO[0000] Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/<ID>
{"level":"info","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/<ID>.part"}
{"level":"info","logger":"client","msg":"opened snapshot stream; downloading"}
{"level":"info","msg":"fetching snapshot","endpoint":"<REDACTED>"}
{"level":"info","logger":"client","msg":"completed snapshot read; closing"}
{"level":"info","msg":"fetched snapshot","endpoint":"<REDACTED>","size":"2.0 MB","took":"now"}
{"level":"info","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/<ID>"}
INFO[0000] Saving etcd snapshot <ID> to S3
INFO[0000] Checking if S3 bucket <BUCKET_NAME> exists
INFO[0000] S3 bucket <BUCKET_NAME> exists
INFO[0000] S3 upload complete for <ID>
INFO[0000] Saving current etcd snapshot set to k3s-etcd-snapshots ConfigMap


Support for custom text when time ago is less than 60 seconds

Right now, react-timeago updates by the second when the time ago value is less than 60 seconds.
It's fine for some use cases but can be distracting to the user in some cases.
I propose that we should be able to set a uniform value when the value is less than 60 seconds like less than a minute or a few seconds ago.

It seems that my use case has been solved in issue #68.


Jupyter install fails on Windows Server with "Microsoft Visual C++ 14.0 or greater is required"

Azure Data Studio Version: Latest
OS Version: Windows Server 2019 and 2022 Datacenter (Azure and GCP VMs tried)

Steps to Reproduce:

Create VM in Azure or GCP with Windows Server 2019 or 2022

Start ADS

Create new Notebook

Select Python Kernel

Hit "Install" (when it prompts to install Jupyter support)

Fails with:
running build_ext
building 'psutil._psutil_windows' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

Installing C++ Build Tools at that link is 5GB! (and issue with filling up boot drive when using ephemeral VMs)

Does this issue occur when all extensions are disabled?: Yes/No
Yes

@corivera can you take a look please?

@stuartpa are you using our provided python package (New Install) or an existing python instance you installed separately?

The former (this is a completely vanilla Windows Server Datacenter VM (tried both Azure and GCP), with no other software involved.

Do you see the same issue when using a separately installed python instance?

I was just trying to install ADS on a vanilla Windows Server VM and select the Python kernel.  I haven't tried anything more than this.  I have tried multiple cloud providers and multiple versions of Windows Server (2019 and 2022)

Seems like this is just a jupyter dependency (required by jupyter's psutil dependent package), since I get the same error when trying to install it using a standalone python instance on Windows Server.


There is an error with cli: Error: No such option: --name in datahub ingest deploy

Describe the bug
There is an error with cli: Error: No such option: --name in datahub ingest deploy
command datahub ingest deploy --name "test" --schedule "5 * * * *" --time-zone "Europe/London" -c recipe.yaml
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
Update datahub doc about cli or fix an issue with cli.
Screenshots

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Additional context

@pedro93 yu.gavrilov@S7-13-038829:~$ datahub --version acryl-datahub, version <IP_ADDRESS>

Please use <IP_ADDRESS>

@pedro93 Thanks for helping. actually it seams to be working. but got other error twith reciepies. will try to solve it ..


It doesn't save when using the toUni method

I'm trying to be familiar with this API.
I just want to save a record using the reactive style.
there are some boilerplate codes:
Multi<Victim> victim = Multi.createFrom().resource(driver::rxSession, session -> session.writeTransaction(writeVictimTransactionWorker)).withFinalizer(ModelTest::closeSession).map(..)
and finally we will get a Multi. when I subscribe it, everything is fine.  But I just want a save result,so I use 'toUni ' to change it to a Uni.
when I subscribe the gotten Uni, the record doesn't really saved,although it emits the result

I have found the reason:
When using .toUni ,.select().first(1) methods,after emit the element, there will be a cancel calling.
I can use the .collect().asList().map() method, but it's a little tricky.
Is there a better way to write saving code?
By the way,the document is broken for now

Hi @zhangpp520
All the reactive frameworks I know of (Mutiny, Project Reactor) emit a cancellation event when you use first() or similar. It's best you save all of the things coming back or directly use a Uni.

By the way,the document is broken for now

i don't understand this. What exactly is broken?


Show album art for local files

Please explain the problem/why you would want to see a change and any workarounds you might have in place.

This might not be possible, so some investigation is required. It was reported in the Spotify Notifications repo issues but is still relevant here, I believe.

If this is not possible from the Spotify API, then it's hacky to try to get this to work with Now Playing.
I think this should simply be closed and left to Spotify themselves to figure out.


BuiltInPlugins wont work anymore

I had a plugin that allowed me to execute script at a higher level that used the BuiltInPlugins but this week I found that it kept getting not loaded for security mesures but before it loaded fine and when I was looking at the fflags "DoNotLoadUnverifiedBuiltInPlugins" and "VerifyBuiltInPlugins" there were not there anymore these 2 fflags allowed to disable the security mesures soo yea Im stuck with that rn

Roblox removed the flags unfortunately.
I'll be looking into whether it's possible to patch the exe around this, but otherwise I can't address it.

Get in touch on Twitter and I can show you a workaround.

My Twitter tag is @mrsbobette06
Le ven. 12 mars 2021, à 22 h 07, Max @.***> a écrit :

Get in touch on Twitter and I can show you a workaround.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/CloneTrooper1019/Roblox-Studio-Mod-Manager/issues/86#issuecomment-797856060,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJXTEL2FBBGW3M62IIHDCX3TDLCHFANCNFSM4ZDG4IVQ
.


Expose openapi / swagger

Would be nice to have an openapi / swagger UI to the apis

@mdnmdn It seems like a useful feature, I insert it as a task

We could use: https://github.com/flasgger/flasgger
From a rapid overview has 3 different approaches:

in python function comment ( https://github.com/flasgger/flasgger#using-docstrings-as-specification )
with external yaml ( https://github.com/flasgger/flasgger#using-external-yaml-files )
using python code vai dictionary or specific objects

I don't like too much the first approach because the final code results bloated, the others are better for me, maybe using different files for the specs.

@mdnmdn I would use the yaml approach


[QuantNoise] QuantNoise convertion for custom / own model outside of fairseq?

Will pytorch release code to convert custom model using quant-noise? especially on digital speech processing, for example, ForwardTacotron.

Thanks for the interest. We are investigating the possibility. In the meantime, we have an "integration with your own code" section of our ReadMe: https://github.com/pytorch/fairseq/tree/master/examples/quant_noise


Empty list of changes

Hi!
Trying to use the plugin at my work environment. Private Gerrit repository (2.12.5 version). I think that it is a configuration problem, but I don't know how to solve it.
The repository has a lot of subdirectories. Some of them are Python projects that I'm managing without further problems using PyCharm.
I have configured Gerrit plugin and the credentials seem to be OK, because the "Test" button on the settings interface says that everything is OK.
When I try to fetch the changes, anything is happening, only an empty list.
What can I do to see what is happening? Any log or somewhere to look at?
Best regards

@JoseLSegura Have your tried all of these hints? https://github.com/uwolfer/gerrit-intellij-plugin#list-of-changes-is-empty

Similar for me. I checked all hints points Gerrit plugin and Git setup, and remotes.
I can fetch and push my commits to gerrit successfully, But don`t see changes in gerrit window.
Also when I try to select branch for filtering results I see my project path and branches of my project, so I think that I have right setup. Will be appreciate for help.

I have the same problem. After some experiments I've found that changes are fetched if remote url in git config file is using ssh protocol. However if remote url has https protocol it can't fetch those.
In both cases hostname is the same. But https protocol is defined as https://<fqdn>/a/<project path>, when ssh is ssh://<fqdn>:<port>/<project path>

This is still an issue when cloning projects using https instead of ssh, is this something that you are looking into?

Edit: if someone is coming here for this issue, using https in gerrit changes the base path, as stated by ZsergUA above. The server path used to login is https://, but for cloning it is https:///a. Configuring Clone Base URL to be https:///a solves this issue.

This should be added in the Troubleshoot section of the Readme.

@vyguera Thanks for the suggestion. Would you be willing to come up with a proposal text? Happy to add it then!


Memory issue with training on Monai with large datasets

Hey All! So as discussed with @naga-karthik I've had some issues with the "aggregated training".
I am running into memory issues where no fixes seem to work.
I am using Naga's training script and tried pretty much all solutions I could find, but I always get a RuntimeError('Pin memory thread exited unexpectedly'). This can be fixed by using 0 workers in these lines, but makes the training super slow (15+ mins per epoch).
I tried with the exact same configurations as Naga too. The only difference is the amount of images. Plus, it works when removing most images from the dataset.json file (so when working with a much smaller set).
I am still investigating this issue...

This is indeed strange. Can you also post the exact arguments that you're using with main.py?

Pretty much no argument, just correct paths to data and default unet, 100 epochs, and your hard coded values.

@louisfb01 can you please list in this issue thread the various discussions on this topic in MONAI GH, slack, forums, etc.

Solutions tried:

Adding this line to the training script did not work torch.multiprocessing.set_sharing_strategy('file_system'), as in this issue.
Increasing ulimit was another solution but it is limited to 2000 for a user in Romane. Increasing from 1024 to 2000 did not solve the problem.
Changing pin_memory to False in those lines did not solve the issue, rather caused another problem received 0 items of ancdata, similar to here.
Changing the training batch size to 1,2,4 did not help.
Changing the number of workers to 1,2,4,10 did not help.
Changing CacheDataset's number of workers did not help either.
Changing CacheDataset's cache_rate to 0.1 instead of 0.25 did not help.

Next to try is adding the code snippet below as per this answer:
import resource
rlimit = resource.getrlimit(resource.RLIMIT_NOFILE)
resource.setrlimit(resource.RLIMIT_NOFILE, (2048, rlimit[1]))

The current workaround is to use num_workes=0 as in this issue.

@louisfb01 can you please show us the full STDOUT

Here is more information about the environment (python version 3.9.17) and the output I get from running main.py (training with PyTorch, PyTorch lightning, MONAI).
 STDOUT
(monai_training) lobouz@romane:~/github/contrast-agnostic-softseg-spinalcord/monai$ CUDA_VISIBLE_DEVICES=3 python main.py -m unet -nspv 4 -ncv 1 -initf 8 -bs 4 -lr 1e-3 -cve 4 -stp -epb
Global seed set to 42
2023-07-18 10:01:13.007 | INFO     | __main__:main:472 -  Training on fold 1 out of 1 folds!
/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/utilities/parsing.py:196: UserWarning: Attribute 'loss_function' is an instance of `nn.Module` and is already saved during checkpointing. It is recommended to ignore them using `self.save_hyperparameters(ignore=['loss_function'])`.
  rank_zero_warn(
wandb: W&B API key is configured. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.15.5
wandb: Run data is saved locally in /home/GRAMES.POLYMTL.CA/lobouz/contrast-agnostic/saved_models/wandb/run-20230718_100119-9xlhu2uf
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run unet_nf=8_nrs=2_lr=0.001_20230718-1001
wandb: ⭐ View project at https://wandb.ai/whats_ai/contrast-agnostic
wandb: 🚀 View run at https://wandb.ai/whats_ai/contrast-agnostic/runs/9xlhu2uf
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/monai/utils/deprecate_utils.py:321: FutureWarning: monai.transforms.io.dictionary LoadImaged.__init__:image_only: Current default value of argument `image_only=False` has been deprecated since version 1.1. It will be changed to `image_only=True` in version 1.3.
  warn_deprecated(argname, msg, warning_category)
Loading dataset: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 476/476 [01:45<00:00,  4.53it/s]
Loading dataset: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 119/119 [00:37<00:00,  3.20it/s]
Loading dataset: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 29/29 [00:10<00:00,  2.78it/s]
You are using a CUDA device ('NVIDIA RTX A6000') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [3]

  | Name          | Type         | Params
-----------------------------------------------
0 | net           | UNet         | 1.2 M
1 | loss_function | SoftDiceLoss | 0
-----------------------------------------------
1.2 M     Trainable params
0         Non-trainable params
1.2 M     Total params
4.809     Total estimated model params size (MB)
Sanity Checking DataLoader 0: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00,  1.34s/it]Current epoch: 0
Average Soft Dice (VAL): 0.0071
Average Hard Dice (VAL): 0.0020
Best Average Soft Dice: 0.0071 at Epoch: 0
----------------------------------------------------
Epoch 0:   3%|████▌                                                                                | 14/477 [00:23<12:59,  1.68s/it, v_num=u2uf]Exception in thread Thread-28:
Traceback (most recent call last):
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/torch/utils/data/_utils/pin_memory.py", line 51, in _pin_memory_loop
    do_one_step()
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/torch/utils/data/_utils/pin_memory.py", line 28, in do_one_step
    r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL)
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/multiprocessing/queues.py", line 122, in get
    return _ForkingPickler.loads(res)
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/torch/multiprocessing/reductions.py", line 307, in rebuild_storage_fd
    fd = df.detach()
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/multiprocessing/reduction.py", line 189, in recv_handle
    return recvfds(s, 1)[0]
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/multiprocessing/reduction.py", line 164, in recvfds
    raise RuntimeError('received %d items of ancdata' %
RuntimeError: received 0 items of ancdata
Traceback (most recent call last):
  File "/home/GRAMES.POLYMTL.CA/lobouz/github/contrast-agnostic-softseg-spinalcord/monai/main.py", line 605, in <module>
    main(args)
  File "/home/GRAMES.POLYMTL.CA/lobouz/github/contrast-agnostic-softseg-spinalcord/monai/main.py", line 516, in main
    trainer.fit(pl_model)
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 531, in fit
    call._call_and_handle_interrupt(
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 570, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 975, in _run
    results = self._run_stage()
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1018, in _run_stage
    self.fit_loop.run()
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 201, in run
    self.advance()
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 354, in advance
    self.epoch_loop.run(self._data_fetcher)
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 133, in run
    self.advance(data_fetcher)
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 189, in advance
    batch = next(data_fetcher)
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/loops/fetchers.py", line 136, in __next__
    self._fetch_next_batch(self.dataloader_iter)
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/loops/fetchers.py", line 150, in _fetch_next_batch
    batch = next(iterator)
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/utilities/combined_loader.py", line 284, in __next__
    out = next(self._iterator)
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/utilities/combined_loader.py", line 65, in __next__
    out[i] = next(self.iterators[i])
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 634, in __next__
    data = self._next_data()
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1329, in _next_data
    idx, data = self._get_data()
  File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1290, in _get_data
    raise RuntimeError('Pin memory thread exited unexpectedly')
RuntimeError: Pin memory thread exited unexpectedly

Environment details (pip list)
(monai_training) lobouz@romane:~$ pip list
Package                       Version
----------------------------- --------------
absl-py                       1.1.0
aiohttp                       3.8.4
aiosignal                     1.3.1
appdirs                       1.4.4
astor                         0.8.1
asttokens                     2.2.1
astunparse                    1.6.3
async-timeout                 4.0.2
attrs                         21.2.0
awscli                        1.22.34
backcall                      0.2.0
backports.functools-lru-cache 1.6.5
beautifulsoup4                4.11.2
beniget                       0.4.1
bids-validator                1.9.9
blinker                       1.4
botocore                      1.23.34
Brotli                        1.0.9
brz-etckeeper                 0.0.0
cachetools                    5.2.0
certifi                       2023.5.7
cffi                          1.15.1
chardet                       4.0.0
charset-normalizer            3.1.0
click                         8.1.3
cmake                         3.26.4
colorama                      0.4.6
coloredlogs                   15.0.1
comm                          0.1.3
command-not-found             0.3
commonmark                    0.9.1
contourpy                     1.1.0
cryptography                  3.4.8
csv-diff                      1.1
cycler                        0.11.0
dbus-python                   1.2.18
debugpy                       1.6.7
decorator                     5.1.1
Deprecated                    1.2.13
dictdiffer                    0.9.0
dill                          <IP_ADDRESS>
distlib                       0.3.4
distro                        1.7.0
distro-info                   1.1build1
dnspython                     2.1.0
docker                        5.0.3
docker-compose                1.29.2
docker-pycreds                0.4.0
dockerpty                     0.4.1
docopt                        0.6.2
docutils                      0.17.1
entrypoints                   0.4
executing                     1.2.0
filelock                      3.12.2
flatbuffers                   2.0.7
fonttools                     4.40.0
formulaic                     0.3.4
frozenlist                    1.3.3
fsleyes                       1.7.0
fsleyes-props                 1.8.2
fsleyes-widgets               0.14.2
fslpy                         3.13.0
fsspec                        2023.6.0
gast                          0.4.0
gdown                         4.6.4
gitdb                         4.0.10
GitPython                     3.1.31
gmpy2                         2.1.2
google-auth                   2.19.0
google-auth-oauthlib          1.0.0
google-pasta                  0.2.0
gpg                           1.16.0-unknown
grpcio                        1.54.2
h5py                          3.7.0
httplib2                      0.20.2
humanfriendly                 10.0
humanize                      4.4.0
idna                          3.4
imageio                       2.22.4
imgaug                        0.2.5
importlib-metadata            6.8.0
importlib-resources           6.0.0
interface-meta                1.3.0
iotop                         0.6
ipykernel                     6.24.0
ipython                       8.14.0
ivadomed                      2.9.7
jax                           0.4.11
jedi                          0.18.2
jeepney                       0.7.1
Jinja2                        3.1.2
jmespath                      0.10.0
joblib                        1.3.0
jsonschema                    3.2.0
jupyter_client                8.3.0
jupyter_core                  5.3.1
keras                         2.12.0
Keras-Preprocessing           1.1.2
keyring                       23.5.0
kiwisolver                    1.4.4
launchpadlib                  1.10.16
lazr.restfulclient            0.14.4
lazr.uri                      1.0.6
libclang                      14.0.1
lightning-utilities           0.9.0
lit                           16.0.6
loguru                        0.7.0
Markdown                      3.3.6
MarkupSafe                    2.1.3
matplotlib                    3.7.2
matplotlib-inline             0.1.6
ml-dtypes                     0.1.0
monai                         1.2.0
monai-weekly                  1.2.dev2311
more-itertools                8.10.0
mpmath                        1.3.0
multidict                     6.0.4
nest-asyncio                  1.5.6
netifaces                     0.11.0
networkx                      3.1
nibabel                       5.1.0
num2words                     0.5.12
numpy                         1.25.0
nvidia-cublas-cu11            <IP_ADDRESS>
nvidia-cuda-cupti-cu11        11.7.101
nvidia-cuda-nvrtc-cu11        11.7.99
nvidia-cuda-runtime-cu11      11.7.99
nvidia-cudnn-cu11             <IP_ADDRESS>
nvidia-cufft-cu11             <IP_ADDRESS>
nvidia-curand-cu11            <IP_ADDRESS>
nvidia-cusolver-cu11          <IP_ADDRESS>
nvidia-cusparse-cu11          <IP_ADDRESS>
nvidia-nccl-cu11              2.14.3
nvidia-nvtx-cu11              11.7.91
oauthlib                      3.2.0
onnxruntime                   1.13.1
opt-einsum                    3.3.0
osfclient                     0.0.5
packaging                     23.1
pandas                        2.0.3
parso                         0.8.3
pathtools                     0.1.2
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        10.0.0
pip                           23.1.2
platformdirs                  3.8.0
ply                           3.11
pooch                         1.7.0
promise                       2.3
prompt-toolkit                3.0.39
protobuf                      3.20.3
psutil                        5.9.5
ptyprocess                    0.7.0
pure-eval                     0.2.2
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pybids                        0.15.5
pycparser                     2.21
Pygments                      2.15.1
PyGObject                     3.42.1
PyJWT                         2.3.0
pymacaroons                   0.13.0
PyNaCl                        1.5.0
PyOpenGL                      3.1.6
pyparsing                     3.0.9
pyrsistent                    0.18.1
PySocks                       1.7.1
python-apt                    2.4.0+ubuntu1
python-dateutil               2.8.2
python-dotenv                 0.19.2
python-magic                  0.4.24
pythran                       0.10.0
pytorch-ignite                0.4.11
pytorch-lightning             2.0.4
pytz                          2023.3
PyWavelets                    1.4.1
PyYAML                        6.0
pyzmq                         25.1.0
requests                      2.31.0
requests-oauthlib             1.3.1
requests-toolbelt             0.9.1
rich                          12.6.0
roman                         3.3
rsa                           4.8
s3transfer                    0.5.0
scikit-image                  0.19.3
scikit-learn                  1.3.0
scipy                         1.11.1
screen-resolution-extra       0.0.0
seaborn                       0.12.1
SecretStorage                 3.3.1
sentry-sdk                    1.21.1
setproctitle                  1.3.2
setuptools                    68.0.0
shellingham                   1.5.0
shortuuid                     1.0.11
SimpleITK                     2.2.1
six                           1.16.0
smmap                         3.0.5
sos                           4.4
soupsieve                     2.4
SQLAlchemy                    1.3.24
ssh-import-id                 5.11
stack-data                    0.6.2
sympy                         1.12
systemd-python                234
tensorboard                   2.12.3
tensorboard-data-server       0.7.0
tensorboard-plugin-wit        1.8.1
tensorflow                    2.12.0
tensorflow-estimator          2.12.0
tensorflow-io-gcs-filesystem  0.26.0
termcolor                     1.1.0
texttable                     1.6.4
threadpoolctl                 3.1.0
tifffile                      2022.10.10
torch                         2.0.0
torchaudio                    2.0.1
torchio                       0.18.86
torchmetrics                  0.11.4
torchvision                   0.15.1
tornado                       6.3.2
tqdm                          4.65.0
traitlets                     5.9.0
triton                        2.0.0
typer                         0.7.0
typing_extensions             4.7.1
tzdata                        2023.3
ufw                           0.36.1
unattended-upgrades           0.1
urllib3                       2.0.3
virtualenv                    20.13.0+ds
wadllib                       1.3.6
wandb                         0.15.5
wcwidth                       0.2.6
websocket-client              1.2.3
Werkzeug                      2.1.2
wheel                         0.40.0
wrapt                         1.14.1
wxPython                      4.0.7
xkit                          0.0.0
yarl                          1.9.2
zipp                          3.15.0

Updated the answer above with a new temporary fix.
Adding this line to the training script did work torch.multiprocessing.set_sharing_strategy('file_system'), as in this issue. I believe I previously did not add it in the right section of the code by mistake. It now works as expected!
It is not super clean but it at least allows us to train normally for now.

It is not super clean

Why is that?

I just realized this magic syntax also fixed an issue for me in the past 😅
https://github.com/jcohenadad/model-seg-ms-mp2rage-monai/commit/450f72e471b33e668109f55195db92811ef15d78

Is there any deeper explanation anywhere as to why this fix is working?

Is there any deeper explanation anywhere as to why this fix is working?

This is what I found:
torch.multiprocessing is a wrapper around the native multiprocessing module. It registers custom reducers, that use shared memory to provide shared views on the same data in different processes. Once the tensor/storage is moved to shared_memory (see share_memory_()), it will be possible to send it to other processes without making any copies. (from torch documentation)
And from what I understand, this memory issue comes from using the CacheDataset and it has to do with pytorch's sharing strategy. The function torch.multiprocessing.set_sharing_strategy('file_system') will enable a flag to not create extra file descriptors, more information here.
It still seems to be a temporary fix to "having a high enough limit" in our system. This can be done increasing ulimit, but not in our case since we are limited as a user in Romane.

Still, if your system has high enough limits, and file_descriptor is a supported strategy, we do not recommend switching to this one.

And from what I understand, this memory issue comes from using the CacheDataset and it has to do with pytorch's sharing strategy.

ah this is a good point! Monai also provides PyTorch's native Dataset class. Could you please try using that once and remove the multiprocessing fix to see if it is really CacheDataset that's the culprit? (you just have switch to Dataset and use the right arguments, just a line of code)

I tried using MONAI's Dataset class instead of CacheDataset.
Using Dataset does not allow us to remove  torch.multiprocessing.set_sharing_strategy('file_system').
Though using MONAI's Dataset class instead of CacheDataset fixed an issue with all the aggregated datasets only happening at 73% of the epoch:
Runtime Error: unable to map 2112 bytes from file </torch_275418_2316075744_27627>: Cannot allocate memory (12)

I think the next step is to investigate using compute Canada and train with more GPUs and memory.

👍

closing this as it is not relevant anymore -- I was able to train the model on 11 datasets at the moment

Hi @naga-karthik ! I am re-opening this issue as I am facing the same problem as @louisfb01 when training using monai and large datasets. My training crashes during the validation step even though I am using torch.multiprocessing.set_sharing_strategy('file_system'). Here is my code..
The error I get is the following:

Error message

Traceback (most recent call last):
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/torch/utils/data/_utils/pin_memory.py", line 51, in _pin_memory_loop
    do_one_step()
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/torch/utils/data/_utils/pin_memory.py", line 28, in do_one_step
    r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL)
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/multiprocessing/queues.py", line 122, in get
    return _ForkingPickler.loads(res)
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/torch/multiprocessing/reductions.py", line 324, in rebuild_storage_filename
    storage = torch.UntypedStorage._new_shared_filename_cpu(manager, handle, size)
RuntimeError: unable to mmap 68 bytes from file </torch_1405374_144476348_9956>: Cannot allocate memory (12)
Traceback (most recent call last):
  File "/home/plbenveniste/ms_lesion_agnostic/ms-lesion-agnostic/monai/train_monai_unet_lightning.py", line 823, in <module>
    main()
  File "/home/plbenveniste/ms_lesion_agnostic/ms-lesion-agnostic/monai/train_monai_unet_lightning.py", line 815, in main
    trainer.fit(pl_model)
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 544, in fit
    call._call_and_handle_interrupt(
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 580, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 987, in _run
    results = self._run_stage()
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1033, in _run_stage
    self.fit_loop.run()
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 205, in run
    self.advance()
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 363, in advance
    self.epoch_loop.run(self._data_fetcher)
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 141, in run
    self.on_advance_end(data_fetcher)
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 295, in on_advance_end
    self.val_loop.run()
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/pytorch_lightning/loops/utilities.py", line 182, in _decorator
    return loop_run(self, *args, **kwargs)
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 128, in run
    batch, batch_idx, dataloader_idx = next(data_fetcher)
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/pytorch_lightning/loops/fetchers.py", line 133, in __next__
    batch = super().__next__()
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/pytorch_lightning/loops/fetchers.py", line 60, in __next__
    batch = next(self.iterator)
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/pytorch_lightning/utilities/combined_loader.py", line 341, in __next__
    out = next(self._iterator)
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/pytorch_lightning/utilities/combined_loader.py", line 142, in __next__
    out = next(self.iterators[0])
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 633, in __next__
    data = self._next_data()
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1328, in _next_data
    idx, data = self._get_data()
  File "/home/plbenveniste/miniconda3/envs/venv_monai/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1289, in _get_data
    raise RuntimeError('Pin memory thread exited unexpectedly')
RuntimeError: Pin memory thread exited unexpectedly

I had to set num_workers to zero in the validation to get the code to work. Is there any cleaner one around this ? (Btw I am running this on this server: GPU/CPU Server - 512 cores-threads – Epyc 3.0GHz – 128G RAM – 4 GPUs (4x Nvidia A6000) and the memory is not reaching its limit.

@plbenveniste Have you had any luck with finding a solution?


develop app CR versioning story

Issues like:

[ ] how to express that different app CRs are versions of the same app
[ ] how registering a new app version affects existing clusters (if at all)
[ ] how/if to allow a cluster CR to reference "latest version of this app"

Related to issue #229.
The consensus at the moment seems to be that:

The distroID property is used to identify app CRs that are "for the same app" even if of different versions.
The version property identifies revisions of a given distroID.
The resource name is the hook referenced by KubeDirectorCluster.

If we follow through on something like the approach described in issue #229, then an existing CR could be edited to replace it with a newer version of the same distroID, to allow live upgrade of any clusters referencing that CR.
The admin could instead create an entirely new CR (with a different name) instead of editing an existing CR. This would leave any existing clusters alone, and new clusters could choose which version of the app they want to instantiate.
In this scenario there's no built-in way for a KDCluster to express "latest version of this app". The client would have to look for app CRs that have the distroID of interest, examine their version properties, and choose the app CR that has the desired version.

Developing this as part of our live app upgrade work, so it is also reasonable to expect this for the 1.0 release.


Add wallet Contact List, torStarting, wallet-ts offer/contact types

Replaces #144
Adds wallet Contact List, Tor Starting awareness and polling, and wallet-ts types and functions for Offers and Contacts.

Tor Starting

Offer Peer labeling from Contacts

It doesn't seem like the tor started checkbox is checked even though i see my tor address and it is true from the command line

Showing contact on the main contracts view will come next. The backend on master does not currently support this.


Slow progress with instant filtering + adaptive choices

(Original issue 11 created by None on 2011-03-20T22:50:59+00:00)
On version 4.1, with instant filtering and adaptive choices, choices are extracted for
all editors as the user enters text on one editor.

As result, system feels rather slow

Original issue reported on code.google.com by luismpena on 2011-03-20 22:50:59

(Original issue 11 last updated on 2015-04-26T00:25:19.781745+00:00)
(Issue automaticaly closed due to status in Bitbucket: resolved)

(Original comment by coderazzi on 2011-03-20T23:16:42+00:00)
(No text was entered with this change)

Original issue reported on code.google.com by coderazzi on 2011-03-20 23:16:42

(Original comment by coderazzi on 2011-03-23T04:07:19+00:00)
Solved on release 4.1.2

Original issue reported on code.google.com by coderazzi on 2011-03-23 04:07:19

(Original comment by coderazzi on 2011-11-14T23:12:34+00:00)
(No text was entered with this change)

Original issue reported on code.google.com by coderazzi on 2011-11-14 23:12:34

(Original comment by coderazzi on 2015-04-26T00:25:19.790480+00:00)
Already resolved (previously, before migrating to BitBucket)


False positive in

I don't see anything about

A more sophisticated, but still hacky, setting for raw-ignore-regex is /\<script\>[^]*?\<\/script\>/, which will avoid checking the contents inside <script>...</script> (exact spelling) at all. However, I do hope to find a sensible solution to make htmlling handle embedded non-html stuff like Javascript and svgs gracefully.

Alright some time over at https://regex101.com/ yielded the following.
Unfortunately it ignores the whole script tag part, not just the stuff in the middle.
"raw-ignore-regex": "/(\\<script.*?\\>\\<\/script\\>|\\<script.*?\\>[\\w\\W]+?\\<\/script\\>)/"

Lots of extra escaping


Launcher shortcuts for keys

Instead of opening the app & navigating to the key every time, it would be nice to have a shortcut that sends you directly to the key so you can emulate it with only one extra click.

What about widgets?


Asynchronous File Saving

Sponge Version:  7.2.0
Forge Version:  1.12.2-<IP_ADDRESS>54
Total Economy Version:  1.8.2
Description of issue
File saving happens on the main thread in most cases. We've forked the plugin to use sponge's executor service that you use in autosave on all saving functionality, however this may be a change you're interested in implementing yourself.
Console errors
None
Steps to reproduce
Cause any file to be saved. A larger than normal file will cause the main thread to hang.
Related Configuration Files
Create a Gist (https://gist.github.com/) for each configuration file related to this issue

Hi there :)
If you already have the code, why not clean it up a bit and submit a PR for that feature 🙂
We would love to see it ❤


It is better for interfacer not to analyze vendor folder

interfacer is a good tool for gophers. I have tried it and got many help. I have a small issue that is currently the interfacer tries to analyze the vendor folder in the golang repo. In my opinion, It is better for interfacer not to analyze vendor folder.
$interfacer ./...
/Users/tony/Test/GoToolsProjects/src/app-weixin/vendor/github.com/astaxie/beego/session/couchbase/sess_couchbase.go:40:12: could not import github.com/couchbase/go-couchbase (cannot find package "github.com/couchbase/go-couchbase" in any of:
	/Users/tony/Test/GoToolsProjects/src/app-weixin/vendor/github.com/couchbase/go-couchbase (vendor tree)
	/Users/tony/.bin/go17/src/github.com/couchbase/go-couchbase (from $GOROOT)
	/Users/tony/Test/GoToolsProjects/src/github.com/couchbase/go-couchbase (from $GOPATH))

The generally suggested (by Go authors) unix-y solution to this is to use grep -v /vendor/:
$ interfacer $(go list ./... | grep -v /vendor/)

That way each tool doesn't have to re-implement this functionality.

Agreed. Otherwise this should make its way to a library like go/loader.

@mvdan I accept the suggestion by @shurcooL . Btw,  Add this to README is better.


docs: Add Clever Cloud deploy documentation page

Status
READY
Description
I've added a documentation page that helps deploying a dart_frog project on Clever Cloud, a PaaS provider that uses git to deploy.
Type of Change

[ ] ✨ New feature (non-breaking change which adds functionality)
[ ] 🛠 Bug fix (non-breaking change which fixes an issue)
[ ] ❌ Breaking change (fix or feature that would cause existing functionality to change)
[ ] 🧹 Code refactor
[ ] ✅ Build configuration change
[x] 📝 Documentation
[ ] 🗑 Chore

Hum...
I got why the spell check is KO but its unfortunate as the reasons are
Warning: Unknown word (WORKDIR)
Warning: Unknown word (cleverapps)

But the first one is for the Dockerfile and the second one is from the url :/

Thank you @wolfenrain and @Mayaa-s for all the edits ! This is way better now 👍


write_sdc result is gzipped unprompted

Describe the bug
When attempting to write_sdc, the result appears to be a gzip file:

The command is just write_sdc. There's no -gzip command being passed.
Regardless, read_sdc fails on this gzipped file.
Expected Behavior
The file should be plain text, unless we're missing something?
Environment
Git commit: 5d4aeb1c605719de28366908f4fbaae439a57d05
kernel: Linux 6.0.19
os: NixOS 22.11 (Raccoon)
cmake version 3.24.3
-- The CXX compiler identification is Clang 11.1.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /nix/store/naqfglx95i283dwna1zzqfrq3zgyfg89-clang-wrapper-11.1.0/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- OpenROAD version: v2.0-7456-gd5e88bef0
-- System name: Linux
-- Compiler: Clang 11.1.0
-- Build type: RELEASE
-- Install prefix: /var/empty/local
-- C++ Standard: 17
-- C++ Standard Required: ON
-- C++ Extensions: OFF
-- Could NOT find GTest (missing: GTEST_LIBRARY GTEST_INCLUDE_DIR GTEST_MAIN_LIBRARY)
-- The C compiler identification is Clang 11.1.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /nix/store/naqfglx95i283dwna1zzqfrq3zgyfg89-clang-wrapper-11.1.0/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Python: /nix/store/65cp4izx3bllnwqn7c7dhrq9h9gmjkal-python3-3.10.9/bin/python3.10 (found version "3.10.9") found components: Interpreter
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Performing Test C_COMPILER_SUPPORTS__-Wall
-- Performing Test C_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register
-- Performing CMake Warning at src/CMakeLists.txt:226 (message):
  spdlog: SPDLOG_FMT_EXTERNAL=ON

CMake Error at /nix/store/bjnxp0pm2xdgsmv4m8bb2y60bjd1irc2-boost-1.79.0-dev/lib/cmake/Boost-1.79.0/BoostConfig.cmake:141 (find_package):
  Found package configuration file:

    /nix/store/bjnxp0pm2xdgsmv4m8bb2y60bjd1irc2-boost-1.79.0-dev/lib/cmake/boost_serialization-1.79.0/boost_serialization-config.cmake

  but it set boost_serialization_FOUND to FALSE so package
  "boost_serialization" is considered to be NOT FOUND.  Reason given by
  package:

  No suitable build variant has been found.

  The following variants have been tried and rejected:

  * libboost_serialization.so.1.79.0 (shared, Boost_USE_STATIC_LIBS=ON)

Call Stack (most recent call first):
  /nix/store/bjnxp0pm2xdgsmv4m8bb2y60bjd1irc2-boost-1.79.0-dev/lib/cmake/Boost-1.79.0/BoostConfig.cmake:262 (boost_find_component)
  /nix/store/0257lz452871n4sp11afxfhm1g94dj4b-cmake-3.24.3/share/cmake-3.24/Modules/FindBoost.cmake:594 (find_package)
  src/drt/CMakeLists.txt:49 (find_package)

Test C_COMPILER_SUPPORTS__-Wno-register - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Success
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive - Success
-- Performing Test C_COMPILER_SUPPORTS__-x
-- Performing Test C_COMPILER_SUPPORTS__-x - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-x
-- Performing Test CXX_COMPILER_SUPPORTS__-x - Failed
-- Performing Test C_COMPILER_SUPPORTS__c++
-- Performing Test C_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__c++
-- Performing Test CXX_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Failed
-- TCL library: /nix/store/378rxqdx6lh4swhw49ifbilc6hbk7h2r-tcl-8.6.11/lib/libtcl.so
-- TCL header: /nix/store/378rxqdx6lh4swhw49ifbilc6hbk7h2r-tcl-8.6.11/include/tcl.h
-- TCL readline library: /nix/store/90fgbs1df2m2m7x0p40s35fviz1y3kj4-tclreadline-2.3.8/lib/libtclreadline.so
-- TCL readline header: /nix/store/90fgbs1df2m2m7x0p40s35fviz1y3kj4-tclreadline-2.3.8/include
-- Found SWIG: /nix/store/b31v2mp1mla6qbiwjlgixjnh5x426aw8-swig-3.0.12/bin/swig (found suitable version "3.0.12", minimum required is "3.0")
-- Found Boost: /nix/store/bjnxp0pm2xdgsmv4m8bb2y60bjd1irc2-boost-1.79.0-dev/lib/cmake/Boost-1.79.0/BoostConfig.cmake (found version "1.79.0")
-- boost: 1.79.0
-- Found Python3: /nix/store/65cp4izx3bllnwqn7c7dhrq9h9gmjkal-python3-3.10.9/include/python3.10 (found version "3.10.9") found components: Development Development.Module Development.Embed
-- Found ZLIB: /nix/store/fblaj5ywkgphzpp5kx41av32kls9256y-zlib-1.2.13/lib/libz.so (found version "1.2.13")
-- spdlog: 1.10.0
-- Found BISON: /nix/store/1gl7s2jiq3hfliz3giv7hwc6gwl8w5p2-bison-3.8.2/bin/bison (found version "3.8.2")
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- STA version: 2.4.0
-- STA git sha: 20805c83cf4817f6d2ca7867a52dac226515a21e
-- System name: Linux
-- Compiler: Clang 11.1.0
-- Build type: RELEASE
-- Build CXX_FLAGS: -O3 -DNDEBUG
-- Install prefix: /var/empty/local
-- Found FLEX: /nix/store/5lzf74h8ygk5y5r1cn8a6qc9567cwcw1-flex-2.6.4/bin/flex (found version "2.6.4")
-- TCL library: /nix/store/378rxqdx6lh4swhw49ifbilc6hbk7h2r-tcl-8.6.11/lib/libtcl.so
-- TCL header: /nix/store/378rxqdx6lh4swhw49ifbilc6hbk7h2r-tcl-8.6.11/include/tcl.h
-- SSTA: 0
-- STA executable: /home/donn/efabless/openroad/src/sta/app/sta
-- GPU is not enabled
-- Found Protobuf: /nix/store/lyklcycj3jdbg6bp80n7pjivizzcfr0c-protobuf-3.21.8/lib/libprotobuf.so (found version "3.21.8")
-- Found re2: /nix/store/7b2zq7xn1yayhxmrwgsxklfckxyzhfqv-re2-2022-06-01-dev/lib/cmake/re2/re2Config.cmake (found version "9.0.0")
-- Found GLPK: /nix/store/ycwzcwxr4a3ldpa7b19ypp9zpcxx4gql-glpk-5.0/lib/libglpk.so
-- Found Eigen3: /nix/store/gzbmbqgjgqm91syx80vn8lzj6gmljw5q-eigen-3.4.0/share/eigen3/cmake/Eigen3Config.cmake (found version "3.4.0")
-- GUI is enabled
-- Configuring incomplete, errors occurred!

To Reproduce
./run.sh
repro.tar.gz
Relevant log output
No response
Screenshots
No response
Additional Context
No response

Is this really your build configuration: Configuring incomplete, errors occurred!
I ran with the head of OR (d5e88bef0089b8d0525f46ac1e7f74c352b16e26) and ./05-openroad-floorplan/spm.sdc is a plain text file.

I define USE_SYSTEM_BOOST, the ./etc/Env.sh account for that.

Ugh, yep, a corner case to end all corner cases.
So write_sdc uses gzopen:
https://github.com/The-OpenROAD-Project/OpenSTA/blob/882681a674e7089e5f02de4ba2050c6be32b6cd1/sdc/WriteSdc.cc#L330
...which in my compiled OR binary is:
https://github.com/The-OpenROAD-Project/abc/blob/95b3543e928640dfa25f9e882e72a090a8883a9c/src/misc/zlib/gzlib.c#L198
as a shim for:
https://github.com/The-OpenROAD-Project/abc/blob/95b3543e928640dfa25f9e882e72a090a8883a9c/src/misc/zlib/gzlib.c#L92
The OpenSTA invocation uses "T", a mode flag that isn't supported by this implementation which it simply silently ignores. My hypothesis currently is that if ABC is built in an environment without a system zlib, it builds the included zlib, which causes this break. Testing further...

Yeah, it's because I compile OpenROAD abc standalone then OpenROAD. I opened a PR for abc.

You'll need to email Alan, the abc author, directly. He doesn't often respond to PRs unless pointed at one though email

@QuantamHD This is OpenROAD's ABC port...

I'm not sure we're committing patches to those forks.


Is the 3.0.6 release tagged somewhere?

I see the JARs on Maven Central. wondering what's changed

I just merged a PR that the release included. So 3.0.x currently is what is in 3.0.6. Chee Seng will be making a tag as soon as he gets to it.

@SethTisue I just created the release here:
https://github.com/scalatest/scalatest/releases/tag/release-3.0.6
with the release note, sorry for the delay.

no worries, thanks!


Can't copy BlockBlob as PageBlob type to another container

Which version of the AzCopy was used?
10.4.0
Which platform are you using? (ex: Windows, Mac, Linux)
Windows 10 1909 (18363.778)
What command did you run?
./azcopy.exe copy "https://storageaccount1.blob.core.windows.net/vhds/VM001_OsDisk_1.vhd?SAS-URI" "https://storageaccount2.blob.core.windows.net/vhds/VM001.vhd?SAS-URI" --blob-type=PageBlob
What problem was encountered?
I'm trying to copy BlockBlob vhd file to another storage account, but I need to convert it to PageBlob, so I have specified --blob-type=PageBlob parameter. All the time with this parameter job fails, without this parameter job completes successfully.
How can we reproduce the problem in the simplest way?
Have you found a mitigation/solution?
No.

That is supposed to work.  Can I suggest you delete the target (destination) blob, then try one more time. If that fails, please provide more details about the nature of the failure. E.g. the section of the log file where the failure is reported.  You can email log extracts (or the whole file) to me if you don't want to post the extract here.

Hi @AIGARSJ are you still experiencing this issue? I will close this for now due to inactivity on the thread but please feel free to repoen it if you are still hitting it.


Removed events from grid in local map

Also created boiler plates for pawns that have some type of interaction with the player. They work with both raycasts (in case we want something like pokemon) and area2ds for something that should only trigger on contact.
Closes #50

Instead of matching based on some kind of strict type, it could be more useful for NPCs to have what they need injected into them and their interact functions would fully control how an interaction works.  You could cut down on the number of needed signal types and make the NPCs more powerful that way too.


#set par(justify: true) affects code formatting

Hi, thanks for the lib.
I noticed that #set par(justify: true) affects how code is formatted.
See https://typst.app/project/rDWZvNPK0LKNweW5mBOTRP

Could you just add a par(justify:false) to the codeblocks on the library side?

I'll do that since I think in general this can be considered an upstream bug: https://github.com/typst/typst/issues/3454

This is done in one of my recent commits. Will remove that line in the next version of Typst (0.13)

Thanks!


feat: add sponsor section

Your checklist for this pull request
🚨Please review the guidelines for contributing to this repository.

[ ] Make sure you are requesting to pull a topic/feature/bugfix branch (right side). Don't request your default branch!
[ ] Make sure you are making a pull request against the default branch (left side). Also you should start your branch off default branch.
[ ] Check the commit's or even all commits' message styles matches our requested structure.
[ ] Check your code additions will fail neither code linting checks nor unit test.
[ ] I have added necessary documentation (if appropriate)

Description
Please describe your pull request.
❤Thank you!
Post merge checklist

[ ] Follow steps from the guidelines for contributing to this repository.
[ ] If you are a new contributor, ping in the thread and one of the maintainers will add you to all-contributors list.

Complete this PR ASAP @praptippradhan

Fix the issues after pulling changes

I guess you have pushed from dev branch directly so we are getting all the previous commits. Please reset your commits.


feat: added support in logSuccess and logInfo for multiple parameters

Coverage report

St.:grey_question:
Category
Percentage
Covered / Total

🟢
Statements
100%
91/91

🟢
Branches
100%
9/9

🟢
Functions
100%
6/6

🟢
Lines
100%
91/91

Test suite run success
5 tests passing in 1 suite.
Report generated by 🧪jest coverage report action from cc40122de030bd391c094d5ab81524db932cf4a9


consider having delay which common in JS/TS land to do a sleep

bluebird and in general many JS codebases use delay... which pushes cahtgpt to use it.

top-level delay, Promise.delay or ds.delay ?

import { delay } from "@devicescript/core"

feel free to add ds_impl.delay = sleep ;)


Update SEP-12 GET /customer schema with 'provided_fields' object

resolves #66

Preview is available here:http://stellar-anchor-tests-pr68.previews.kube001.services.stellar-ops.com/


CI: do linkcheck on SPECS docs

Some of the specs are full of links to third party pages of tools and other resources. We need to make sure these links don't rot over time.

We checklinks with netlify:
https://github.com/scientific-python/specs/blob/f3fad788a636b1a31957c6986f294e56b269beeb/netlify.toml#L6

ah, OK. Also from cron? E.g. I don't expect a link to not work at the time of adding it freshly in the PR.

No cron checks right now. Link checks are quite expensive, but we could do it once a week or so and file an issue. But I haven't set that up before, so hoping someone else is interested...

Oh, sure, I'm not suggesting more frequent checks than weekly/fortnightly.
I do have projects that run linkchecks, but all of those are either rely on jupyter-book or sphinx linkcheck functionality, but I'm sure someone have already found a linkcheck action that works reasonably well so we can just pull the config from an existing project.