Adding IQ2_TN for use with ternary models

They have abandoned the Q1_3 and Q2_2 quants in PR-8151 in llama.cpp, and have moved on to TQ1_0 and TQ2_0. Like k-quants, these use blocks of 256 weights and utilize Q8_K for quantized dot products on the CPU. This removes support for Bitnet b1.58 (unless one adds padding to a multiple of 256), so they are now focussing on the TriLM models. Unlike the previous Q1_3 and Q2_2, where the quantized data only holds the ternary -1/0/+1 values and the tensor scale is added via a separate ggml_scale operation, the new TQ1_0 and TQ2_0 include a scale in each block of 256. This basically wastes 0.0625 bpw, but has the advantage that one can simply reuse the standard llama.cpp computation graphs.
Based on the PP-512 and TG-128 figures posted in PR-8151, TQ2_0 performance is much better than the earlier Q2_0 attempt, so I became curious to see how @compilade's implementation compares to what we can do with iqk_mul_mat in this repo, and here we are.
The PR adds IQ2_TN (TN as TriNet). Implementation for Zen4, AVX2, ARM_NEON, CUDA and Metal is provided.
Let's look at some performance comparisons. We will focus on the largest TriLM model, which has ~4B parameters. Quantized with 2.0625 bpw the model size is 1.08 GiB.
AVX2
AVX2 was tested on a 32-core Ryzen-5975WX CPU. Not everybody has a 32-core CPU handy, so I have added performance values for fewer threads.

threads
test
t/s (PR-8151)
t/s (this PR)
Speedup

32
pp512
430.18 ± 0.56
490.73 ± 0.62
1.141

16
pp512
258.47 ± 0.21
306.37 ± 0.03
1.185

8
pp512
141.94 ± 0.04
175.45 ± 0.06
1.236

4
pp512
74.72 ± 0.02
91.78 ± 0.01
1.228

1
tg128
15.75 ± 0.01
15.71 ± 0.01
1.000

2
tg128
24.22 ± 0.02
26.50 ± 0.00
1.094

4
tg128
33.66 ± 0.14
41.63 ± 0.04
1.237

8
tg128
44.34 ± 0.07
48.62 ± 0.03
1.097

16
tg128
49.58 ± 0.05
48.09 ± 0.03
0.970

I would say @compilade has done remarkably well here, coming to within ~14% for PP performance. Although, for fewer than 32 threads, the performance gap increases to about ~23%. My guess is that the 23% is a more realistic value for the performance difference, and as the number of threads increases we see more the effect of ggml inefficiencies (thread synchronization, operations that do not scale with number of threads, etc.), which then narrows the gap. Nevertheless, even 23% is remarkable considering the performance differences for other quants (see main page). For TG the performance is the same for 1 thread (not much one can do there, the bit arrangement is so simple that there aren't many different ways to implement effectively with AVX2). The implementation in this PR then becomes faster, I guess due to better cache utilization. But this better per thread performance leads to too much memory bandwidth contention above 8 threads, so TQ2_0 is able to arrive at a slightly better performance at 16 threads.
Zen4
I have also tested on a Zen4 CPU (16-core Ryzen-7950X). Zen4 implements some of the AVX512 instruction set, and there is a dedicated implementation for that for IQ2_TN. The TQ2_0 quants are implemented in pure AVX2, so one might think the performance comparison is unfair. But, at least as far as I know,  the Zen4 core implements 512-bit instructions as two separate 256-bit instructions in hardware, so one does not gain much by operating on 512-bit wide vectors. The main advantage comes from having more vector registers (32 vs 16 on AVX2), but the way matrix multiplications are done in ggml (a series of vector x vector dot products), one cannot really take advantage of that. Anyway, here is the performance comparison on the Ryzen-7950X CPU

threads
test
t/s (PR-8151)
t/s (this PR)
Speedup

16
pp512
276.74 ± 0.75
429.97 ± 1.41
1.553

8
pp512
151.50 ± 0.46
250.88 ± 0.31
1.656

4
pp512
78.82 ± 0.64
131.29 ± 0.23
1.665

1
tg128
18.76 ± 0.40
20.11 ± 0.05
1.072

2
tg128
29.38 ± 0.05
35.69 ± 0.07
1.215

4
tg128
46.39 ± 0.04
48.62 ± 0.01
1.048

8
tg128
47.94 ± 0.03
48.28 ± 0.04
1.007

Here the PP performance gap is more significant at around 66%, reducing to 55% at 16 threads. If we look at TG performance for 1 thread, the ~7% performance difference comes from using _mm512_dpbusd_epi32, which is a fused multiply-add operation, whereas on AVX2 one needs to use _mm256_maddubs_epi16 followed by _mm256_add_epi16 to accumulate the result. The TG performance gap then widens due to better cache utilization, and then decreases towards zero with increasing numbers of threads as the memory bandwidth is saturated. The 66% PP performance gap is hence the combination of the ~7% due to the use a fused multiply-add, and ~60% due to better utilization of vector registers while performing a multiplication of a row in the left matrix with several columns in the right matrix, where the unpacked quants for a block are held in vector registers.
ARM_NEON
Here @compilade's implementation does not do very well, at least not on the M2-Max laptop where I have tested. But perhaps this is just due to the fact that @compilade used a Cortex A72 CPU in their development, and that CPU may as well behave very differently from the M2-Max.

threads
test
t/s (PR-8151)
t/s (this PR)
Speedup

8
pp512
79.15 ± 0.21
206.60 ± 0.14
2.610

2
tg128
17.61 ± 0.01
28.42 ± 0.05
1.614

4
tg128
32.40 ± 0.02
49.23 ± 0.09
1.519

8
tg128
51.64 ± 0.70
76.37 ± 0.22
1.479

CUDA and Metal
There is no GPU implementation in PR-8151, so here just the performance values for this PR. CUDA is tested on RTX-4080, Metal on a 30-code M2-Max GPU.

backend
test
t/s (this PR)

CUDA
pp512
9937     ± 81

CUDA
tg128
299.19  ± 0.15

Metal
pp512
891.52  ± 0.49

Metal
tg128
98.52 ± 0.16

I have not bothered implementing the MMQ stuff, so CUDA PP performance is via dequantize and cuBLAS gemm.

This is great!

ARM_NEON
Here @compilade's implementation does not do very well

Yeah, I did not particularly optimize the ARM_NEON implementation for recent ARM CPUs (yet), especially since I did not use vdotq_s32 (although I was planning to), because the Cortex-A72 and the Cortex-A53 in the CPUs of my test machines do not support that and were faster with vmlal_s8 than with ggml_vdotq_s32.

I see IQ2_TN mostly has the same format as TQ2_0, except that the float16 scale is before the packed weights instead of after.
But if I understand it correctly, both store the packed values in the same order and packed in the same way (same offset). Does that mean the Metal and CUDA implementations for IQ2_TN would also work for TQ2_0?
Do you have plans for IQ2_TN to replace TQ2_0, or is this something done in parallel to see how fast it can get with better matrix multiplication than lots of dot products?
Either way, I really appreciate your work on this. This was a pleasant surprise to see in my notifications.


Feature request: change size of projector preview

It would be very useful if one would be able to increase the size of the projector preview. The default settings are smaller than necessary. Look at the attached screenshot - there is space to increase the size. Since every monitor is different the default size should be kept as is but the user should be able to increase the size by dragging the projector window.

This would be a nice feature but no reason to have this in for OS 2.1. I propose to put it in the OS 2.2. agenda to release OS 2.1 soon.

A projector preview does not exist anymore.
@emanuelschuetze can this be closed?

We will work on a new preview for 3.x - This issue can be closed.


Integration tests for all Request Object scenarios

Thank you for your contribution!
We are closing this issue since it has not been prioritized for a long time. Chances are that it has already been solved in more recent versions. If not, we will be re-evaluating this when it becomes a priority.


Adds the loading of the info-content data

this PR adds the loading of a new cache for the information content data.

I don't see where you are getting the ic out of redis?
When we do get it out, we may have a case where both conflated entities have their own ic, in which case we should choose the minimum one.
Do you really want to set all the databases to 0 in the config?

this part just gets the data into redis. i have not worked on getting it out of redis for the website.
all the redis caches use db 0. presumably there is only 1 db per redis instance.

Connections, and redis setup look good. For local testing maybe having different db (0-6) on the default config might be a good idea (since the default config refers to a single redis instance )


Should lintOnSave be renamed?

I actually do want lint on save, which I have configured through my editor, but I choose to keep the lintOnSave option in vue.config.js disabled because it does more than that - it halts compilation on any linting errors (even if it's a minor stylistic issue). I've seen this behavior frustrate some as they didn't expect it, so perhaps we should rename the option to better match what it actually does. haltCompilationOnLintingErrors might be a little unwieldly, but I'm open to other ideas.
Another option might be to always run the linter, but only if compilation fails due to a syntax error. I don't know if eslint-loader actually allows that or if it can be done some other way, but I think that's more helpful behavior that people are probably less likely to want to disable.

it halts compilation on any linting errors. [...] I've seen this behavior frustrate some as they didn't expect it

I think this feature would more acurately be named lintOnCompile. Had similar feedback for the webpack template about it being disruptive.

Another option might be to always run the linter, but only if compilation fails due to a syntax error.

Not too long ago I changed the eslint-loader settings in the webpack template to only emit warning, and webpack to only fail on errors (to accomodate those finding the original behavior disruptive).
Since syntax errors are catched by babel as well, this allows people to see errors in the console while the compilation does finish as expected and the workflow isn't disrupted.

@LinusBorg What has your feedback been since making that change? If we can give people useful warnings in the terminal without interrupting their workflow if they don't want to look at them, then we might not even need an option to disable lint-on-compile. Thoughts?

Well I have no received any complaints since we neither stop compiling nor show linting errors in the overlay, but can't say more than that one way or another.
Yet I would still feel we need some way to disable eslint-loader altogether because not everyone wants to lint during compilation, even if the results "only" clutter up their console. I added that as an option to the webpack template as well with those other changes.

What about allowing lintOnSave to take values such as 'error' and 'warning'?

@yyx990803 I like it! 😄

Agree


Add support for Wyze Cam Floodlight Pro

Is your feature request related to a problem? Please describe:
This feature is not related to a problem.
Describe the solution you'd like:
I would like to control the Wyze Cam Floodlight Pro using Homebridge in the same way the v2 and v3 cameras are controllable.  Specially, I want to turn the camera on and off.
Describe alternatives you've considered:
none
Additional context:
My v2 and v3 cameras are controllable using this plugin.  I would also like to control the floodlight camera.  The Homebridge logs show this:
[3/10/2024, 12:33:43 PM] [Wyze] [Camera] Unsupported device type: (Name: Floodlight Cam) (MAC: LD_CFP_zzzzzzzzzzzz) (Model: LD_CFP)
Please let me know if you need additional information.
-Matt

@jfarmer08, thanks for the help.  I added an entry for WyzeCamFloodlightPro to the enums.js file, and the camera is now controllable using Homebridge.
const CameraModels = {
  WyzeCamv1Hd: "WYZEC1",
  WyzeCamV2: "WYZEC1-JZ",
  WyzeCamV3: "WYZE_CAKP2JFUS",
  WyzeCamV3Pro: "HL_CAM3P",
  WyzeCamFloodlight: "WYZE_CAKP2JFUS",
  WyzeCamPan: "WYZECP1_JEF",
  WyzeCamPanv2: "HL_PAN2",
  WyzeCamPanv3: "HL_PAN3",
  WyzeCamOutdoor: "WVOD1",
  WyzeCamOutdoor2: "HL_WCO2",
  WyzeCamFloodlightPro: "LD_CFP"
}

Thanks for your help!
[4/19/2024, 3:44:37 PM] [Wyze] [Camera] [Privacy Switch] Retrieving previous service for LD_CFP_zzzzzzzzzzzz (Zzzz Cam)
[4/19/2024, 3:44:37 PM] [Wyze] [Camera] [Privacy Switch] Adding service for LD_CFP_zzzzzzzzzzzz (Zzzz Cam)

-Matt

Have you found any issues with this?

I was able to add the switch. To follow up, is it possible to add a switch for the flood light? Thank you @jfarmer08


joblib Parallel issue with specific complex '3m1s' in the PDBBind data

It could be a problem with RDKit. My version is "rdkit 2021.09.4".
import pickle
pdbbind_dir = "PDBBind_processed/"
name = '3m1s'
lig = read_molecule(os.path.join(pdbbind_dir, name, f'{name}_ligand.sdf'), sanitize=True,
                    remove_hs=True)
if lig == None:  # read mol2 file if sdf file cannot be sanitized
    lig = read_molecule(os.path.join(pdbbind_dir, name, f'{name}_ligand.mol2'), sanitize=True,
                        remove_hs=True)
# lig = Chem.MolFromSmiles('O=C[Ru+9]12345(C6=C1C2C3=C64)n1c2ccc(O)cc2c2c3c(c4ccc[n+]5c4c21)C(=O)NC3=O')
pickle.dump(lig, open("test.pkl", "bw"))
pickle.load(open("test.pkl", "rb"))

RuntimeError: invalid value in pickle

Hi, thanks for the issue!
Since you are directly writing and then reading the pickle again, I am rather sure that this problem is not related to the repository.
It seems like this is rather a general issue with pickle and your concrete setup.
Let me know if I am misunderstanding something!

Understood, thanks for the clarification.
However, the get_receptor function only reads the receptor files.
Can you maybe check if the ligand that is returned by rdkit is None?

yes, I checked by adding print('hello') right after "def get_receptor(rec_path, lig, cutoff):".
it will print 'hello' as expected until it reaches "3m1s".
so I think joblib Parallel want to pickle the input.and that causes the error.

I understand, but can you maybe check with the script that you wrote above if object that is returned by the function read_molecule is None?

it's not None. RDkit can read the mol2 file.

Okay, then it is not a problem with RDKit and the RDKit version I think.
Could you maybe let me know the full error message?
Usually it prints more than just the final joblib.externals.loky.process_executor._RemoteTraceback: ... RuntimeError: invalid value in pickle
and that information might be helpful

Sure.
Get receptors: 10340it [25:17, 15.49it/s]exception calling callback for <Future at 0x7fb5adb2ca90 state=finished raised BrokenProcessPool>
joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 407, in _process_worker
    call_item = call_queue.get(block=True, timeout=timeout)
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
RuntimeError: invalid value in pickle
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/externals/loky/_base.py", line 625, in _invoke_callbacks
    callback(self)
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/parallel.py", line 359, in __call__
    self.parallel.dispatch_next()
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/parallel.py", line 794, in dispatch_next
    if not self.dispatch_one_batch(self._original_iterator):
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/parallel.py", line 861, in dispatch_one_batch
    self._dispatch(tasks)
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/parallel.py", line 779, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 531, in apply_async
    future = self._workers.submit(SafeFunction(func))
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/externals/loky/reusable_executor.py", line 178, in submit
    fn, *args, **kwargs)
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 1115, in submit
    raise self._flags.broken
joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.
Get receptors: 10360it [25:32, 19.78it/s]joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 407, in _process_worker
    call_item = call_queue.get(block=True, timeout=timeout)
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
RuntimeError: invalid value in pickle
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "train.py", line 304, in <module>
    main_function()
  File "train.py", line 295, in main_function
    train_wrapper(args)
  File "train.py", line 141, in train_wrapper
    return train(args, run_dir)
  File "train.py", line 169, in train
    train_data = PDBBind(device=device, complex_names_path=args.train_names,lig_predictions_name=args.train_predictions_name, is_train_data=True, **args.dataset_params)
  File "/gxr/luwei/packages/EquiBind/datasets/pdbbind.py", line 128, in __init__
    self.process()
  File "/gxr/luwei/packages/EquiBind/datasets/pdbbind.py", line 271, in process
    receptor_representatives = pmap_multi(get_receptor, zip(rec_paths, ligs), n_jobs=self.n_jobs, cutoff=self.chain_radius, desc='Get receptors')
  File "/gxr/luwei/packages/EquiBind/commons/utils.py", line 47, in pmap_multi
    delayed(pickleable_fn)(*d, **kwargs) for i, d in tqdm(enumerate(data),desc=desc)
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/parallel.py", line 1056, in __call__
    self.retrieve()
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/parallel.py", line 935, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 542, in wrap_future_result
    return future.result(timeout=timeout)
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/externals/loky/_base.py", line 625, in _invoke_callbacks
    callback(self)
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/parallel.py", line 359, in __call__
    self.parallel.dispatch_next()
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/parallel.py", line 794, in dispatch_next
    if not self.dispatch_one_batch(self._original_iterator):
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/parallel.py", line 861, in dispatch_one_batch
    self._dispatch(tasks)
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/parallel.py", line 779, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 531, in apply_async
    future = self._workers.submit(SafeFunction(func))
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/externals/loky/reusable_executor.py", line 178, in submit
    fn, *args, **kwargs)
  File "/gxr/luwei/anaconda3/envs/equibind/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 1115, in submit
    raise self._flags.broken
joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.
Get receptors: 10399it [31:33,  5.49it/s]

Sorry, I am not sure what the cause is here. One thing that would definitely work is running the get_receptor function without joblib. (Which would require rewriting the line of code)
Also, I found some posts of users circumventing similar problems by using different versions of joblib.
Can you maybe try:
conda install -c anaconda joblib=1.1.0
conda install -c anaconda joblib=1.2.0

Thanks for the wonderful work!
I solved the problem by replacing following codes in process_mol.py:
coords = lig.GetConformer().GetPositions()==>coords = get_rdkit_coords(lig).numpy()
and removing some training complexes:
3zs1 3p3h 4jfv 4acu 3w8o 4mdn 2jld 3m1s 3p3j 4jfw 5tgy 4dcy 3p44 4i60 4jhq
and validation complexes:
4xkc 3p55 3rj7

Great to hear!
Thanks for the information that some versions have issues with the ligands in the mentioned complexes.


libmicrohttpd dependents: fix build

Motivation for this change
microhttpd 0.9.71 introduced a breaking change that broke xmr-stak (and possibly other packages, I will be checking others)
This PR updates the libmicrohttpd derivation to allow for having multiple versions of it in nixpkgs, and updates xmr-stak, fileshare, and faustlive to use 0.9.70, prior to the breaking change.
cpp-ethereum (now known as aleth) is also broken due to this - however, it requires libjon-rpc-cpp to be fixed as well.
Additionally, aleth is now abandon ware (https://gitter.im/ethereum/aleth?at=5e7cb22c2520175199ead0ba), it may be worthwhile to consider removing it since nothing seems to depend on it, and actually running it will not get you very far.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 105242 1

2 packages built:

libmicrohttpd_0_9_70
xmr-stak

Result of nixpkgs-review pr 105242 run on x86_64-linux 1

3 packages built:

libmicrohttpd_0_9_70
libmicrohttpd_0_9_71
xmr-stak

Result of nixpkgs-review pr 105242 run on x86_64-darwin 1

2 packages built:

libmicrohttpd_0_9_70
libmicrohttpd_0_9_71

Result of nixpkgs-review pr 105242 1

4 packages built:

faustlive
fileshare
libmicrohttpd_0_9_70
xmr-stak


Why do examples use plain hex representations instead of Base64Url?

The current two examples use plain hex representations of the SGX quote and EnclaveHeldData:

https://github.com/Azure-Samples/microsoft-azure-attestation/blob/e6038be4926c7a8ece81c7b0f1af8c34d5acb0e4/sgx.attest.sample/genquotes/host/quotefile.cpp#L47
https://github.com/Azure-Samples/microsoft-azure-attestation/blob/e6038be4926c7a8ece81c7b0f1af8c34d5acb0e4/intel.sdk.attest.sample/genquotes/host/host.cpp#L193

However, the MAA defines the Attestation Request's JSON fields as Base64Url encoded:

https://docs.microsoft.com/en-us/azure/attestation/basic-concepts#attestation-request

How can these examples work if they submit Attestation Requests with plain-hex values, even though MAA expects Base64Url-encoded values?

Ok, figured it out.
The "ValidateQuotes" utility program actually modifies the values in the JSON file: https://github.com/Azure-Samples/microsoft-azure-attestation/blob/b51c146d5246683abfc6253b4a82e5d0d52c28fb/intel.sdk.attest.sample/validatequotes.core/EnclaveInfo.cs#L25
And only then it sends the modified values (now encoded in Base64Url) to the MAA service.
Why was it implemented this way? Seems like an unnecessary step.

We encode the quote and data in https://github.com/Azure-Samples/microsoft-azure-attestation/blob/b51c146d5246683abfc6253b4a82e5d0d52c28fb/intel.sdk.attest.sample/validatequotes.core/EnclaveInfo.cs#L21-L29

I would like to re-use the validatequotes.core program for Graphene. This particular gimmick doesn't make sense to me. Could we do the following:

The examples are augmented to save QuoteHex and EnclaveHeldDataHex in the JSON file as base64Url.
The validatequotes.core program doesn't contain this additional logic of converting to base64Url.

Or am I missing some important reason to have this intermediate conversion?

Good point. I don't see a reason why you cannot save QuoteHex and EnclaveHeldDataHex in the JSON file as base64Url and remove the step from the validatequotes.core application.
The purpose of the samples is to demo basics of the quote generation, and validation with MAA service.
@gkostal could shed some light on whether there was a particular reason to separate encoding from writing to JSON.
My guess would be that it was easier to verify genquote_host output.

Given that we want to add support for MAA to Graphene, it would be great to agree on a particular format. We (Graphene developers) can go either way, but seems like Base64Url-encoded values in the JSON file is more straight-forward.
You can also check my draft PR for MAA support in Graphene: https://github.com/oscarlab/graphene/pull/1793.

Looking at https://github.com/oscarlab/graphene/pull/1793/files#diff-e85fb2d6fe3934933684754ee6a71705R32 gives me an idea about why encoding was implemented in the validatequotes.core.
Anyhow, it makes sense to write encoded values to JSON. I'd like to clarify some details before making any changes.
Is the plan to publish the Graphene sample into this repo? If yes, how do you want to lay out the samples? In other words, do you plan to use this app https://github.com/Azure-Samples/microsoft-azure-attestation/tree/master/intel.sdk.attest.sample/validatequotes.core or do you prefer to create a new folder similar to https://github.com/Azure-Samples/microsoft-azure-attestation/tree/master/intel.sdk.attest.sample or https://github.com/Azure-Samples/microsoft-azure-attestation/tree/master/sgx.attest.sample?

@dimakuv - the reason that the quote is hex formatted in the internal enclave info JSON file is simply because I was interested in the human readability of the enclave attribute fields in JSON file.  This file is an implementation detail for this sample since it separates the generation of a quote in one application (must be written in C/C++ due to OE SDK integration) and the verification of the quote with MAA (which originally was much simpler to write in C# due to the need to perform AAD authentication).
The MAA service no longer requires AAD authentication for making attest API calls.  Hence, if I were rewriting this sample app today, I'd combine both the genquote and validatequotes applications into one written in C/C++.
That said, I don't have any issue with switching the format of the strings in the enclave info JSON files to be base64url, so feel free to submit a PR with these changes if you like.
Closing this issue now since I believe the questions are all answered now.


Grad cam issue

Hello ,
base_model, preprocess_input = Classifiers.get('seresnext50')
model = base_model(input_shape=(512, 512, 20, 1 ), weights=None , include_top = False )
x = Flatten()(model.output)
x = Dense(1024, activation= 'sigmoid')(x)
x  = Dense(2, activation= 'sigmoid')(x)
Trying to train a model , the accuracy is everything resides upto expectation, but the gradcam are quite off from the region of the focus - how the accuracy is good but the grad cam is off the focus of targeted area .
ref -  for the gradcam generation code , the results are always at the border of the image.

https://github.com/fitushar/3D-Grad-CAM/blob/master/3DGrad-CAM.ipynb


hdinsight cluster in existing vnet using powershell

Hi All
below is the error i am getting while i use powershell script to launch hdinsight cluster in existing vnet using the below script :
$vnetName = "acctvnet"
$subnetName = "subnet2"
$resourceGroupName = "test"
$location = "West US"
$vnet = Get-AzVirtualNetwork     -Name $vnetName
-ResourceGroupName $resourceGroupName
$subnet = $vnet.Subnets | Where-Object Name -eq $subnetName
$defaultStorageAccountName = Read-Host -Prompt "Enter the name of the storage account"
Create an Az.Storage account and container
New-AzStorageAccount     -ResourceGroupName $resourceGroupName
-Name $defaultStorageAccountName     -Type Standard_LRS
-Location $location
$defaultStorageAccountKey = (Get-AzStorageAccountKey                                 -ResourceGroupName $resourceGroupName
-Name $defaultStorageAccountName)[0].Value
$defaultStorageContext = New-AzStorageContext                                 -StorageAccountName $defaultStorageAccountName
-StorageAccountKey $defaultStorageAccountKey
Get information for the HDInsight cluster
$clusterName = Read-Host -Prompt "Enter the name of the HDInsight cluster"
Cluster login is used to secure HTTPS services hosted on the cluster
$httpCredential = Get-Credential -Message "Enter Cluster login credentials" -UserName "admin"
SSH user is used to remotely connect to the cluster using SSH clients
$sshCredentials = Get-Credential -Message "Enter SSH user credentials"
Default cluster size (# of worker nodes), version, type, and OS
$clusterSizeInNodes = "1"
$clusterVersion = "3.6"
$clusterType = "Hadoop"
$clusterOS = "Linux"
Set the storage container name to the cluster name
$defaultBlobContainerName = $clusterName
Create a blob container. This holds the default data store for the cluster.
New-AzStorageContainer `
-Name $clusterName -Context $defaultStorageContext
Create the HDInsight cluster
 New-AzHDInsightCluster `
-ResourceGroupName $resourceGroupName `
-VirtualNetworkId  $vnet `
-SubnetName $subnetName `
-ClusterName $clusterName `
-Location $location `
-ClusterSizeInNodes $clusterSizeInNodes `
-ClusterType $clusterType `
-OSType $clusterOS `
-Version $clusterVersion `
-HttpCredential $httpCredential `
-DefaultStorageAccountName "$defaultStorageAccountName.blob.core.windows.net" `
-DefaultStorageAccountKey $defaultStorageAccountKey `
-DefaultStorageContainer $clusterName `
-SshCredential $sshCredentials

ERROR:
New-AzHDInsightCluster : Operation returned an invalid status code 'BadRequest'
At line:1 char:1

New-AzHDInsightCluster `

Since this issue is not related directly to the doc, I would recommend you to create a thread on the forums - [https://social.msdn.microsoft.com/Forums/en-US/home?forum=hdinsight](https://social.msdn.microsoft.com/Forums/en-US/home?forum=hdinsight
Once you post your issue on forums, it will have visibility across the community which is a better suited audience for such types of issues.


Validation errors with filevault-maven-plugin 1.2.2 in asset-share-commons.all-3.8.4.zip

When the asset-share-commons.all-3.8.4.zip package is embedded into a project which is using maven-filevault-plugin v1.2.2, the build is throwing multiple validation errors like:
[ERROR] ValidationViolation: "jackrabbit-docviewparser: Invalid XML found: Given root node name 'cq:design_dialog' (implicitly given via filename) cannot be resolved. The prefix used in the filename must be declared as XML namespace in the child docview XML as well!", filePath=jcr_root/apps/asset-share-commons/components/details/pdf/_cq_design_dialog/.content.xml, nodePath=/apps/asset-share-commons/components/details/pdf/cq:design_dialog
Other Error examples-
[ERROR] ValidationViolation: "jackrabbit-docviewparser: Invalid XML found: Given root node name 'cq:dialog' (implicitly given via filename) cannot be resolved. The prefix used in the filename must be declared as XML namespace in the child docview XML as well!", filePath=jcr_root/apps/asset-share-commons/components/details/pdf/_cq_dialog/.content.xml, nodePath=/apps/asset-share-commons/components/details/pdf/cq:dialog
[ERROR] ValidationViolation: "jackrabbit-docviewparser: Invalid XML found: Given root node name 'cq:dialog' (implicitly given via filename) cannot be resolved. The prefix used in the filename must be declared as XML namespace in the child docview XML as well!", filePath=jcr_root/apps/asset-share-commons/components/details/video/_cq_dialog/.content.xml, nodePath=/apps/asset-share-commons/components/details/video/cq:dialog
[ERROR] ValidationViolation: "jackrabbit-docviewparser: Invalid XML found: Given root node name 'cq:template' (implicitly given via filename) cannot be resolved. The prefix used in the filename must be declared as XML namespace in the child docview XML as well!", filePath=jcr_root/apps/asset-share-commons/components/modals/download/_cq_template/.content.xml, nodePath=/apps/asset-share-commons/components/modals/download/cq:template
[ERROR] ValidationViolation: "jackrabbit-docviewparser: Invalid XML found: Given root node name 'cq:dialog' (implicitly given via filename) cannot be resolved. The prefix used in the filename must be declared as XML namespace in the child docview XML as well!", filePath=jcr_root/apps/asset-share-commons/components/modals/share/_cq_dialog/.content.xml, nodePath=/apps/asset-share-commons/components/modals/share/cq:dialog
[ERROR] ValidationViolation: "jackrabbit-docviewparser: Invalid XML found: Given root node name 'cq:dialog' (implicitly given via filename) cannot be resolved. The prefix used in the filename must be declared as XML namespace in the child docview XML as well!", filePath=jcr_root/apps/asset-share-commons/components/search/common/_cq_dialog/.content.xml, nodePath=/apps/asset-share-commons/components/search/common/cq:dialog
[ERROR] ValidationViolation: "jackrabbit-docviewparser: Invalid XML found: Given root node name 'cq:dialog' (implicitly given via filename) cannot be resolved. The prefix used in the filename must be declared as XML namespace in the child docview XML as well!", filePath=jcr_root/apps/asset-share-commons/components/search/date-range/_cq_dialog/.content.xml, nodePath=/apps/asset-share-commons/components/search/date-range/cq:dialog
[ERROR] ValidationViolation: "jackrabbit-docviewparser: Invalid XML found: Given root node name 'cq:dialog' (implicitly given via filename) cannot be resolved. The prefix used in the filename must be declared as XML namespace in the child docview XML as well!", filePath=jcr_root/apps/asset-share-commons/components/search/freeform-text/_cq_dialog/.content.xml, nodePath=/apps/asset-share-commons/components/search/freeform-text/cq:dialog
[ERROR] ValidationViolation: "jackrabbit-docviewparser: Invalid XML found: Given root node name 'cq:dialog' (implicitly given via filename) cannot be resolved. The prefix used in the filename must be declared as XML namespace in the child docview XML as well!", filePath=jcr_root/apps/asset-share-commons/components/search/path/_cq_dialog/.content.xml, nodePath=/apps/asset-share-commons/components/search/path/cq:dialog
[ERROR] ValidationViolation: "jackrabbit-docviewparser: Invalid XML found: Given root node name 'cq:dialog' (implicitly given via filename) cannot be resolved. The prefix used in the filename must be declared as XML namespace in the child docview XML as well!", filePath=jcr_root/apps/asset-share-commons/components/search/property/_cq_dialog/.content.xml, nodePath=/apps/asset-share-commons/components/search/property/cq:dialog
[ERROR] ValidationViolation: "jackrabbit-docviewparser: Invalid XML found: Given root node name 'cq:dialog' (implicitly given via filename) cannot be resolved. The prefix used in the filename must be declared as XML namespace in the child docview XML as well!", filePath=jcr_root/apps/asset-share-commons/components/search/sort/_cq_dialog/.content.xml, nodePath=/apps/asset-share-commons/components/search/sort/cq:dialog
[ERROR] ValidationViolation: "jackrabbit-docviewparser: Invalid XML found: Given root node name 'cq:dialog' (implicitly given via filename) cannot be resolved. The prefix used in the filename must be declared as XML namespace in the child docview XML as well!", filePath=jcr_root/apps/asset-share-commons/components/search/tags/_cq_dialog/.content.xml, nodePath=/apps/asset-share-commons/components/search/tags/cq:dialog

Resolved temporarily with
<skipSubPackageValidation>true</skipSubPackageValidation>


Identifiers, names, refs, nightmare docs, logging hell level 999

Why do you need both identifiers and names? Is one of them not enough to identify the appender at logger definition?
for example: appender.console.name = STDOUT
I know my english is terrible, but this is the most incomprehensible documentation I have ever read:
https://logging.apache.org/log4j/2.x/manual/configuration.html#Properties
It is full of ambiguous statements. Brain surgery is easier to learn than log4j documentation. It has more information than in the Bible.  It is not easy to use at all compared to version 1, what worked really well! Unnecessarily overcomplicated.

The Log4j configuration is a tree structure organized as "Nodes" where a Node is nothing but a container of attributes and child Nodes. All the configuration dialects Log4j supports are converted into a Node tree. This works great for Yaml, JSON, and XML. Unfortunately, by their nature properties are not hierarchical. This means using properties is somewhat fitting a round peg in a square hole. For years Log4j didn't support properties due to this but we finally added it due to user's requests.  We have made slight "improvements" over time at user's requests but in the end I really don't think it makes it any simpler.
I should note that the way Log4j 1 handled configuration was completely different. Properties were the original and only format. XML was bolted on after the fact and was basically a different configuration system from using properties.
I, for one, highly encourage users to use a different one of the other formats.

Dear @rgoers,
Thank you very much for your quick answer! You are really kind!
Best Regards: Bela

@robozb,
The identifiers in the properties format are almost useless. One of the last attempts to make the properties format more digestible (#733 and related issues) proposed to use them as default appender names, but the feature never made it out to the main code.
Such a usage would be in line with what Log4j 1.x did, but is not coherent with other formats: most formats use the identifiers/keys/element names to indicate the plugin type. You can see it in Ralph's YAML example:
 appenders:
   Console:
     name: STDOUT
     PatternLayout:
       Pattern: "%m%n"
    File:
      name: FILE
      fileName: ${filename}
      bufferedIO: false
      PatternLayout:
        Pattern: "%d %p %C{1.} [%t] %m%n"

The keys Console, File and PatternLayout associated to YAML objects are used as default values for the missing type properties.
We could probably implement something similar in the properties format:
appender.Console.name = STDOUT
appender.Console.PatternLayout.pattern = "%m%n"

appender.File.name = FILE
appender.File.fileName = ${filename}
appender.File.bufferedIO = false
appender.File.PatternLayout.pattern = %d %p %C{1.} [%t] %m%n

However there isn't much to be done if multiple appenders of the same type are required. In the YAML format we currently support the following shorthand:
Appenders:
  File:
    - name: FILE1
      fileName: file1.log
    - name: FILE2
      fileName: file2.log

(i.e. for YAML arrays the default type is the key associated to the array). I don't think anything similar would be possible for the properties format.

Dear @ppkarwasz,
Thank you for the details!
Regards: Bela


MP:	protein import into mitochondrial outer membrane

protein import into mitochondrial outer membrane
is defined
Definition | Parents
The process comprising the insertion of proteins from outside the organelle into the mitochondrial outer membrane, mediated by large outer membrane translocase complexes.
should this be "mitochondrial outer membrane organization"?

Makes sense to me.

I added the missing parentage but have a question for @pgaudet about the logical definitions for protein import into outer mitochondrial membrane vs protein import into inner mitochondrial membrane:
protein import into outer mitochondrial membrane:
transport
and ('occurs in' some cytoplasm)
and ('has target end location' some 'mitochondrial outer membrane')
and (imports some protein)
protein import into inner mitochondrial membrane
transport
and ('has target start location' some cytosol)
and ('has target end location' some 'mitochondrial inner membrane')
and (imports some protein)
and ('results in transport across' some 'mitochondrial outer membrane')
The definition for import into the outer membrane does not have a 'has target start location' in the logical def, but rather an 'occurs in' relation to the cytoplasm.  Is that intended?

pinging @pgaudet

Thanks for the reminder.
Note that "protein import into mitochondrial outer membrane" has been renamed "protein insertion into mitochondrial outer membrane".
There is no start location because the protein can technically come from the mitochondrial inner membrane OR from the cytoplasm. I think we had looked at this together. The paper cited shows nice diagrams:
https://www.sciencedirect.com/science/article/pii/S0167488908002474?via%3Dihub
If I recall correctly some proteins are imported from the cytoplasms into the intermembrane space, and then inserted into the outer membrane, while others are inserted directly.
However if all proteins are encoded in the nucleus, I think the logical definition would be OK with 'has start location = cytosol' even if the protein the transits through the intermembrane space.
Pascale

OK!

I dont know that all mitochondrial outer membrane proteins are encoded in the nucleus in all species, so I would prefer to leave the logical definition as it is, unless you think it should be changed ?

I'm pretty sure that they are. The mitochondrial geneome complement appears to be fairly constant (yeast to human), ~13 proteins  and only encoded etc components and an endonuclease.
So I think you could change it...

etc =electron transport chain

I thought some species had about 40 genes?
I'll add the relation, if it's causing problems we'll hear.

Yeah but I'm  with Steve and he says not (he did a lot of work on mitochondria  in the past)
I think the 40 would include rRNA and tRNA...

ok perfect then !

Also added
protein insertion into mitochondrial membrane
transport
and ('has target start location' some cytosol)
and ('transports or maintains localization of' some protein)
and ('has target end location' some 'mitochondrial membrane')
And changed the regulation terms to be consistent:
-name: regulation of protein import into mitochondrial outer membrane
+name: regulation of protein insertion into mitochondrial outer membrane
(plus the positive and negative regulation terms)
Thanks, Pascale


Expecting value: line 1 column 1 (char 0)

I encountered this problem when I configured the openai key according to the documentation, and until then everything was done according to the documentation
In the previous step, I ran run.py

Can you see any errors displayed in the console

发件人: jiangzhiyu1016 @.>
发送时间: 2023年4月18日 5:06
收件人: newfyu/Braindoor @.>
抄送: Subscribed @.***>
主题: [newfyu/Braindoor] Expecting value: line 1 column 1 (char 0) (Issue #1)
[image]https://user-images.githubusercontent.com/56686900/232675974-099ac4ed-45a4-4a45-aa65-dcf5b256fe63.png
I encountered this problem when I configured the openai key according to the documentation, and until then everything was done according to the documentation
In the previous step, I ran run.py
―
Reply to this email directly, view it on GitHubhttps://github.com/newfyu/Braindoor/issues/1, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKJ2EPSO4M7RA7MIOSTQDQTXBYOLTANCNFSM6AAAAAAXCCTVVU.
You are receiving this because you are subscribed to this thread.Message ID: @.***>

I have tested on multiple Windows computers but have not been successful

非常抱歉，windows上的测试可能不太充分。最近在做一次大的改动，会有一个全新的桌面UI（如下）。完成后会我再到window上测试，也会提供一个打包好环境直接运行的文件。谢谢你的关注和测试，但还需要一点时间，基本完成后再麻烦你测试。
￼

2023年4月22日 13:15，jiangzhiyu1016 @.***> 写道：
I have tested on multiple Windows computers but have not been successful
—
Reply to this email directly, view it on GitHub https://github.com/newfyu/Braindoor/issues/1#issuecomment-1518514113, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKJ2EPXDOIPMK7OCNIUD323XCNSNVANCNFSM6AAAAAAXCCTVVU.
You are receiving this because you commented.


Scipy build fails on Solaris 10 SPARC

I have a working copy of Python 3.65 and Numpy 1.14.5 in /usr/local/Python3 on Solaris 10.
-bash-3.2# uname -a
SunOS python365 5.10 Generic_150400-17 sun4u sparc SUNW,SPARC-Enterprise Solaris
Relevant environment variables are as follows:
ATLAS=/usr/local/Python3/ATLAS/install/lib
MAIL=/var/mail/root
PATH=/usr/local/Python3/Python-3.6.5/install/bin:/opt/csw/bin:/opt/csw/gnu:/usr/sbin:/usr/bin:/usr/local/bin
BLAS=/usr/local/Python3/ATLAS/install/lib
LAPACK=/usr/local/Python3/ATLAS/install/lib
LAPACK_SRC=/usr/local/Python3/ATLAS/install/lib
PYTHONPATH=/usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages
I downloaded the Scipy source and extracted it into /usr/local/Python3/scipy-1.1.0
I type "python3 setup.py install" which fails with the following message after about 30 minutes:

### Error message:
<!--

compiling C++ sources
C compiler: g++ -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC

compile options: '-I/usr/local/Python3/scipy-1.1.0/scipy/special -I/usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include -I/usr/local/Python3/Python-3.6.5/install/include/python3.6m -I/usr/local/Python3/scipy-1.1.0/scipy/special/c_misc -Iscipy/_lib -I/usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include -I/usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include -I/usr/local/Python3/Python-3.6.5/install/include/python3.6m -c'
g++: scipy/special/_ufuncs_cxx.cxx
In file included from /usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include/numpy/ndarraytypes.h:1816:0,
                 from /usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include/numpy/arrayobject.h:4,
                 from scipy/special/_ufuncs_cxx.cxx:583:
/usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^
g++: scipy/special/_faddeeva.cxx
In file included from /usr/local/Python3/Python-3.6.5/install/include/python3.6m/Python.h:8:0,
                 from scipy/special/_faddeeva.h:13,
                 from scipy/special/_faddeeva.cxx:3:
/usr/local/Python3/Python-3.6.5/install/include/python3.6m/pyconfig.h:1456:0: warning: "_FILE_OFFSET_BITS" redefined
 #define _FILE_OFFSET_BITS 64
 ^
In file included from /opt/csw/lib/gcc/sparc-sun-solaris2.10/5.5.0/include-fixed/iso/math_iso.h:19:0,
                 from /opt/csw/lib/gcc/sparc-sun-solaris2.10/5.5.0/include-fixed/math.h:22,
                 from /opt/csw/include/c++/5.5.0/cmath:44,
                 from /opt/csw/include/c++/5.5.0/complex:44,
                 from scipy/special/_faddeeva.cxx:1:
/opt/csw/lib/gcc/sparc-sun-solaris2.10/5.5.0/include-fixed/sys/feature_tests.h:196:0: note: this is the location of the previous definition
 #define _FILE_OFFSET_BITS 32
 ^
g++: scipy/special/Faddeeva.cc
In file included from /usr/local/Python3/Python-3.6.5/install/include/python3.6m/Python.h:8:0,
                 from scipy/special/Faddeeva.cc:120:
/usr/local/Python3/Python-3.6.5/install/include/python3.6m/pyconfig.h:1456:0: warning: "_FILE_OFFSET_BITS" redefined
 #define _FILE_OFFSET_BITS 64
 ^
In file included from /opt/csw/lib/gcc/sparc-sun-solaris2.10/5.5.0/include-fixed/iso/math_iso.h:19:0,
                 from /opt/csw/lib/gcc/sparc-sun-solaris2.10/5.5.0/include-fixed/math.h:22,
                 from /opt/csw/include/c++/5.5.0/cmath:44,
                 from /opt/csw/include/c++/5.5.0/complex:44,
                 from scipy/special/Faddeeva.hh:30,
                 from scipy/special/Faddeeva.cc:23:
/opt/csw/lib/gcc/sparc-sun-solaris2.10/5.5.0/include-fixed/sys/feature_tests.h:196:0: note: this is the location of the previous definition
 #define _FILE_OFFSET_BITS 32
 ^
g++: scipy/special/_wright.cxx
In file included from /usr/local/Python3/Python-3.6.5/install/include/python3.6m/Python.h:8:0,
                 from scipy/special/_wright.h:13,
                 from scipy/special/_wright.cxx:3:
/usr/local/Python3/Python-3.6.5/install/include/python3.6m/pyconfig.h:1456:0: warning: "_FILE_OFFSET_BITS" redefined
 #define _FILE_OFFSET_BITS 64
 ^
In file included from /opt/csw/lib/gcc/sparc-sun-solaris2.10/5.5.0/include-fixed/iso/math_iso.h:19:0,
                 from /opt/csw/lib/gcc/sparc-sun-solaris2.10/5.5.0/include-fixed/math.h:22,
                 from /opt/csw/include/c++/5.5.0/cmath:44,
                 from /opt/csw/include/c++/5.5.0/complex:44,
                 from scipy/special/_wright.cxx:1:
/opt/csw/lib/gcc/sparc-sun-solaris2.10/5.5.0/include-fixed/sys/feature_tests.h:196:0: note: this is the location of the previous definition
 #define _FILE_OFFSET_BITS 32
 ^
g++: scipy/special/wright.cc
In file included from /usr/local/Python3/Python-3.6.5/install/include/python3.6m/Python.h:8:0,
                 from scipy/special/wright.cc:75:
/usr/local/Python3/Python-3.6.5/install/include/python3.6m/pyconfig.h:1456:0: warning: "_FILE_OFFSET_BITS" redefined
 #define _FILE_OFFSET_BITS 64
 ^
In file included from /opt/csw/lib/gcc/sparc-sun-solaris2.10/5.5.0/include-fixed/iso/math_iso.h:19:0,
                 from /opt/csw/lib/gcc/sparc-sun-solaris2.10/5.5.0/include-fixed/math.h:22,
                 from /opt/csw/include/c++/5.5.0/cmath:44,
                 from /opt/csw/include/c++/5.5.0/complex:44,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/lib/gcc/sparc-sun-solaris2.10/5.5.0/include-fixed/sys/feature_tests.h:196:0: note: this is the location of the previous definition
 #define _FILE_OFFSET_BITS 32
 ^
In file included from scipy/special/wright.cc:80:0:
scipy/special/_round.h: In function âdouble add_round_up(double, double)â:
scipy/special/_round.h:17:19: error: âisnanâ was not declared in this scope
     if (sc_isnan(a) || sc_isnan(b)) {
                   ^
scipy/special/_round.h:17:19: note: suggested alternative:
In file included from /opt/csw/include/c++/5.5.0/complex:44:0,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/include/c++/5.5.0/cmath:886:5: note:   âstd::isnanâ
     isnan(_Tp __f)
     ^
In file included from scipy/special/wright.cc:80:0:
scipy/special/_round.h: In function âdouble add_round_down(double, double)â:
scipy/special/_round.h:37:19: error: âisnanâ was not declared in this scope
     if (sc_isnan(a) || sc_isnan(b)) {
                   ^
scipy/special/_round.h:37:19: note: suggested alternative:
In file included from /opt/csw/include/c++/5.5.0/complex:44:0,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/include/c++/5.5.0/cmath:886:5: note:   âstd::isnanâ
     isnan(_Tp __f)
     ^
scipy/special/wright.cc: In function âint wright::wrightomega_ext(std::complex<double>, std::complex<double>*, std::complex<double>*)â:
scipy/special/wright.cc:116:16: error: âisnanâ was not declared in this scope
   if(sc_isnan(x) || sc_isnan(y))
                ^
scipy/special/wright.cc:116:16: note: suggested alternative:
In file included from /opt/csw/include/c++/5.5.0/complex:44:0,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/include/c++/5.5.0/cmath:886:5: note:   âstd::isnanâ
     isnan(_Tp __f)
     ^
scipy/special/wright.cc:124:21: error: âisinfâ was not declared in this scope
   else if(sc_isinf(x) && (x < 0.0) && (-pi < y) && (y<= pi))
                     ^
scipy/special/wright.cc:124:21: note: suggested alternative:
In file included from /opt/csw/include/c++/5.5.0/complex:44:0,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/include/c++/5.5.0/cmath:877:5: note:   âstd::isinfâ
     isinf(_Tp __f)
     ^
In file included from scipy/special/_round.h:10:0,
                 from scipy/special/wright.cc:80:
scipy/special/cephes/dd_idefs.h: At global scope:
scipy/special/cephes/dd_idefs.h:53:1: warning: âdouble quick_two_diff(double, double, double*)â defined but not used [-Wunused-function]
 quick_two_diff(double a, double b, double *err)
 ^
scipy/special/cephes/dd_idefs.h:130:1: warning: âdouble two_sqr(double, double*)â defined but not used [-Wunused-function]
 two_sqr(double a, double *err)
 ^
scipy/special/cephes/dd_idefs.h:148:1: warning: âdouble two_div(double, double, double*)â defined but not used [-Wunused-function]
 two_div(double a, double b, double *err)
 ^
scipy/special/cephes/dd_idefs.h:169:1: warning: âdouble two_nint(double)â defined but not used [-Wunused-function]
 two_nint(double d)
 ^
scipy/special/cephes/dd_idefs.h:179:1: warning: âdouble two_aint(double)â defined but not used [-Wunused-function]
 two_aint(double d)
 ^
scipy/special/cephes/dd_idefs.h:187:1: warning: âint two_comp(double, double)â defined but not used [-Wunused-function]
 two_comp(const double a, const double b)
 ^
In file included from /usr/local/Python3/Python-3.6.5/install/include/python3.6m/Python.h:8:0,
                 from scipy/special/wright.cc:75:
/usr/local/Python3/Python-3.6.5/install/include/python3.6m/pyconfig.h:1456:0: warning: "_FILE_OFFSET_BITS" redefined
 #define _FILE_OFFSET_BITS 64
 ^
In file included from /opt/csw/lib/gcc/sparc-sun-solaris2.10/5.5.0/include-fixed/iso/math_iso.h:19:0,
                 from /opt/csw/lib/gcc/sparc-sun-solaris2.10/5.5.0/include-fixed/math.h:22,
                 from /opt/csw/include/c++/5.5.0/cmath:44,
                 from /opt/csw/include/c++/5.5.0/complex:44,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/lib/gcc/sparc-sun-solaris2.10/5.5.0/include-fixed/sys/feature_tests.h:196:0: note: this is the location of the previous definition
 #define _FILE_OFFSET_BITS 32
 ^
In file included from scipy/special/wright.cc:80:0:
scipy/special/_round.h: In function âdouble add_round_up(double, double)â:
scipy/special/_round.h:17:19: error: âisnanâ was not declared in this scope
     if (sc_isnan(a) || sc_isnan(b)) {
                   ^
scipy/special/_round.h:17:19: note: suggested alternative:
In file included from /opt/csw/include/c++/5.5.0/complex:44:0,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/include/c++/5.5.0/cmath:886:5: note:   âstd::isnanâ
     isnan(_Tp __f)
     ^
In file included from scipy/special/wright.cc:80:0:
scipy/special/_round.h: In function âdouble add_round_down(double, double)â:
scipy/special/_round.h:37:19: error: âisnanâ was not declared in this scope
     if (sc_isnan(a) || sc_isnan(b)) {
                   ^
scipy/special/_round.h:37:19: note: suggested alternative:
In file included from /opt/csw/include/c++/5.5.0/complex:44:0,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/include/c++/5.5.0/cmath:886:5: note:   âstd::isnanâ
     isnan(_Tp __f)
     ^
scipy/special/wright.cc: In function âint wright::wrightomega_ext(std::complex<double>, std::complex<double>*, std::complex<double>*)â:
scipy/special/wright.cc:116:16: error: âisnanâ was not declared in this scope
   if(sc_isnan(x) || sc_isnan(y))
                ^
scipy/special/wright.cc:116:16: note: suggested alternative:
In file included from /opt/csw/include/c++/5.5.0/complex:44:0,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/include/c++/5.5.0/cmath:886:5: note:   âstd::isnanâ
     isnan(_Tp __f)
     ^
scipy/special/wright.cc:124:21: error: âisinfâ was not declared in this scope
   else if(sc_isinf(x) && (x < 0.0) && (-pi < y) && (y<= pi))
                     ^
scipy/special/wright.cc:124:21: note: suggested alternative:
In file included from /opt/csw/include/c++/5.5.0/complex:44:0,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/include/c++/5.5.0/cmath:877:5: note:   âstd::isinfâ
     isinf(_Tp __f)
     ^
In file included from scipy/special/_round.h:10:0,
                 from scipy/special/wright.cc:80:
scipy/special/cephes/dd_idefs.h: At global scope:
scipy/special/cephes/dd_idefs.h:53:1: warning: âdouble quick_two_diff(double, double, double*)â defined but not used [-Wunused-function]
 quick_two_diff(double a, double b, double *err)
 ^
scipy/special/cephes/dd_idefs.h:130:1: warning: âdouble two_sqr(double, double*)â defined but not used [-Wunused-function]
 two_sqr(double a, double *err)
 ^
scipy/special/cephes/dd_idefs.h:148:1: warning: âdouble two_div(double, double, double*)â defined but not used [-Wunused-function]
 two_div(double a, double b, double *err)
 ^
scipy/special/cephes/dd_idefs.h:169:1: warning: âdouble two_nint(double)â defined but not used [-Wunused-function]
 two_nint(double d)
 ^
scipy/special/cephes/dd_idefs.h:179:1: warning: âdouble two_aint(double)â defined but not used [-Wunused-function]
 two_aint(double d)
 ^
scipy/special/cephes/dd_idefs.h:187:1: warning: âint two_comp(double, double)â defined but not used [-Wunused-function]
 two_comp(const double a, const double b)
 ^
error: Command "g++ -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/Python3/scipy-1.1.0/scipy/special -I/usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include -I/usr/local/Python3/Python-3.6.5/install/include/python3.6m -I/usr/local/Python3/scipy-1.1.0/scipy/special/c_misc -Iscipy/_lib -I/usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include -I/usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include -I/usr/local/Python3/Python-3.6.5/install/include/python3.6m -c scipy/special/wright.cc -o build/temp.solaris-2.10-sun4u.32bit-3.6/scipy/special/wright.o -MMD -MF build/temp.solaris-2.10-sun4u.32bit-3.6/scipy/special/wright.o.d" failed with exit status 1

I'm just shooting in the dark here but can you also try to compile against Openblas if you have the means?

It's some problem with the isinf/isnan/etc macros in c++ code, unlikely to be lapack related.

@rgommers
I'll try GCC 4.9.2 today.  I actually was able to install everything successfully using Sun compilers, but I received numerous import errors with all but two of the Scipy extensions.  I read that this can happen if you use different Fortran compilers, which I did.  I built ATLAS/LAPACK with gfortran and Scipy with Sun Fortran.  I'm trying to go GCC all the way this time.

Indeed, you never want to mix Fortran compilers

### Error message:
<!--
compiling C++ sources
C compiler: g++ -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC

compile options: '-I/usr/local/Python3/scipy-1.1.0/scipy/special -I/usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include -I/usr/local/Python3/Python-3.6.5/install/include/python3.6m -I/usr/local/Python3/scipy-1.1.0/scipy/special/c_misc -Iscipy/_lib -I/usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include -I/usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include -I/usr/local/Python3/Python-3.6.5/install/include/python3.6m -c'
g++: scipy/special/_ufuncs_cxx.cxx
In file included from /usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include/numpy/ndarraytypes.h:1816:0,
                 from /usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include/numpy/arrayobject.h:4,
                 from scipy/special/_ufuncs_cxx.cxx:583:
/usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^
g++: scipy/special/_faddeeva.cxx
In file included from /usr/local/Python3/Python-3.6.5/install/include/python3.6m/Python.h:8:0,
                 from scipy/special/_faddeeva.h:13,
                 from scipy/special/_faddeeva.cxx:3:
/usr/local/Python3/Python-3.6.5/install/include/python3.6m/pyconfig.h:1456:0: warning: "_FILE_OFFSET_BITS" redefined
 #define _FILE_OFFSET_BITS 64
 ^
In file included from /opt/csw/lib/gcc/sparc-sun-solaris2.10/4.9.2/include-fixed/iso/math_iso.h:20:0,
                 from /opt/csw/lib/gcc/sparc-sun-solaris2.10/4.9.2/include-fixed/math.h:26,
                 from /opt/csw/include/c++/4.9.2/cmath:44,
                 from /opt/csw/include/c++/4.9.2/complex:44,
                 from scipy/special/_faddeeva.cxx:1:
/opt/csw/lib/gcc/sparc-sun-solaris2.10/4.9.2/include-fixed/sys/feature_tests.h:196:0: note: this is the location of the previous definition
 #define _FILE_OFFSET_BITS 32
 ^
g++: scipy/special/Faddeeva.cc
In file included from /usr/local/Python3/Python-3.6.5/install/include/python3.6m/Python.h:8:0,
                 from scipy/special/Faddeeva.cc:120:
/usr/local/Python3/Python-3.6.5/install/include/python3.6m/pyconfig.h:1456:0: warning: "_FILE_OFFSET_BITS" redefined
 #define _FILE_OFFSET_BITS 64
 ^
In file included from /opt/csw/lib/gcc/sparc-sun-solaris2.10/4.9.2/include-fixed/iso/math_iso.h:20:0,
                 from /opt/csw/lib/gcc/sparc-sun-solaris2.10/4.9.2/include-fixed/math.h:26,
                 from /opt/csw/include/c++/4.9.2/cmath:44,
                 from /opt/csw/include/c++/4.9.2/complex:44,
                 from scipy/special/Faddeeva.hh:30,
                 from scipy/special/Faddeeva.cc:23:
/opt/csw/lib/gcc/sparc-sun-solaris2.10/4.9.2/include-fixed/sys/feature_tests.h:196:0: note: this is the location of the previous definition
 #define _FILE_OFFSET_BITS 32
 ^
g++: scipy/special/_wright.cxx
In file included from /usr/local/Python3/Python-3.6.5/install/include/python3.6m/Python.h:8:0,
                 from scipy/special/_wright.h:13,
                 from scipy/special/_wright.cxx:3:
/usr/local/Python3/Python-3.6.5/install/include/python3.6m/pyconfig.h:1456:0: warning: "_FILE_OFFSET_BITS" redefined
 #define _FILE_OFFSET_BITS 64
 ^
In file included from /opt/csw/lib/gcc/sparc-sun-solaris2.10/4.9.2/include-fixed/iso/math_iso.h:20:0,
                 from /opt/csw/lib/gcc/sparc-sun-solaris2.10/4.9.2/include-fixed/math.h:26,
                 from /opt/csw/include/c++/4.9.2/cmath:44,
                 from /opt/csw/include/c++/4.9.2/complex:44,
                 from scipy/special/_wright.cxx:1:
/opt/csw/lib/gcc/sparc-sun-solaris2.10/4.9.2/include-fixed/sys/feature_tests.h:196:0: note: this is the location of the previous definition
 #define _FILE_OFFSET_BITS 32
 ^
g++: scipy/special/wright.cc
In file included from /usr/local/Python3/Python-3.6.5/install/include/python3.6m/Python.h:8:0,
                 from scipy/special/wright.cc:75:
/usr/local/Python3/Python-3.6.5/install/include/python3.6m/pyconfig.h:1456:0: warning: "_FILE_OFFSET_BITS" redefined
 #define _FILE_OFFSET_BITS 64
 ^
In file included from /opt/csw/lib/gcc/sparc-sun-solaris2.10/4.9.2/include-fixed/iso/math_iso.h:20:0,
                 from /opt/csw/lib/gcc/sparc-sun-solaris2.10/4.9.2/include-fixed/math.h:26,
                 from /opt/csw/include/c++/4.9.2/cmath:44,
                 from /opt/csw/include/c++/4.9.2/complex:44,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/lib/gcc/sparc-sun-solaris2.10/4.9.2/include-fixed/sys/feature_tests.h:196:0: note: this is the location of the previous definition
 #define _FILE_OFFSET_BITS 32
 ^
In file included from scipy/special/wright.cc:80:0:
scipy/special/_round.h: In function âdouble add_round_up(double, double)â:
scipy/special/_round.h:17:19: error: âisnanâ was not declared in this scope
     if (sc_isnan(a) || sc_isnan(b)) {
                   ^
scipy/special/_round.h:17:19: note: suggested alternative:
In file included from /opt/csw/include/c++/4.9.2/complex:44:0,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/include/c++/4.9.2/cmath:837:5: note:   âstd::isnanâ
     isnan(_Tp __f)
     ^
In file included from scipy/special/wright.cc:80:0:
scipy/special/_round.h: In function âdouble add_round_down(double, double)â:
scipy/special/_round.h:37:19: error: âisnanâ was not declared in this scope
     if (sc_isnan(a) || sc_isnan(b)) {
                   ^
scipy/special/_round.h:37:19: note: suggested alternative:
In file included from /opt/csw/include/c++/4.9.2/complex:44:0,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/include/c++/4.9.2/cmath:837:5: note:   âstd::isnanâ
     isnan(_Tp __f)
     ^
scipy/special/wright.cc: In function âint wright::wrightomega_ext(std::complex<double>, std::complex<double>*, std::complex<double>*)â:
scipy/special/wright.cc:116:16: error: âisnanâ was not declared in this scope
   if(sc_isnan(x) || sc_isnan(y))
                ^
scipy/special/wright.cc:116:16: note: suggested alternative:
In file included from /opt/csw/include/c++/4.9.2/complex:44:0,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/include/c++/4.9.2/cmath:837:5: note:   âstd::isnanâ
     isnan(_Tp __f)
     ^
scipy/special/wright.cc:124:21: error: âisinfâ was not declared in this scope
   else if(sc_isinf(x) && (x < 0.0) && (-pi < y) && (y<= pi))
                     ^
scipy/special/wright.cc:124:21: note: suggested alternative:
In file included from /opt/csw/include/c++/4.9.2/complex:44:0,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/include/c++/4.9.2/cmath:828:5: note:   âstd::isinfâ
     isinf(_Tp __f)
     ^
In file included from scipy/special/_round.h:10:0,
                 from scipy/special/wright.cc:80:
scipy/special/cephes/dd_idefs.h: At global scope:
scipy/special/cephes/dd_idefs.h:53:1: warning: âdouble quick_two_diff(double, double, double*)â defined but not used [-Wunused-function]
 quick_two_diff(double a, double b, double *err)
 ^
scipy/special/cephes/dd_idefs.h:130:1: warning: âdouble two_sqr(double, double*)â defined but not used [-Wunused-function]
 two_sqr(double a, double *err)
 ^
scipy/special/cephes/dd_idefs.h:148:1: warning: âdouble two_div(double, double, double*)â defined but not used [-Wunused-function]
 two_div(double a, double b, double *err)
 ^
scipy/special/cephes/dd_idefs.h:169:1: warning: âdouble two_nint(double)â defined but not used [-Wunused-function]
 two_nint(double d)
 ^
scipy/special/cephes/dd_idefs.h:179:1: warning: âdouble two_aint(double)â defined but not used [-Wunused-function]
 two_aint(double d)
 ^
scipy/special/cephes/dd_idefs.h:187:1: warning: âint two_comp(double, double)â defined but not used [-Wunused-function]
 two_comp(const double a, const double b)
 ^
In file included from /usr/local/Python3/Python-3.6.5/install/include/python3.6m/Python.h:8:0,
                 from scipy/special/wright.cc:75:
/usr/local/Python3/Python-3.6.5/install/include/python3.6m/pyconfig.h:1456:0: warning: "_FILE_OFFSET_BITS" redefined
 #define _FILE_OFFSET_BITS 64
 ^
In file included from /opt/csw/lib/gcc/sparc-sun-solaris2.10/4.9.2/include-fixed/iso/math_iso.h:20:0,
                 from /opt/csw/lib/gcc/sparc-sun-solaris2.10/4.9.2/include-fixed/math.h:26,
                 from /opt/csw/include/c++/4.9.2/cmath:44,
                 from /opt/csw/include/c++/4.9.2/complex:44,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/lib/gcc/sparc-sun-solaris2.10/4.9.2/include-fixed/sys/feature_tests.h:196:0: note: this is the location of the previous definition
 #define _FILE_OFFSET_BITS 32
 ^
In file included from scipy/special/wright.cc:80:0:
scipy/special/_round.h: In function âdouble add_round_up(double, double)â:
scipy/special/_round.h:17:19: error: âisnanâ was not declared in this scope
     if (sc_isnan(a) || sc_isnan(b)) {
                   ^
scipy/special/_round.h:17:19: note: suggested alternative:
In file included from /opt/csw/include/c++/4.9.2/complex:44:0,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/include/c++/4.9.2/cmath:837:5: note:   âstd::isnanâ
     isnan(_Tp __f)
     ^
In file included from scipy/special/wright.cc:80:0:
scipy/special/_round.h: In function âdouble add_round_down(double, double)â:
scipy/special/_round.h:37:19: error: âisnanâ was not declared in this scope
     if (sc_isnan(a) || sc_isnan(b)) {
                   ^
scipy/special/_round.h:37:19: note: suggested alternative:
In file included from /opt/csw/include/c++/4.9.2/complex:44:0,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/include/c++/4.9.2/cmath:837:5: note:   âstd::isnanâ
     isnan(_Tp __f)
     ^
scipy/special/wright.cc: In function âint wright::wrightomega_ext(std::complex<double>, std::complex<double>*, std::complex<double>*)â:
scipy/special/wright.cc:116:16: error: âisnanâ was not declared in this scope
   if(sc_isnan(x) || sc_isnan(y))
                ^
scipy/special/wright.cc:116:16: note: suggested alternative:
In file included from /opt/csw/include/c++/4.9.2/complex:44:0,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/include/c++/4.9.2/cmath:837:5: note:   âstd::isnanâ
     isnan(_Tp __f)
     ^
scipy/special/wright.cc:124:21: error: âisinfâ was not declared in this scope
   else if(sc_isinf(x) && (x < 0.0) && (-pi < y) && (y<= pi))
                     ^
scipy/special/wright.cc:124:21: note: suggested alternative:
In file included from /opt/csw/include/c++/4.9.2/complex:44:0,
                 from scipy/special/wright.hh:4,
                 from scipy/special/wright.cc:1:
/opt/csw/include/c++/4.9.2/cmath:828:5: note:   âstd::isinfâ
     isinf(_Tp __f)
     ^
In file included from scipy/special/_round.h:10:0,
                 from scipy/special/wright.cc:80:
scipy/special/cephes/dd_idefs.h: At global scope:
scipy/special/cephes/dd_idefs.h:53:1: warning: âdouble quick_two_diff(double, double, double*)â defined but not used [-Wunused-function]
 quick_two_diff(double a, double b, double *err)
 ^
scipy/special/cephes/dd_idefs.h:130:1: warning: âdouble two_sqr(double, double*)â defined but not used [-Wunused-function]
 two_sqr(double a, double *err)
 ^
scipy/special/cephes/dd_idefs.h:148:1: warning: âdouble two_div(double, double, double*)â defined but not used [-Wunused-function]
 two_div(double a, double b, double *err)
 ^
scipy/special/cephes/dd_idefs.h:169:1: warning: âdouble two_nint(double)â defined but not used [-Wunused-function]
 two_nint(double d)
 ^
scipy/special/cephes/dd_idefs.h:179:1: warning: âdouble two_aint(double)â defined but not used [-Wunused-function]
 two_aint(double d)
 ^
scipy/special/cephes/dd_idefs.h:187:1: warning: âint two_comp(double, double)â defined but not used [-Wunused-function]
 two_comp(const double a, const double b)
 ^
error: Command "g++ -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/Python3/scipy-1.1.0/scipy/special -I/usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include -I/usr/local/Python3/Python-3.6.5/install/include/python3.6m -I/usr/local/Python3/scipy-1.1.0/scipy/special/c_misc -Iscipy/_lib -I/usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include -I/usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/numpy-1.14.5-py3.6-solaris-2.10-sun4u.32bit.egg/numpy/core/include -I/usr/local/Python3/Python-3.6.5/install/include/python3.6m -c scipy/special/wright.cc -o build/temp.solaris-2.10-sun4u.32bit-3.6/scipy/special/wright.o -MMD -MF build/temp.solaris-2.10-sun4u.32bit-3.6/scipy/special/wright.o.d" failed with exit status 1
-bash-3.2#

Okay. Looks like this is the problem: https://github.com/scipy/scipy/blob/master/scipy/_lib/_c99compat.h#L119. isnan and sc_isnan (and their inf equivalents) are only defined for (__STDC_VERSION__ < 199901L). My best guess: Solaris 10 has a higher __STDC_VERSION__ but still no isnan/isinf. I'm not sure why we have the logic there as it is, rather than unconditionally checking if isnan/isinf are defined.
Introduced in gh-8006. Done by @xoviat, who isn't around anymore.
@elinay6 if the above makes sense to you, could you try and edit that file to remove the #if check on line 119 and see if you get things to compile then?

@rgrommers:  I modified _c99compat.h as directed and the installation completed successfully:
creating dist
creating 'dist/scipy-1.1.0-py3.6-solaris-2.10-sun4u.32bit.egg' and adding 'build/bdist.solaris-2.10-sun4u.32bit/egg' to it
removing 'build/bdist.solaris-2.10-sun4u.32bit/egg' (and everything under it)
Processing scipy-1.1.0-py3.6-solaris-2.10-sun4u.32bit.egg
creating /usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/scipy-1.1.0-py3.6-solaris-2.10-sun4u.32bit.egg
Extracting scipy-1.1.0-py3.6-solaris-2.10-sun4u.32bit.egg to /usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages
Adding scipy 1.1.0 to easy-install.pth file
Installed /usr/local/Python3/Python-3.6.5/install/lib/python3.6/site-packages/scipy-1.1.0-py3.6-solaris-2.10-sun4u.32bit.egg
Processing dependencies for scipy==1.1.0
Finished processing dependencies for scipy==1.1.0
Using GCC 4.9.2 causes me to receive import errors on 5 of the Scipy extensions.  However, all of the extensions import properly using GCC 5.5.  Thanks so much for your help!

In case anyone is interested, these are the import errors I receive after compiling everything with GCC 4.9.2:
Error message:

Great, thanks @elinay6. It would also be useful if you could check if __STDC_VERSION__ is defined, and if so what value it has.
__STDC_VERSION__ < 199901L means "before C99", so is in principle correct. However Solaris seems to be doing something not quite right here.
I've marked this for 1.2.0, we can fix this after the release split still imho.

Thinking about it again, what we do really is correct and changing it may introduce some other unforeseen issue.
It's possible that Solaris claiming to have C99 but not supporting isnan/isinf is related to https://github.com/oracle/solaris-userland/issues/1#issuecomment-374533392. @elinay6 could you put the full build log in a gist and link it? I'm pretty sure we're not inserting -std=c99 anywhere, but wanted to check. I suspect that that may also fix the issue.

@pv thoughts on removing #if (__STDC_VERSION__ < 199901L) and doing the checks under it unconditionally?
A comment at the top of the file says
This file should only affect compilers that do not support C99 natively.

however I'm not sure if there's a reason for that besides a bit of efficiency.

@rgommers Scipy build logs for GCC 4.9.2 and 5.5:
https://gist.github.com/elinay6/e7ae1c2b0006eb96cf4fe73233158b03
https://gist.github.com/elinay6/b5a7aa5eb3826ef10071cec57970e1dc

The conditional block under #if (__STDC_VERSION__ < 199901L) checks if definitions exist for fma,  isinf,isnan, isfinite before defining them to something.  Under C99, are these always macros or can they be functions?  I recall seeing some paths through gcc 's  which undef'ed them before defining inline functions with the same name.

@pvanmulbregt good question. I believe C99 says it's always a macro, however here the issue is in C++ code. This MSDN page says:

When compiled as C++, the isnan macro is not defined, and an isnan template function is defined instead. It returns a value of type bool instead of an integer.

There could also be platforms where C99 isn't followed closely enough.

gh-9214 has a proposed patch, not 100% correct I think, but it does work around the problem with sc_isnan.

Copying the patch from gh-9214:
--- scipy/special/_round.h.orig 2018-05-05 17:10:11.000000000 +0000
+++ scipy/special/_round.h
@@ -9,6 +9,10 @@
 #include "_c99compat.h"
 #include "cephes/dd_idefs.h"

+#if defined(__cplusplus)
+using std::isinf;
+using std::isnan;
+#endif

 double add_round_up(double a, double b)
 {

I'm actually not quite sure if the fix should go into each file that uses sc_isnan et al and may be used in C++ code, or in _lib/_c99compat.h.

Let's not bump this yet - I won't get around to it before the release branching, but would like to have another look and a possible fix can be backported.

We don't do a 1.4.2, and it's fixed in >= 1.5.0 - closing.


adding repositories.drupal to global composer config is ignored; local is used

If I add packagist repo to global config
composer global config repositories.drupal composer https://packagist.drupal-composer.org
cat ~/.composer/composer.json
    {
        "repositories": {
            "drupal": {
                "type": "composer",
                "url": "https://packagist.drupal-composer.org"
            }
        }
    }

any subsequent package require fails
cd <drupalroot>
composer require  drupal/big_pipe:8.1.0-beta3

    ./composer.json has been updated
    Loading composer repositories with package information
    Updating dependencies (including require-dev)
    Your requirements could not be resolved to an installable set of packages.

      Problem 1
        - The requested package drupal/big_pipe could not be found in any version, there may be a typo in the package name.

    Potential causes:
     - A typo in the package name
     - The package is not available in a stable-enough version according to your minimum-stability setting
       see <https://getcomposer.org/doc/04-schema.md#minimum-stability> for more details.

    Read <https://getcomposer.org/doc/articles/troubleshooting.md> for further common problems.

    Installation failed, reverting ./composer.json to its original content.

But if I add the repo locally
composer config repositories.drupal composer https://packagist.drupal-composer.org
cat ./composer.json
    ...
      "repositories": {
        "drupal": {
          "type": "composer",
          "url": "https://packagist.drupal-composer.org"
        }
      }
    }

all packages install with no issue
composer require  drupal/big_pipe:8.1.0-beta3
    ./composer.json has been updated
    Loading composer repositories with package information
    Updating dependencies (including require-dev)
      - Installing drupal/big_pipe (8.1.0-beta3)
        Loading from cache

    > Drupal\Core\Composer\Composer::vendorTestCodeCleanup
    Writing lock file
    Generating autoload files
    > Drupal\Core\Composer\Composer::preAutoloadDump
    > Drupal\Core\Composer\Composer::ensureHtaccess

Iiuc, global configs should be respected.
Is there additional config needed here? or is the use of global config broken?

The global repo config is not related to your project configuration. Global repos don't apply to the project configuration, thats the default behaviour of composer.

Hm.  I'd read this
https://getcomposer.org/doc/03-cli.md#composer-home-config-json

    You may put a config.json file into the location which COMPOSER_HOME points to.
    Composer will merge this configuration with your project's composer.json when you
    run the install and update commands.

    This file allows you to set repositories and configuration for the user's projects.

    In case global configuration matches local configuration, the local configuration in
    the project's composer.json always wins.

to suggest setting
export COMPOSER_HOME="${HOME}/.composer"

is supposed to do the trick.
It apparently doesn't.
Perhaps I've misread, or, drupal8's project is for some reason insensitive to it.

Sounds like a composer bug, could you open a issue against composer directly?

Your schema is wrong. It should looks like a normal composer.json.
~/.composer/config.json:
 {
   "repositories": [
   {
     "type": "composer",
     "url": "https://packagist.drupal-composer.org"
   }
   ]
 }

solved at https://github.com/composer/composer/issues/4909


AssertionError when running LS_AMEGO.py with a specific video in Python

I encountered an AssertionError while trying to run the Python script LS_AMEGO.py with a specific video of EPIC-KITHCENS (ID: P02_106). Below is the traceback:

Traceback (most recent call last):
File "/path/to/AMEGO/LS_AMEGO.py", line 170, in
processor.process()
File "/path/to/AMEGO/LS_AMEGO.py", line 123, in process
self.features_track[track_num] = self.extract_feat(track_boundaries)
File "/path/to/AMEGO/LS_AMEGO.py", line 111, in extract_feat
features.append(self.net(image.unsqueeze(0).cuda()))
File "path/to/anaconda3/envs/amego/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/path/to/.cache/torch/hub/facebookresearch_swag_main/models/vision_transformer.py", line 360, in forward
assert h == w == self.image_size
AssertionError

It appears the error is related to the assert h == w == self.image_size line in the Vision Transformer model.
In this case, h and w are both 224, while self.image_size is 512.
Could you please advise on how to resolve this mismatch? Any guidance would be appreciated.

Hello! I just added a patch, including the transformations adopted by the SWAG models ce683bd. Let me know if you encounter other problems.


How to install fzf with dein?

Problems summary
I am converting my vimrc to rein from neobundle.
I am having trouble getting fzf to install properly
Expected
Environment Information

OS: OS X
Vim version: nvim 0.1.2

Provide a minimal .vimrc with less than 50 lines (Required!)
" Your minimal .vimrc
set runtimepath^=~/.vim/bundle/dein.nvim/
call dein#begin(path)
call dein#add('junegunn/fzf', { 'build': './install' })
call dein#add('junegunn/fzf.vim', { 'depends': 'junegunn/fzf' })
call dein#end()
call dein#update()

The reproduce ways from Vim starting (Required!)

start vim, run :Files command
Not an editor command: Files

Upload the log messages by :redir and :message
Output when running nvim:
nvim nvim/init.vim
[dein] Update started: (2016/04/11 12:19:00)
[dein] (1/2): |fzf.vim| git clone --recursive https://github.com/junegunn/fzf.vim.git "/Users/mr1483/.vim/bundle/repos/github.com/junegunn/fzf.vim"
[dein] (1/2) [==========          ] fzf.vim
[dein] (2/2): |fzf| git clone --recursive https://github.com/junegunn/fzf.git "/Users/mr1483/.vim/bundle/repos/github.com/junegunn/fzf"
[dein] (2/2) [====================] fzf
[dein] (1/2): |fzf.vim| Updated
[dein] (2/2): |fzf| Updated
[dein] Building...
[dein] Downloading bin/fzf ...
[dein]   - Found in $PATH
[dein]   - Creating symlink: /Users/mr1483/.fzf/bin/fzf -> bin/fzf
[dein]   - Checking fzf executable ... 0.11.4
[dein] Generate ~/.fzf.bash ... OK
[dein] Generate ~/.fzf.zsh ... OK
[dein] Update fish_user_paths ... OK
[dein] Symlink /Users/mr1483/.config/fish/functions/fzf_key_bindings.fish ... OK
[dein] Update /Users/mr1483/.bashrc:
[dein]   - [ -f ~/.fzf.bash ] && source ~/.fzf.bash
[dein]     - Already exists: line #2
[dein] Update /Users/mr1483/.zshrc:
[dein]   - [ -f ~/.fzf.zsh ] && source ~/.fzf.zsh
[dein]     - Already exists: line #143
[dein] Update /Users/mr1483/.config/fish/functions/fish_user_key_bindings.fish:
[dein]   - fzf_key_bindings
[dein]     - Already exists: line #2
[dein] Finished. Restart your shell or reload config file.
[dein]    source ~/.bashrc  # bash
[dein]    source ~/.zshrc   # zsh
[dein]    fzf_key_bindings  # fish
[dein] Use uninstall script to remove fzf.
[dein] For more information, see: https://github.com/junegunn/fzf
[dein] Updated plugins:
[dein]   fzf.vim
[dein]     https://github.com/junegunn/fzf.vim/compare/...7cfff6b16f43c74702e93e20ec9bc846647650ba
[dein]   fzf
[dein]     https://github.com/junegunn/fzf/compare/...e5ac2ebd7c6f845d34b908455a7f12d1be40d6a4
[dein] Runtimepath updated: (2016/04/11 12:19:12)
Press ENTER or type command to continue

fzf is not vim plugin.you should use dein install it .and you can have a look at my dotfiles

fzf.vim is a vim plugin.
With NeoBundle, I could use this:
NeoBundle 'junegunn/fzf', { 'dir': '~/.fzf', 'do': 'yes \| ./install' }
NeoBundle 'junegunn/fzf.vim'

I want to know how I can do the same with dein

Not sure if it works but try
call dein#add('junegunn/fzf', { 'build': './install', 'merged': 0 })

As @wsdjeg said, it seems fzf itself is not a vim plugin so it could not be merged (dein.vim merge plugins into a single directory in default, so you may need explicit 'merged': 0 in setting dictionary.)

This works, thanks @lambdalisue and @wsdjeg!

Grad to hear that ;-)

Added PR to address this for our children: https://github.com/junegunn/fzf.vim/pull/115

This depends usage is wrong.
call dein#add('junegunn/fzf.vim', { 'depends': 'junegunn/fzf' })

This is the correct.
call dein#add('junegunn/fzf.vim', { 'depends': 'fzf' })

fzf is not Vim plugin.  So you should set rtp to ''.
call dein#add('junegunn/fzf', { 'build': './install', 'rtp': '' })

Thanks @Shougo

Note that now 'junegunn/fzf' contains a vim plugin as well so it needs the rtp.
call dein#add('junegunn/fzf', { 'build': './install --all', 'merged': 0 })
call dein#add('junegunn/fzf.vim', { 'depends': 'fzf' })

Worked for me.


[Dataflow Streaming] Move throwExceptionOnLargeOutput out of OperationalLimits

Planning to move OperationslLimits along with other settings from backend into a common class. throwExceptionOnLargeOutput is calculated from PipelineOptions, moving it out of OperationalLimits to avoid making the new classes depend on PipelineOptions.

R: @scwhittle

cc: @acrites


task/internal/syslog: Add capability to ignore kernel failures

Adds capability to ignore kernel failures to jobs. This is done by
adding 'syslog' dict to config dictionary which holds the 'ignorelist'
of kernel failures.
Also removes old kernel failurs from exclude list.
Fixes: https://tracker.ceph.com/issues/50150
Signed-off-by: Kotresh HR<EMAIL_ADDRESS>

Since this started out as "begin grepping kernel logs for kclient warnings/failures", is there a suspicion that the existing greps don't work (or perhaps work but not on all distros)?  Have you attempted to trigger a WARNING or BUG and observe the test failing?

Since this started out as "begin grepping kernel logs for kclient warnings/failures", is there a suspicion that the existing greps don't work (or perhaps work but not on all distros)? Have you attempted to trigger a WARNING or BUG and observe the test failing?

Honestly, we didn't know about this syslog grep at the start. I was concerned there was no detection of kernel faults at all since I would occasionally see warnings that should fail tests in a syslog.
So I guess we're both concerned that this grep wasn't catching genuine faults/warnings, that it's not easily configurable, and that it's not easily discovered.

I remember it breaking a few years ago -- something to do with traditional syslog vs systemd journal interaction but it was fixed then.  It might be the case that it broke again, which is why I asked.  The issue is not with the grep invocation but rather with what is being grepped (i.e. does that kern.log file actually contain any kernel log messages at the time grep is invoked, etc).
Removing outdated exclude items is obviously lower priority than resurrecting the core functionality in case it is broken.

@kotreshhr what's the status on this PR?

@kotreshhr what's the status on this PR?

@batrick I think it's good to be merged. Here is the run triggered and Jeff's comments on the same. Sorry, I should have discussed here. It was in the cephfs-team mailing list. Copying it here for reference.

Hi Jeff,
I was working on removing old kernel failures which are masked with the
present teuthology code.
I ran the teuthology on my PR [1] with filter "-s fs --filter mount/kclient"
and found following unique
failures in syslog. Just wanted to check with you are these real failures
which need attention?
The complete run can be found at [2].

Many thanks for doing this. Better testing with teuthology is definitely
an area where I could use a lot of help. FWIW, the pulpito links below
only work if you're on the sepia lab VPN.
The kernel is built from commit 7c1f4b5e3842, so this is missing some of
the more recent patches in testing branch.

'2021-08-14T12:28:01.665935+00:00 smithi071 kernel: WARNING: CPU: 3 PID:
39974 at fs/ceph/mds_client.c:4495 check_session_state+0x55/0x60 [ceph] '
in syslog
http://pulpito.front.sepia.ceph.com/khiremat-2021-08-09_13:23:53-fs-wip-khiremat-test-kernel-exclude-failures-distro-basic-smithi/6327912

I suspect this is fixed with a recent patch from Xiubo (1a45b99820e7).
Do we know whether this test does a 'umount -f' ?

'2021-08-14T13:05:50.330033+00:00 smithi041 kernel: INFO: task
fsstress:47378 blocked for more than 122 seconds. ' in syslog
http://pulpito.front.sepia.ceph.com/khiremat-2021-08-09_13:23:53-fs-wip-khiremat-test-kernel-exclude-failures-distro-basic-smithi/6327931

Doesn't look familiar. It looks like the client is just waiting on an
OSD reply (probably for a write since this is in sync()).

'2021-08-14T14:06:15.476389+00:00 smithi062 kernel: INFO: task
kcompactd0:59 blocked for more than 122 seconds. ' in syslog
http://pulpito.front.sepia.ceph.com/khiremat-2021-08-09_13:23:53-fs-wip-khiremat-test-kernel-exclude-failures-distro-basic-smithi/6327954

Also doesn't look familiar. The task is hung waiting for dirty pages to
be written out. We don't know to which filesystem (but it seems likely
that it was cephfs). Possibly related to #2 above.

'2021-08-14T13:37:56.676192+00:00 smithi052 kernel: WARNING: CPU: 3 PID:
101 at crypto/testmgr.c:5653 alg_test+0x245/0x450 ' in syslog
http://pulpito.front.sepia.ceph.com/khiremat-2021-08-09_13:23:53-fs-wip-khiremat-test-kernel-exclude-failures-distro-basic-smithi/6327978

Looks unrelated to ceph at all. This one happened at boot time. Probably
some bug in the crypto code. This is a -rc4 based kernel after all, so
there are bugs in there...

'2021-08-14T17:11:12.918237+00:00 smithi027 kernel: [ 969.167486] INFO:
task ffsb:47395 blocked for more than 120 seconds. ' in syslog
http://pulpito.front.sepia.ceph.com/khiremat-2021-08-09_13:23:53-fs-wip-khiremat-test-kernel-exclude-failures-distro-basic-smithi/6328144

Not much to go on here. There are no stack traces, AFAICT and it's
unclear why. Maybe we need to do something in Ubuntu distros to turn
them on?

'2021-08-15T06:15:50.378319+00:00 smithi019 kernel: INFO: task iozone:67133
blocked for more than 120 seconds. ' in syslog
http://pulpito.front.sepia.ceph.com/khiremat-2021-08-09_13:23:53-fs-wip-khiremat-test-kernel-exclude-failures-distro-basic-smithi/6328732

Hung (briefly) while waiting on ceph_fsync. It looks like this one might
have eventually recovered as we don't see any more messages about
blocked tasks even after another 120s. This may just be due to slow OSD
responses, and could also be related to #2 and/or #3.

 It has 1000+ jobs. Please use a subset.

Teuthology run triggered: http://pulpito.front.sepia.ceph.com/khiremat-2022-01-13_16:30:18-fs-wip-khiremat-44574-syslog-warn-ignorelist-distro-basic-smithi/

I killed this @kotreshhr . It has 1000+ jobs. Please use a subset.

@batrick I am not sure I understand --subset option correctly. I did give 3/16. Any suggestion?

Here is what I usually run and it gets about 300 jobs:
teuthology-suite --machine-type smithi --email<EMAIL_ADDRESS>-p 95 --suite fs --force-priority --subset $((RANDOM % 32))/32 --ceph wip-branch

Here is what I usually run and it gets about 300 jobs:
teuthology-suite --machine-type smithi --email<EMAIL_ADDRESS>-p 95 --suite fs --force-priority --subset $((RANDOM % 32))/32 --ceph wip-branch

Thanks @batrick, here is the teuthology run:
http://pulpito.front.sepia.ceph.com/khiremat-2022-01-16_11:16:23-fs-wip-khiremat-44574-syslog-warn-ignorelist-distro-basic-smithi/

Here is what I usually run and it gets about 300 jobs:
teuthology-suite --machine-type smithi --email<EMAIL_ADDRESS>-p 95 --suite fs --force-priority --subset $((RANDOM % 32))/32 --ceph wip-branch

Thanks @batrick, here is the teuthology run: http://pulpito.front.sepia.ceph.com/khiremat-2022-01-16_11:16:23-fs-wip-khiremat-44574-syslog-warn-ignorelist-distro-basic-smithi/

There's some ... weird failures in that run. Please do --rerun of it to see if they're transient.

Here is what I usually run and it gets about 300 jobs:
teuthology-suite --machine-type smithi --email<EMAIL_ADDRESS>-p 95 --suite fs --force-priority --subset $((RANDOM % 32))/32 --ceph wip-branch

Thanks @batrick, here is the teuthology run: http://pulpito.front.sepia.ceph.com/khiremat-2022-01-16_11:16:23-fs-wip-khiremat-44574-syslog-warn-ignorelist-distro-basic-smithi/

There's some ... weird failures in that run. Please do --rerun of it to see if they're transient.

Rerun QA link:
http://pulpito.front.sepia.ceph.com/khiremat-2022-01-22_07:42:15-fs-wip-khiremat-44574-syslog-warn-ignorelist-distro-basic-smithi/

Good to merge IMO from CephFS side. @idryomov do you want to have another look? rbd run needed?

I'd like to see a confirmation that the whole grep invocation is actually working. Could you please build a kernel that would emit a custom WARNING or ERROR

Would it be crazy to add a patch to upstream kernel where a mount option triggers a warning/error so we can continuously verify this works?

Definitely crazy IMO.  But we could find a way to trigger something with one of the "bad" keywords in it on vanilla kernels.  For example, take all OSDs out against some in-flight I/O -- that should trigger "INFO: task ... blocked for more than %ld seconds".

... or we could carry the kind of patch you envision in our testing branch.

Would it be crazy to add a patch to upstream kernel where a mount option triggers a warning/error so we can continuously verify this works?

I wouldn't use a mount option. This sounds more like a job for debugfs or maybe consider the fault injection framework documented here:
https://www.kernel.org/doc/html/latest/fault-injection/fault-injection.html

Hmm, I think we can just use the network namespaces to network partition the kernel mount (suspend_netns in mount.py), wait a reasonable amount of time for complaints like the osd timeout, then exit. Just need to grep for the expected failure message (or fail) and make sure ignorelist works by ignoring the message. We don't grep for the expected failure message, perhaps that should go in this PR.

I bought this up in cephfs standup last while discussing https://tracker.ceph.com/issues/64471 with @batrick and recalled this change. So, this was close to getting merged, but @idryomov suggested a validation before merging. Can we agree on a way forward to validate this?

I bought this up in cephfs standup last while discussing https://tracker.ceph.com/issues/64471 with @batrick and recalled this change. So, this was close to getting merged, but @idryomov suggested a validation before merging. Can we agree on a way forward to validate this?

To start, @kotreshhr needs to rebase. Then I suggest trying Ilya's suggestion of stopping OSDs while some application is writing a large file. This shouldn't be hard to test...

I bought this up in cephfs standup last while discussing https://tracker.ceph.com/issues/64471 with @batrick and recalled this change. So, this was close to getting merged, but @idryomov suggested a validation before merging. Can we agree on a way forward to validate this?

To start, @kotreshhr needs to rebase. Then I suggest trying Ilya's suggestion of stopping OSDs while some application is writing a large file. This shouldn't be hard to test...

@batrick Does that mean, I write a teuthology test that kills OSDs while writing large file ?

I bought this up in cephfs standup last while discussing https://tracker.ceph.com/issues/64471 with @batrick and recalled this change. So, this was close to getting merged, but @idryomov suggested a validation before merging. Can we agree on a way forward to validate this?

To start, @kotreshhr needs to rebase. Then I suggest trying Ilya's suggestion of stopping OSDs while some application is writing a large file. This shouldn't be hard to test...

@batrick Does that mean, I write a teuthology test that kills OSDs while writing large file ?

As discussed during standup - that's a good start to validate the filter.


NPM Registry

Obligated question/issue:
Is there any reason why this beautiful middleware is not on npm.
It would be handy to have it available by doing this:
npm install --save redux-websocket-middleware

Flattered! The reason is because I'm still working on extracting this from another project of mine, and trying to figure out a good public API. The code in this repo is is a very work-in-progress state.
I hope to have some time next week to do so! Stay tuned.

I think the code looks pretty cool, you should publish it.


simple relay isn't logging the ingester name in some log events

example:
<14>1 2022-03-10T08:01:18.51902-07:00 3ca78ee40121 simplerelay - SimpleRelay/[simple.go]():179 [gw@1 address="[<IP_ADDRESS>]():37948" readertype="LINE" mode="tcp" listener="default"] accepted connection

Maybe we don't care because the appname is in there, but other simple relay logs do log the ingester name:
<14>1 2022-03-10T08:01:03.016651-07:00 3ca78ee40121 simplerelay - ingest/[muxer.go]():727 [gw@1 ingester="simplerelay" ingesteruuid="3dd0b19b-1ea0-4a02-9eba-e8bc74ace5c7"] Ingester gone hot

Closing as invalid