AttributeError error when importing localgraphclustering

Issue 96 may have  similar problem as me.
(https://github.com/kfoynt/LocalGraphClustering/issues/96)
But there was little information so I will provide some details here.
I am running python3.6 in jupyter notebook, with macOS 10.14.5. The error message is shown below:
AttributeError                            Traceback (most recent call last)
<ipython-input-3-8da92a9ab2a6> in <module>
----> 1 import localgraphclustering as lgc

/anaconda3/lib/python3.6/site-packages/localgraphclustering/__init__.py in <module>
----> 1 from .GraphLocal import GraphLocal
      2 from .GraphDrawing import GraphDrawing
      3 from .fiedler import fiedler, fiedler_local
      4 from .approximate_PageRank import approximate_PageRank
      5 from .approximate_PageRank_weighted import approximate_PageRank_weighted

/anaconda3/lib/python3.6/site-packages/localgraphclustering/GraphLocal.py in <module>
      8 import warnings
      9 import collections as cole
---> 10 from .cpp import *
     11 import random
     12

/anaconda3/lib/python3.6/site-packages/localgraphclustering/cpp/__init__.py in <module>
     41 from .MQI_cpp import *
     42 from .proxl1PRaccel import *
---> 43 from .proxl1PRrand_cpp import *
     44 from .SimpleLocal_cpp import *
     45 from .sweepcut_cpp import *

/anaconda3/lib/python3.6/site-packages/localgraphclustering/cpp/proxl1PRrand_cpp.py in <module>
     50
     51 _graphlib_funs_proxl1PRrand64 = _setup_proxl1PRrand_args(
---> 52     'int64','int64', _graphlib.proxl1PRrand64)
     53 _graphlib_funs_proxl1PRrand32 = _setup_proxl1PRrand_args(
     54     'uint32','uint32', _graphlib.proxl1PRrand32)

/anaconda3/lib/python3.6/ctypes/__init__.py in __getattr__(self, name)
    359         if name.startswith('__') and name.endswith('__'):
    360             raise AttributeError(name)
--> 361         func = self.__getitem__(name)
    362         setattr(self, name, func)
    363         return func

/anaconda3/lib/python3.6/ctypes/__init__.py in __getitem__(self, name_or_ordinal)
    364
    365     def __getitem__(self, name_or_ordinal):
--> 366         func = self._FuncPtr((name_or_ordinal, self))
    367         if not isinstance(name_or_ordinal, int):
    368             func.__name__ = name_or_ordinal

AttributeError: dlsym(0x7fec89d9ade0, proxl1PRrand64): symbol not found

It seems to me that something goes wrong with the c library. But I'm not familiar with this. Could anyone give some help? Thanks in advance.

Do you have a C++ compiler installed? How did you install the package?

Do you have a C++ compiler installed? How did you install the package?

I think I have installed gcc, some information about it is shown below :
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 10.0.1 (clang-1<IP_ADDRESS>)
Target: x86_64-apple-darwin18.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

And i installed the package by :
pip3 install localgraphclustering

Do you have a C++ compiler installed? How did you install the package?

I think I have installed gcc, some information about it is shown below :
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 10.0.1 (clang-1<IP_ADDRESS>)
Target: x86_64-apple-darwin18.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

And i installed the package by :
pip install localgraphclustering

Can you try to run 'bin/createGraphLibFile.sh' from the package repository and see if that can finish successfully?

Do you have a C++ compiler installed? How did you install the package?

I think I have installed gcc, some information about it is shown below :
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 10.0.1 (clang-1<IP_ADDRESS>)
Target: x86_64-apple-darwin18.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

And i installed the package by :
pip install localgraphclustering

Can you try to run 'bin/createGraphLibFile.sh' from the package repository and see if that can finish successfully?

No, it doesn't work.
However, I clone the repo and try to use :
python setup.py install
instead, and it works. I can now import localgraphclustering in terminal, though jupyter notebook still fails.
While using this method, I encountered another problem :
The plotly.plotly module is deprecated,
please install the chart-studio package and use the
chart_studio.plotly module instead

I fixed it by changing the code in ncpplots.py.
Next I may try to do something with my jupyter notebook...


Add bug button to skillmap header to toggle Usabilla

I am not sure I understand the change. Don't we have a feedback button on the side?  We are adding another bug button at the top too?


Add NewWithPool to expose redis pool options

NewWithPool is important in case you need to fine tune your internal
redis pool.
I need to fine tune the redis pool to match my use cases, currently the New() function is hiding a LOT
of options that is supported by the underlying redis pool.
It's important to have the option to fine tune your underlying pool.

LGTM, thanks!


Add type generic for route meta data

What problem does this feature solve?
My route metadata should be type save
What does the proposed API look like?
Add a generic to RouteRecordRaw.
const routes:  RouteRecordRaw<RouteMeta>[]  = // ... Routes are here

Full example
export type RouteMeta = {
  label: string,
  iconClass: string,
}

const routes:  RouteRecordRaw<RouteMeta>[] = [
  {
    path: '/',
    name: 'Home',
    component: Home,
    meta: {
      label: 'My Home',
      iconClass: 'home-icon',
    },
  },
  {
    path: '/settings',
    name: 'Settings',
    component: Settings,
    meta: {
      label: 'Settings',
      iconClass: 'settings-icon',
    }
  },
]

const router = createRouter({
  history: createWebHashHistory(),
  routes
})

export default router

Doesn't make sense for vue2. I opened an issue for vue-router-next.


Lots of small/bigish tweaks

Desktop up next for me!

:white_check_mark: Build TimeHACK AppVeyor-340 completed (commit https://github.com/TimeHACKDevs/TimeHACK/commit/d18c590827 by @Alex-TIMEHACK)


Improve performance

Parsing a XLSX with about 40k rows and 7 columns takes 77.38 s.
For comparison, using Apache POI with JInterface for interop takes about 7.7 s.
Creating this as a reminder to profile the code and see if it is possible to improve the performance.
Edit: The current implementation loads the entire worksheet XML into memory and builds at least two intermediate lists to produce the formatted output. We can improve this by using xmerl_sax_parser and in one pass produce the output list. An alternative that would add an external dependency and some C code but could be faster is https://github.com/processone/fast_xml/
Edit 2: The zip file is opened, loaded and decompressed several times (1. validate, 2. extract shared_strings, 3. extract styles, n. for each worksheet). To improve performance we should load it and decompress only once. We can start a GenServer or Agent to keep the state for a single .xlsx to be processed and stop it after finishing. If all binaries are only accessed by this process, I think the garbage collector will just free all the memory, including the large binaries, once the process dies.

Performance is definitely an issue. I've been playing around with ideas for a few weeks now. Thank you @pma for the suggestions, I'll look into using a GenServer as that sounds like a great idea.

@pma I've implemented SAX parsing and Erlang Term Storage (ETS) to Xlsxir. If you have time, would you mind processing your workbooks again and let me know if performance has improved on your end? You'll need to take a look at the docs as I've majorly changed the functionality. Thanks!

@kennellroxco With the SAX implementation the .xlsx that previously took 77s to extract now takes 9s. Xlsxir.get_list/0 runs in under 1ms. Quite an improvement, congrats!
For comparison, my implementation using Apache POI + JInterface takes 15s to do the equivalent of Xlsxir.extract/3 + Xlsxir.get_list/0.
Have you considered xmerl_sax_parser before deciding to use erlsom? If xmerl_sax_parser has comparable performance while avoiding an external dependency, it would be preferable I think.
With the state being stored in ets, how will the library handle concurrent .xlsx parsing, since it seems to be used as a global variable? How would this approach compare to having extract/3 return a reference to a unique ets table or even the intermediate data structure that is currently stored in ets, and then passed to get_list/1 (in terms of performance and API simplicity)?
In my use case I need to iterate the Excel rows and for each row create an Ecto.Changeset. If all changesets are valid I then insert all into a database. I'm thinking that if we are able to pass a function as argument to get_list that takes the row values and returns the mapped value, we could have the final list with just one pass. This mapping could even be done during the SAX parsing, skipping the intermediate sheet representation altogether (Maybe not that relevant since get_list currently takes 300 µs for a sheet with 55K rows).
Another interesting option to consider is to expose something like File.stream!. This would be quite flexible and allow different ways of processing the data (and cover the mapping use case described in the previous paragraph).

@pma I did initially look at xmerl but had trouble understanding the documentation. I was able to find a great example of erlsom from a blog post by @benjamintanweihao, so that's why I went with erlsom. Now that I understand SAX parsing better, I'll take a look at xmerl again to see if I can get rid of that extra dependency.
Thanks for the comments and suggestions. I do plan to implement a better way of parsing multiple worksheets and like your ideas of mapping during the SAX parsing and providing some sort of streaming option, so I will look into those as well. I appreciate you taking the time to test it out and let me know your thoughts!

Initial performance improvement complete. Additional improvements will follow.


[Question] Does F2 has typescript declaration file ?

F2 Version:
Platform:
Mini Showcase(like screenshots):
CodePen Link:

not yer


There is no setup.py in Python or Packaging folder

Hello,
I tried to install simpleElastix on windows but couldn't install it properly.
files in Packaging folder:

files in Python folder:

When I run the below line,
python setup.py.in install
I got the below error:
`running install
running bdist_egg
running egg_info
creating SimpleITK.egg-info
writing SimpleITK.egg-info\PKG-INFO
writing dependency_links to SimpleITK.egg-info\dependency_links.txt
writing top-level names to SimpleITK.egg-info\top_level.txt
writing manifest file 'SimpleITK.egg-info\SOURCES.txt'
reading manifest file 'SimpleITK.egg-info\SOURCES.txt'
writing manifest file 'SimpleITK.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_py
creating build
creating build\lib.win-amd64-3.8
creating build\lib.win-amd64-3.8\SimpleITK
copying C:\Users\mohammadamin.safarza\build\SimpleITK-build\Wrapping\Python\SimpleITK.py -> build\lib.win-amd64-3.8\SimpleITK
copying C:\Users\mohammadamin.safarza\build\SimpleITK-build\Wrapping\Python_init_.py -> build\lib.win-amd64-3.8\SimpleITK
running build_ext
error: can't copy 'C:\Users\user\build\SimpleITK-build\Wrapping\Python\Packaging@SimpleITK_BINARY_MODULE@': doesn't exist or not a regular file`
Does anyone have any idea to fix this?
Thanks,

Hi,
I was struggling this issue although my environment is MAC M1.
First, I replace @SimpleITK_BINARY_MODULE@ as SimpleITK-build/Wrapping/Python/SimpleITK/_SimpleITK.so file.
in my case, it was located under /Wrapping/Python/SimpleITK, but I'm not sure where you have this _SimpleITK.so file.
It seems fine for me after this one and.. with 'sudo' installing python wrapping and delete the old SimpleITK. https://github.com/SuperElastix/SimpleElastix/issues/8#issuecomment-404890927


Game crash when running world with Gallant's Gauntlets mod

Mod Name
Gallant's Gauntlets
Mod Homepage
https://www.curseforge.com/minecraft/mc-mods/gauntlets
Minecraft version
1.20.1
Describe the bug
Everything is fine until I play the world. Terrain loaded, joinning world text pop up and then the game crashed
Steps to reproduce

...
Logs
https://gist.github.com/Hoa78DaMastermind/c5d7a787422c895bd97add799f7f2540
Additional context
No response

I am getting the same error on Fabric https://mclo.gs/q0fnQTy. Please report this to the mod author.


Adding The Code Hub's blog to the directory

Adding a site for developing on the iPad with Swift Playgrounds and Reality Composer.

Thank you!


Guide links aren't working

While in the guide, 9.2.0 downgrade, ctrtransfer 2.1.0,  aeskeydb.bin, fbi-2.4.2-injectable.zip, data_input_v3.zip are all not working. I've tried multiple browsers (safari, chrome, edge) and they seem not to be clickable for download. I really want this fixed as I'm starting to hack my 3ds

They're magnet links.


[Animated] [Web] Animation Memory Leak

Description
There is a memory leak on Web on the newest version of react-native. When starting a new animation, the previous Animated.Value is not unallocated.
This memory leak was not present in version 0.71.8, which was my previous React Native version before updating to Expo 49.
Because of this, the snack attached below does not have a memory leak present, as it is running using Expo 48, which does not support React native 0.72.4
Below is a screenshot of the memory snapshots from the attached snack code ran on my local system. Snapshot 4 was taken at 0 resets, and Snapshot 5 was taken at 100 resets. The difference between the two is 100 new AnimatedValue allocations, 200 AnimatedProps allocations, 200 AnimatedStyle allocations, and 400 AnimatedInterpolation allocations.
Please view the Snack below for the source code.

The most likely candidates to introduce these are :
b589123, dccb57f, c06323f
which were all found in the changelog between 0.71.8 and 0.72.4
Let me know if you need any more information.
React Native Version
0.72.4
Output of npx react-native info
System:
OS: Windows 10 10.0.19045
CPU: (8) x64 Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
Memory: 5.45 GB / 15.95 GB
Binaries:
Node:
version: 18.16.1
path: C:\Program Files\nodejs\node.EXE
Yarn:
version: 1.22.19
path: ~\AppData\Roaming\npm\yarn.CMD
npm:
version: 9.5.1
path: C:\Program Files\nodejs\npm.CMD
Watchman: Not Found
SDKs:
Android SDK: Not Found
Windows SDK: Not Found
IDEs:
Android Studio: AI-222.4459.24.2221.10121639
Visual Studio:
- 16.11.32602.291 (Visual Studio Community 2019)
Languages:
Java:
version: 17.0.7
path: C:\Program Files\Common Files\Oracle\Java\javapath\javac.EXE
Ruby: Not Found
npmPackages:
"@react-native-community/cli": Not Found
react:
installed: 18.2.0
wanted: 18.2.0
react-native:
installed: 0.72.4
wanted: 0.72.4
react-native-windows: Not Found
npmGlobalPackages:
"react-native": Not Found
Android:
hermesEnabled: false
newArchEnabled: false
iOS:
hermesEnabled: Not found
newArchEnabled: Not found
Steps to reproduce
Create any animated value.
Create an animation and start it.
Create another animation and start it.
The associated AnimatedValue objects should remain in memory without being cleared until the component containing them is unrendered.
Snack, screenshot, or link to a repository
https://snack.expo.dev/@bigcheung/suspicious-tortillas

There is a memory leak on Web on the newest version of react-native. When starting a new animation, the previous Animated.Value is not unallocated.

Why do you mention "Web" specifically here? Is this not an issue on Android or iOS?

There is a memory leak on Web on the newest version of react-native. When starting a new animation, the previous Animated.Value is not unallocated.

Why do you mention "Web" specifically here? Is this not an issue on Android or iOS?

Was unable to test on Android or iOS at the time. Can only confirm that this issue is present on web.

Can we test on Android & iOS before we can proceed with further triaging here?

+1
We seem to be having this identical issue when upgrading from Expo 49 -> 51 and React Native 71 -> 74
@Big-Cheung Did you ever identify a cause?


How can we get the http Cookie when dgp.filter.http.dubboproxy has been chosen

What would you like to be added:
I would like to ask a module which can be used to transform the http request with its Cookie to the Dubbo request so that I can get the Cookie in my Dubbo cluster. The optimal way of usage is that I choose the dgp.filter.http.dubboproxy and at the same time I can get the Cookie with the request body, then I can send then response with the Cookie too. In simple terms, I would like to find a way to add the http Cookie within the whole system which has used the Pixiu as the gateway and the dgp.filter.http.dubboproxy as the filter.
Why is this needed:
Because of the way of authentication we has chosen, we must get the Cookie almost everywhere in the system. But now I have no idea about how to get the Cookie through the dgp.filter.http.dubboproxy filter in Pixiu.

https://github.com/apache/dubbo-go-pixiu/blob/804628011ed9dd9e67d063234b123a8a1390bb0a/pixiu/pkg/filter/http/remote/call.go#L120
you can get cookie in this code
 c.Request.Header.Get("Cookie")


Expose blocktime

You can validate block time in config following JSR380 spec, see https://github.com/iExecBlockchainComputing/iexec-sms/blob/a95cc1dd46a105dc47d3b4fc42bb674bdf03eee3/src/main/java/com/iexec/sms/tee/workflow/TeeWorkflowConfiguration.java#L69-L69 for eg.


Convert samplesheet to value channel

PR checklist

[ ] This comment contains a description of changes (with reason).
[ ] If you've fixed a bug or added code that should be tested, add tests!
[ ] If you've added a new tool - have you followed the pipeline conventions in the contribution docs
[ ] If necessary, also make a PR on the nf-core/nanostring branch on the nf-core/test-datasets repository.
[ ] Make sure your code lints (nf-core lint).
[ ] Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
[ ] Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
[ ] Usage Documentation in docs/usage.md is updated.
[ ] Output Documentation in docs/output.md is updated.
[ ] CHANGELOG.md is updated.
[ ] README.md is updated (including new tool citations and authors/contributors).

Do we need both versions of the gene heatmap file (the one including wo_HKnorm_) for the tests? But I thought the tests were passing before?

@christopher-mohr There seems to be a random behavior in the tests when they are run in github actions where a suffix gets added to the files, making them sometimes fail.
Running them a second time will ensure that they pass, but I created a ticket for this: https://github.com/nf-core/differentialabundance/issues/376.

Perhaps you need to apply a similar solution like @alanmmobbs93 did here: https://github.com/nf-core/nanostring/blob/fad81c8cf68163de556e43f4515f179b4b21c2c7/workflows/nanostring.nf#L98


Performance degradation in a few minutes

From @juniordiscart on June 21, 2016 17:9

VSCode Version: 1.2.1
OS Version: OS X 10.11.5

Steps to Reproduce:

Open (relatively) large C# Unity project (±420 scripts)
Work for about 10-15 mins

Hi
Since update 1.2 (still persists in v1.2.1) Visual Studio Code started having performance degradation after a few minutes of working. The Intellisense is fluent and rapid in the first few minutes (< 1second for suggestion) but gradually takes more time to process information (> 60 seconds for suggestion).
I use the legacy C# plugin to work with a Unity project. The most up-to-date C# extension is not installed. I tried various suggestions I found here (i.e. disabling Git, excluding certain folders), but the issue keeps creeping in. I currently quickly kill VSCode and start the project back up to have a fluent intellisense for a few minutes.
Any help or light that can be shed on the matter?
Thank you.
Copied from original issue: Microsoft/vscode#7928

This is very likely a duplicate of https://github.com/OmniSharp/omnisharp-vscode/issues/269.
Note that the very latest C# extension (1.5-beta4) should address this problem. The C# legacy extension is deprecated and the latest C# extension fully supports Unity projects.


Is there a way to remove audio using ffmpeg-cli-wrapper?

I want to remove the audio sound to reduce the size of the video.
What should I do?
Thanks in advance.

Is it correct that the
.disableAudio()

method means to remove the audio?

The problem solved itself, so I closed the issue.
thank you


can't install msi on Win10 1803

the msi launches but nothing gets installed.
also can't download the msi trough Chrome anymore, only IE.

Duplicate of #377.


about stop token

there is no stop-protection layer in model, so sometimes the generated audio will be too long.

Hi @yanglu1994,
The model uses the stopping mechanism proposed in the original Tacotron paper i.e. When a frame of all zeros is predicted generation is terminated. I haven't experienced any run on audio so could you share an example input where you are seeing this?

It is happened occasionally, so I think model need a stop_protect_layer.

Thanks for sharing. Could you also send the text input so that I can try it out on my side?

My data is Chinese

Oh , I see. Based on the scale of your mel-spectrogram it looks like you're using a different preprocessing steps? The way I set it up is that the mel-spectrogram magnitudes range from -1 to just above 0 so this line and this line when I add padding for the end of sentence I use a value of 0. Did you also change this? If you didn't that could help explain the issue.

The way I set it up is that the mel-spectrogram magnitudes range from -4 to just above 4, I will change my pading_value and retrain.  when I generate the mel, the stop_thershold is-0.2. Should I  change stop_thershold to -2 or -3?

@yanglu1994, I would advise setting the padding to +4 and the stop_threshold to something like 3.5. The reason to go this way around is that you want the end of utterance frame to be different from a typical silent frame. If you sent the padding to - 4 the model will might stop early at a pause in the sentence.
Let me know how it goes.

In my opinion  the padding value must be no energy, so if mel-spectrogram range from -4 to 4, the padding value should be -4. If mel-spectrogram range from -1 to 0, the padding value should be -1.

No, I disagree. If you do that then how will the model distinguish between the end of the utterance and pause or silent frame somewhere in the middle of the utterance? For example, if you have the sentence "Hello, how are you?" the pause at the comma will have low energy near -4 in your case. So if you set stop_threshold to -3.5 then the model will stop generating early. On the other hand, if you use a frame of all 4 it doesn't appear as a natural part of speech so the model can learn to identify the end of the utterance.
That's basically what I used for the LJSpeech model and it seems to work really well so I'd recommend giving it a try.

Hi @yanglu1994, just wanted to follow up to check if you got your model to work?

according to the stop_threshold to end sentence  is not correct sometimes. I still add a stop prediction layere to decide ending position.Now I have solved this problem. Thank you anyway~

I am wondered  if the attention parameters should be changed in different sample rate ? I find data of 16k can't synthesize correct just like data of 48k. the end of 16k alignment always like this.

I met a similar problem that the synthesized audio is too long and the speech ends suddenly. I would like to ask how many steps does it take for you to get tacotron-ljspeech-yspjx3.pt? This pretrained model works well for the long utterance, but the one I trained fails.


"Quarkus: Add extensions to current project" command appears even if a Quarkus project is not open.

To replicate, open a folder that does not contain a Maven-based Quarkus project. When opening the command palette (Ctrl + Shift + P), the "Quarkus: Add extensions to current project" command appears.

This command should not appear because we do not have a Quarkus project open.
However at this point, vscode-quarkus might not have started yet. In this case, we may not be able to detect whether or not we currently have a Quarkus project open.
Maybe instead, we could have something like this:

Open a folder that does not contain a Quarkus project
Invoke the "Quarkus: Add extensions to current project" command (which starts the vscode-quarkus extension since it is one of the activation events)
vscode-quarkus then detects that we are not in a Quarkus project, and displays an error message.

There is an error message of a pom.xml cannot be found, but it would be great if we took it one step further and ensured that if we did find a pom.xml, ensure that it is for a Quarkus project.


Fix bug where deployment script gets stuck

@cesher : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off


Update reference to gomock

code.google.com doesn't exist anymore, so this will update the reference to
github.com.
No need to merge this, let's have this as a reference. This is fixed in HEAD
in the apache repo if I'm not mistaken.

Do the needful. :tractor:


Issue with quantize_static

Accuracy for normal resnet50.onnx model is coming out to be above 70% but after quantizing it, accuracy becomes 0.10%.. What could be the issue?
Any help would be appreciated

@arpdubey, could you share information about how you quantized the model, and provide evaluation script to reproduce the accuracy?

input_model_path = "resnet50.onnx"
output_model_path = "updated20.onnx"
calibration_dataset_path = "test_images/"
dr = resnet50_data_reader.ResNet50DataReader(calibration_dataset_path, input_model_path)       quantize_static(input_model_path,output_model_path,dr,per_channel=False,quant_format=QuantFormat.QOperator,weight_type=QuantType.QUInt8,activation_type=QuantType.QUInt8 )

Resnet50_data_reader
import numpy
import onnxruntime
import os
from onnxruntime.quantization import CalibrationDataReader
from PIL import Image
def _preprocess_images(images_folder: str, height: int, width: int, size_limit=0):
"""
Loads a batch of images and preprocess them
parameter images_folder: path to folder storing images
parameter height: image height in pixels
parameter width: image width in pixels
parameter size_limit: number of images to load. Default is 0 which means all images are picked.
return: list of matrices characterizing multiple images
"""
image_names = os.listdir(images_folder)
if size_limit > 0 and len(image_names) >= size_limit:
batch_filenames = [image_names[i] for i in range(size_limit)]
else:
batch_filenames = image_names
unconcatenated_batch_data = []
for image_name in batch_filenames:
    image_filepath = images_folder + "/" + image_name
    pillow_img = Image.new("RGB", (width, height))
    pillow_img.paste(Image.open(image_filepath).resize((width, height)))
    input_data = numpy.float32(pillow_img) - numpy.array(
        [123.68, 116.78, 103.94], dtype=numpy.float32
    )
    nhwc_data = numpy.expand_dims(input_data, axis=0)
    nchw_data = nhwc_data.transpose(0, 3, 1, 2)  # ONNX Runtime standard
    unconcatenated_batch_data.append(nchw_data)
batch_data = numpy.concatenate(
    numpy.expand_dims(unconcatenated_batch_data, axis=0), axis=0
)
return batch_data

class ResNet50DataReader(CalibrationDataReader):
def init(self, calibration_image_folder: str, model_path: str):
self.enum_data = None
    # Use inference session to get input shape.
    session = onnxruntime.InferenceSession(model_path, providers=['CPUExecutionProvider'])
    (_, _, height, width) = session.get_inputs()[0].shape

    # Convert image to input data
    self.nhwc_data_list = _preprocess_images(
        calibration_image_folder, height, width, size_limit=0
    )
    self.input_name = session.get_inputs()[0].name
    self.datasize = len(self.nhwc_data_list)

def get_next(self):
    if self.enum_data is None:
        self.enum_data = iter(
            [{self.input_name: nhwc_data} for nhwc_data in self.nhwc_data_list]
        )
    return next(self.enum_data, None)

def rewind(self):
    self.enum_data = None

import torch
from PIL import Image
import numpy as np
import onnxruntime
import os
import time
start_time=time.time()
from torchvision import transforms as transforms
onnx_model_path='uodated20.onnx'
session=onnxruntime.InferenceSession(onnx_model_path, providers=["CPUExecutionProvider"])
input_names = session.get_inputs()[0].name
def get_lable(filename):
filename = filename.replace('.JPEG', '')
return " ".join(filename.split('_')[1:])
preprocess = transforms.Compose([
transforms.Resize((224, 224)),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
def accuracy():
with open("imagenet_classes.txt", "r") as f:
categories = [s.strip() for s in f.readlines()]
image_dir = "/home/ext/arpdubey/imagenet-sample-images/"
images = [filename for filename in os.listdir(image_dir) if filename.endswith('.JPEG')]
totalImages = len(images)
correctPrediction = 0
# count=0
for i in images:
ground_truth = get_lable(i)
    image = Image.open(image_dir + i)
    try:
        if image.mode!='RGB':
             image=image.convert('RGB')
        input_tensor = preprocess(image)
        input_tensor=input_tensor.unsqueeze(0)
        input_arr = input_tensor.cpu().detach().numpy()
        ort_outputs = session.run([], {input_names: input_arr})[0]
        output = ort_outputs.flatten()
        top5_catid = np.argsort(-output)[:5]
        if(ground_truth == categories[top5_catid[0]]):
                correctPrediction+=1
                print("hero")
    except RuntimeError:
         print("Error")
print("Total Prediction: ",totalImages)
print("Correct Prediction: ",correctPrediction)
# print("Failed Images Count: ", count)
return (correctPrediction/totalImages)

accuracy=accuracy()
print('Accuracy of the ONNX model on ImageNet dataset: {:.2%}'.format(accuracy))
end_time=time.time()
print("Time taken: ",end_time-start_time,"seconds")

`import torch
from PIL import Image
import numpy as np
import onnxruntime
import os
import time
start_time=time.time()
from torchvision import transforms as transforms
onnx_model_path='1.onnx'
session=onnxruntime.InferenceSession(onnx_model_path, providers=["CPUExecutionProvider"])
input_names = session.get_inputs()[0].name
def get_lable(filename):
filename = filename.replace('.JPEG', '')
return " ".join(filename.split('_')[1:])
preprocess = transforms.Compose([
transforms.Resize((224, 224)),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
def accuracy():
with open("imagenet_classes.txt", "r") as f:
categories = [s.strip() for s in f.readlines()]
image_dir = "/home/ext/arpdubey/imagenet-sample-images/"
images = [filename for filename in os.listdir(image_dir) if filename.endswith('.JPEG')]
totalImages = len(images)
correctPrediction = 0
# count=0
for i in images:
ground_truth = get_lable(i)
    image = Image.open(image_dir + i)
    try:
        if image.mode!='RGB':
             image=image.convert('RGB')
        input_tensor = preprocess(image)
        input_tensor=input_tensor.unsqueeze(0)
        input_arr = input_tensor.cpu().detach().numpy()
        ort_outputs = session.run([], {input_names: input_arr})[0]
        output = ort_outputs.flatten()
        top5_catid = np.argsort(-output)[:5]
        if(ground_truth == categories[top5_catid[0]]):
                correctPrediction+=1
                print("hero")
    except RuntimeError:
         print("Error")
print("Total Prediction: ",totalImages)
print("Correct Prediction: ",correctPrediction)
# print("Failed Images Count: ", count)
return (correctPrediction/totalImages)

accuracy=accuracy()
print('Accuracy of the ONNX model on ImageNet dataset: {:.2%}'.format(accuracy))
end_time=time.time()
print("Time taken: ",end_time-start_time,"seconds")`

@arpdubey, could you share information (like the quantization script) about how you quantized the model, and provide evaluation script to reproduce the accuracy?

Thanks for your response..
When I am quantizing nodes present in the other half of resnet model, it is giving me good accuracy while if I am quantizing starting nodes then accuracy drops down to 0%

how is your calibration data selected?

I have just randomly took it.. Any suggestions from you regarding selection of calibration data?

I had the same problem too, until I modified the normalization step of image preprocessing of resnet50_data_reader.py:
diff --git a/quantization/image_classification/cpu/resnet50_data_reader.py b/quantization/image_classification/cpu/resnet50_data_reader.py
index d07198f..9ebd899 100644
--- a/quantization/image_classification/cpu/resnet50_data_reader.py
+++ b/quantization/image_classification/cpu/resnet50_data_reader.py
@@ -25,9 +25,8 @@ def _preprocess_images(images_folder: str, height: int, width: int, size_limit=0
         image_filepath = images_folder + "/" + image_name
         pillow_img = Image.new("RGB", (width, height))
         pillow_img.paste(Image.open(image_filepath).resize((width, height)))
-        input_data = numpy.float32(pillow_img) - numpy.array(
-            [123.68, 116.78, 103.94], dtype=numpy.float32
-        )
+        input_data = ((numpy.float32(pillow_img) / 255.0 - [0.485, 0.456, 0.406]) /
+                      [0.229, 0.224, 0.225]).astype(numpy.float32)
         nhwc_data = numpy.expand_dims(input_data, axis=0)
         nchw_data = nhwc_data.transpose(0, 3, 1, 2)  # ONNX Runtime standard
         unconcatenated_batch_data.append(nchw_data)

Does this mean the current code is incomplete?
BTW, there's discussion about those magic numbers [123.68, 116.78, 103.94], and it can be seen that they are just [0.485, 0.456, 0.406] * 255.0.


Bug 2069759: Report Better s3 errors

use aws/awserr package to report
better errors with correct format
Signed-off-by: rakeshgm<EMAIL_ADDRESS>(cherry pick from commit 040137e88af0300e65c9d05719a84472e0b3ff4f)
(cherry pick from commit 303b5ff7864fbe68920a4d43261157745dc7f355)

/bugzilla refresh

the fix is in 4.16 branch, will open a backport PR when a bz clone for 4.15 is created


Powering vector of pauli components

Add method to calculate (aI+bX+cY+dZ)^k for k integer as this can then be used to power a depolarizing channel.

AKA implement nearly-quaternion arithmetic. We don't have a way to even make linear combinations of Paulis yet, right? I think there's a series of changes leading up to this change that need to be described.

I'll review where we're at and will post a plan.

OK so this is actually connected to Fourier transforms over non-abelian groups!  Let me try to explain.  TLDR: take the QFT over the Pauli group of the a,b,c,d coeeficients, raise these to a power, and then inverse QFT to recover new Pauli vector!
Let G be a finite group.  Functions on this group to some field (lets just say complex numbers for now)  f: G -> C form a nice ring.  A convenient way to think about this is to represent \sum_x a_x x where a_x are elements of C and x is the group element.  Then you can see that you can multiply such objects together and represent it as another such object (\sum_x a_x x \sum_y b_y y  = \sum_{xy} a_x b_y xy and xy is just another member of the group), and you can also add and subtract linear combinations of them (\sum a_x x + \sum b_x x = \sum (a_x + b_x) x etc).
Now recall that the Fourier transform is a map from functions over G to irreducible representations of G. Lets say the irreps of G are labeled by \rho and that the matrix representation of this irrep at x \in G is the matrix rho_{i,j}(x).  Then  the Fourier transform over G is a mapping that takes functions f: G -> C to functions \tilde{f} : \rho_{i,j} -> C.  Explicitly the FT of f over G at \rho_{i,j} is
\sum_x f(x) \rho_{i, j} (x)
The FT over G of a function f, just like normal FTs retains all of the information about f.  That is given all the \sum_x f(x) \rho_{i,j}(x) over all \rho,i,j values we can perform the inverse FT to recover F.  The inverse FT is given by
f(x) = {1 \over |G|} \sum_\rho d_\rho Tr( \rho(x^{-1}) \tilde{f}(\rho))
where d_\rho is the degree of the irrep \rho and the stuff inside the trace is the matrix product of the irreps.
Now suppose we want to calculate
[\sum_x a_x x]^2
We can write this as
[\sum_x a_x x ][\sum_y a_y y]
= \sum_{x,y} a_x a_y (xy)
= \sum_{x, z) a_x a_{x^{-1} z} z
where we have used z = xy.  In other words his is a new vector
\sum c_z z
where
c_z = \sum_g a_x a_{z x^{-1}}
This is the convolution of a_g with itself.
The trick now is that the Fourier transform of the convolution of two functions f and g (or f with itself) is equal to the product of the Fourier transform o f the functions of f and g.
Let's prove this.  Let f * g be the convolution of f and g, i.e. it is the new function
(f * g)(x) = \sum_y f(y)g(y^{-1} x)
Let's take the FT of this
\tilde(f * g)(\rho_{i,j}) = \sum_ z (f * g)(z) \rho_{i,j} (z)
= \sum_z \sum_y f(y) g(y^{-1} z) \rho_{i,j}(z)
= \sum_z \sum_y f(y) g(y^{-1} z) \rho_{i, k}(y) \sum_k \rho{k, j}(y^{-1}z)
= \sum_y f(y) \sum_k \rho_{i,k}(y) \sum_{y^{-1}z} \rho_{k,j} (y^{-1} z) g(y^{-1}z)
= \sum_y f(y) \sum_k \rho_{i, k}(y)  \sum_w g(w) \rho_{k,j}(w)
= \tilde(f)(\rho_{i,k}) \tilde(g)(\rho_{k,j})
Using all of this we now see that we can use this to calculate things like
[\sum_x a_x x]^4

The a_x represent a function from G to C.
Perform a FT over G on the a_x.  This produces a new set of numbers.  These numbers are labeled by \rho, i, and j (where i and j go from 0 to d_\rho -1 ):
\sum_x a_x \rho_{i,j}(x)
We have a bunch of numbers over \rho_{i,j}, which we can think of as a matrix of size d_\rho.
Take this matrix and raise it to the rth power.  We now have a matrix m_{i,j} for each irrep \rho
Perform an inverse QFT for these numbers over all irreps \rho.  This will map you back into a function from G to c.
That vector is your answer!

So nice that we didn't have to use the non abelian QFT (though I guess you did behind the scenes).

Yeah, the algorithm implemented comes out of a derivation which is probably equivalent to abelian QFT, though I didn't show this. Might be a subject for another math note when I get a chance! In this case, you'll definitely see it!


On which computer do you use onoff?

I'd be very interested in knowing on which computers people use onoff.
The Raspberry Pi?
The BeagleBone or BeagleBone Black?
Maybe something else?
If you happen to read this and use onoff, I would be very, very grateful if you posted a comment below saying on which computer you use onoff.

The BeagleBone, the BeagleBone Black, and the Raspberry Pi.

Hi Brian, I use onoff mainly on the Raspberry Pi, B and A+ models

@8power8 Thanks for the feedback :+1:

I did a search on github to see which computers were being used. It's the Raspberry Pi most of the time. I guess that was to be expected.


reset command should perform a backup

just a simple as putting a ./celerystalk admin backup line in the rest.sh script

copy lib.administrative.backup_all_workspaces(backup_file=arguments["-f"]) and drop it on line 492 of celerystalk


Multiprocessing data loader: work in progress.

Issue, if available:
With a multiprocessing data loader we should overcome data loading bottlenecks.
Will fix issue https://github.com/awslabs/gluon-ts/issues/682.
Description of changes:

Datasets have now ReplicaInfo that clearly indicates what subset of data they are responsible for (non cyclic, i.e. validation or test dataset), or how long their burn in is (in cyclic, training dataset).
Datasets are replicated among workers (only Object reference though, not the physical dataset), this happens exactly one in the beginning of the training
Datasets are not cached by default (caching not implemented so far)
Data loading can now be done in a multiprocessing fashion by specifying the number of workers, this works for training set and validation set (for inference there is some bug for now, but that has the least impact on performance of all datasets)
Parallelisation for non cyclic, aka finite datasets: the underlying time series ids are distributed among the workers // attention: due to how we sample using the splitter from longer time series (the splitter keeps track of the average ts length: I think we should pre-calculate this), this will not produce exact same results as a single thread
Parallelisation for cyclic datasets: workers do a burn in (i.e. skipping over entries) proportional to their id: burn_in_id = int((worker_id/num_workers)*dataset_length) // however, this does not guarantee that the batches will always be sampled from equidistant locations, since some workers could potentially be slower or faster
The data loaders return batches of transformed samples of batch_size in the requested context. The transformation is done according to the provided transformation.
There is no threading support (wouldn't make sense since we are also doing computation heavy transformations), there is no memory pinning support (not necessary since we load the batches into the right context right away)
Which exact batches and samples one gets is nondeterministic if num_workers > 0

Future extensibility:

the main functions to modify will be the batching function and the stacking function, the transformation can already be replaced to any that produces a list of samples if applied to a dataset

Missing functionality:

Shuffling
Dataset caching
Correct documentation

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

So I ran some preliminary tests, and it seem we could get the 3x performance improvement @vafl described, especially for preprocessing heavy models like WaveNetEstimator
There are some things to be done though:

There is a bug with the InferenceDataLoader where the transformed data cannot be pickled when in multiprocessing (thus mp disabled by default there)
** Not such a big problem for now, would only affect inference performance
I also think the data should be transformed to the proper context in the data-loader directly, and thus we don't need memory pinning, the appropriate places have been marked with TODO (also the superfluous transformation to correct context outside of data loader can then be removed)
Finally, I didn't get to reimplement shuffling
Additionally, reintroducing explicit caching mechanisms could be of interest (I removed the caching bug)
I made a couple of other TODO annotation, however, many can be delayed for another pull request

Also, I didn't get to thoroughly and rigorously test everything, so that is to be done too.

This is great!
So none of the TODOs are blocking, right?

I supplemented the proper context handling (i.e. using the provided one in the data loader). Additionally fixed a nondeterminism in the Iterator resetting mechanism for non cyclic datasets, now its deterministic.
So from my side, there is no blocking issue I'm aware of. Pytests pass on my machine with multiple threads and single threads reliably.
Overall I think there are many small improvements to make, which I did not have the time for to do so far, but they are documented in the TODO. I also left some useful debugging prints commented out for the person who will finish this PR.

What is left to do now:

figure out the correct multiprocessing dependency

What needs improvement:

I had to change some of the burn in logic handling (not the idea of what it's supposed to do) and I don't like it, it's not really elegant.
Same situation with how requesting batches is handled when for example Iterators are depleted when using a ValidationDataLoader.

Regarding the pytests:

They should be good enough to disclose braking changes, and also cover some soft constraints that should reveal if the iteration mechanics change drastically

I see one of the tests timed out: "Python package / build (3.7, ubuntu-latest) (pull_request)"
Might want to set num_batches_per_epoch=int(5 * exp_num_batches) to just num_batches_per_epoch=int(2.5 * exp_num_batches), or something. Or put it in an extra test for more time.

I got the following error after ~1h of training:
INFO:gluonts.trainer:Epoch[58] Evaluation metric 'epoch_loss'=8.910754
Worker timed out after 120 seconds. This might be caused by

                - Slow transform. Please increase timeout to allow slower data loading in each worker.
                - Insufficient shared_memory if `timeout` is large enough.
                Please consider reduce `num_workers` or increase shared_memory in system.

#0150it [00:00, ?it/s]#0150it [02:03, ?it/s, validation_avg_epoch_loss=8.88]
Traceback (most recent call last):
  File "sm_entry_point.py", line 56, in <module>
    train_and_eval_model.main(parser.parse_args())
  File "/opt/ml/code/havel_anomaly_scoring/train_and_eval_model.py", line 330, in main
    model_dir=model_dir
  File "/opt/ml/code/havel_anomaly_scoring/train_and_eval_model.py", line 116, in get_model
    rnn_predictor = rnn.train(train_ds, validation_ds)
  File "/opt/ml/code/havel_anomaly_scoring/gluonts_models/rnn/_estimator.py", line 546, in train
    train_output = self.train_model(training_data, validation_data)
  File "/opt/ml/code/gluonts/model/estimator.py", line 213, in train_model
    validation_iter=validation_data_loader,
  File "/opt/ml/code/gluonts/trainer/_base.py", line 305, in __call__
    epoch_no, validation_iter, is_training=False
  File "/opt/ml/code/gluonts/trainer/_base.py", line 231, in loop
    for batch_no, data_entry in enumerate(it, start=1):
  File "/usr/local/lib/python3.6/dist-packages/tqdm/std.py", line 1108, in __iter__
    for obj in iterable:
  File "/opt/ml/code/gluonts/dataset/loader.py", line 103, in __iter__
    for batch in self.parallel_data_loader:
  File "/opt/ml/code/gluonts/dataset/parallelized_loader.py", line 465, in __iter__
    next_batch = next(self)
  File "/opt/ml/code/gluonts/dataset/parallelized_loader.py", line 421, in __next__
    got = ret.get(self._timeout)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 640, in get
    raise TimeoutError
multiprocessing.context.TimeoutError

There was only 50% memory usage, so it wasn't an OOM issue.
@AaronSpieler  Do you have an idea what the cause could be?

Slow transform. Please increase timeout to allow slower data loading in each worker.

Since our Tranformations are somewhat unpredictable, it could really just be that the worker caught a super long time series and then sampled a lot of points from it which ultimately led to the timeout: i.e. Slow transform. Please increase timeout to allow slower data loading in each worker.
Would be interesting to see how long the average batch processing takes in your case:

to check that just measure the time in _worker_fn from the very beginning to right before the return. If the values are close to 120 sec my hypothesis is very much possible.

Slow transform. Please increase timeout to allow slower data loading in each worker.

Since our Tranformations are somewhat unpredictable, it could really just be that the worker caught a super long time series and then sampled a lot of points from it which ultimately led to the timeout: i.e. Slow transform. Please increase timeout to allow slower data loading in each worker.
Would be interesting to see how long the average batch processing takes in your case:

to check that just measure the time in _worker_fn from the very beginning to right before the return. If the values are close to 120 sec my hypothesis is very much possible.

To combat this we could try batch size proportional timeouts. Or keep track of average batch processing time and set the timeout to 3x.
On the other hand, im not quite sure where shared_memory is defined, so could be that shared_memory has a lower bound than the total memory available.

I ran some more tests.
There are a few problems:
The json lines dataset is too slow, so even with multiprocessing it doesn’t speed things up. I then tried parquet files, which were much faster and gave the expected speedup.
However, with longer training runs at some point workers time out. Not sure why this is happening (memory usage is only ~10% on the machine).

Codecov Report

Merging #689 into master will increase coverage by 47.00%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master     #689       +/-   ##
===========================================
+ Coverage   37.49%   84.50%   +47.00%
===========================================
  Files         171      171
  Lines       10822    10822
===========================================
+ Hits         4058     9145     +5087
+ Misses       6764     1677     -5087

Impacted Files
Coverage Δ

src/gluonts/core/exception.py
62.50% <0.00%> (+6.25%)
:arrow_up:

src/gluonts/model/trivial/constant.py
76.92% <0.00%> (+11.53%)
:arrow_up:

src/gluonts/kernels/_kernel_output.py
87.09% <0.00%> (+12.90%)
:arrow_up:

src/gluonts/model/npts/_estimator.py
100.00% <0.00%> (+14.28%)
:arrow_up:

src/gluonts/block/enc2dec.py
91.66% <0.00%> (+16.66%)
:arrow_up:

src/gluonts/dataset/artificial/_base.py
41.22% <0.00%> (+16.95%)
:arrow_up:

src/gluonts/kernels/_periodic_kernel.py
66.66% <0.00%> (+19.99%)
:arrow_up:

src/gluonts/dataset/util.py
74.54% <0.00%> (+21.81%)
:arrow_up:

src/gluonts/block/cnn.py
56.41% <0.00%> (+23.07%)
:arrow_up:

src/gluonts/shell/serve/app.py
50.00% <0.00%> (+23.25%)
:arrow_up:

... and 114 more

Thanks @AaronSpieler this is really great. It works like a charm now.
Before we merge it, I have a few more points I would like to discuss:

The set_replica_info method is a bit invasive for the dataset. Should we just go with the pytorch approach and have a method to get the WorkerInfo that contains the number of workers and the worker id. Then each dataset can decide on its own how to use this (i.e.  burn-in,  modulo, or split by file etc).
For the ListDataset (and also raw lists) we could do something special: instead of passing the full list to each worker, we cut the list up before and pass separate chunks to each worker. That way we would support multiprocessing for pure lists directly and it would also drastically reduce the memory consumption for multiprocessing with lists.
To make it clearer to the user, can we make the num_workers an explicit parameter for the .train method instead of passing this via kwargs?

What do you think?

Thanks @AaronSpieler this is really great. It works like a charm now.
Before we merge it, I have a few more points I would like to discuss:

The set_replica_info method is a bit invasive for the dataset. Should we just go with the pytorch approach and have a method to get the WorkerInfo that contains the number of workers and the worker id? Then each dataset can decide on its own how to use this (i.e.  burn-in,  modulo, or split by file etc).
For the ListDataset (and also raw lists) we could do something special: instead of passing the full list to each worker, we cut the list up before and pass separate chunks to each worker. That way we would support multiprocessing for pure lists directly and it would also drastically reduce the memory consumption for multiprocessing with lists.
To make it clearer to the user, can we make the num_workers an explicit parameter for the .train method instead of passing this via kwargs?

What do you think?

A method get_worker_info should be possible and is less invasive, so I think it's a great idea.
Giving each worker a subset of the data from the beginning is unfortunately not possible  at the moment since we use the worker Pool paradigm, where each worker gets the exact same data to begin with, however, if we manually manage processes at some point this should be possible, and we could also request specific processes for batches, but this would require a major refactoring.
num_workers can of course be an explicit argument, but then the question is how many arguments do we want to expose explicitly? For example num_prefetch could also be an interesting argument.

* Giving each worker a subset of the data from the beginning is unfortunately not possible  at the moment since we use the worker Pool paradigm, where each worker gets the exact same data to begin with, however, if we manually manage processes at some point this should be possible, and we could also request specific processes for batches, but this would require a major refactoring.

Ah, of course. We would have to start each worker separately (or send the data chunk first thing via the queue).

* `num_workers` can of course be an explicit argument, but then the question is how many arguments do we want to expose explicitly? For example `num_prefetch` could also be an interesting argument.

Yes, sure. BTW, if we think we need to experiment more to stabilize it, we can merge it with num_workers=0. Then people can start trying it if they want.

Ah, of course. We would have to start each worker separately (or send the data chunk first thing via the queue).

The problem with a queue, even if feasable, would be that we don't get the desired speedup, because the queue is blocking.

Yes, sure. BTW, if we think we need to experiment more to stabilize it, we can merge it with num_workers=0. Then people can start trying it if they want.

I think that is a good idea.

Ah, of course. We would have to start each worker separately (or send the data chunk first thing via the queue).

The problem with a queue, even if feasable, would be that we don't get the desired speedup, because the queue is blocking.

The idea would be to only use it to send the whole dataset initially. It is currently pickled and send aswell.

Ah, of course. We would have to start each worker separately (or send the data chunk first thing via the queue).

The problem with a queue, even if feasable, would be that we don't get the desired speedup, because the queue is blocking.

The idea would be to only use it to send the whole dataset initially. It is currently pickled and send aswell.

I'm not sure whether when the processes retrieve the subsets of data from the queue they wouldn't block each other until the respective data part is de-pickled?

Thanks to @jaheba, @lostella and especially @vafl for the help!


Backport enum generator refactor (#2334) to 0.54.x

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Covered in #2350.


Computed property not working after hydration

I have something like this:
import { observable, computed } from "mobx";

export class EnvironmentModel {
  @observable public id: string;
  @observable public name: string;

  constructor(id: string, name: string) {
    this.id = id;
    this.name = name;
  }

   @computed get initials() {
     let tokens = this.name.split(" ");
     let res = "blah";
     if (tokens.length === 1) {
       res = tokens[0].slice(0, 2);
     } else {
       res = tokens[0][0] + tokens[1][0];
     }

     return res;
   }

}

import { EnvironmentModel } from "../model";
import { RootStore } from "./Root.store";
import { observable, action } from "mobx";
import { ignore } from "mobx-sync";

let localEnv = new EnvironmentModel("local", "Local Environment");

export class EnvironmentStore {
  @ignore root: RootStore;

  @observable.shallow public current: EnvironmentModel = localEnv;
  @observable.shallow
  public environments: EnvironmentModel[] = observable.array([localEnv]);

  constructor(root: RootStore) {
    this.root = root;
  }

  // TODO: Placeholder function
  @action
  public loginToEnvironment(envName: string, email: string, password: string) {
    let newEnv = new EnvironmentModel(envName, envName);
    this.environments.push(newEnv);
    this.switchEnvironment(newEnv.id);
  }

  @action switchEnvironment(envId: string) {
    let env = this.environments.find(e => e.id === envId);
    if (!env) {
      // do something a toast or w/e
      return;
    }

    this.current = env;
  }
}

Everything works fine the first time when the objects are created on runtime, but once my store is rehydrated upon application load, the computed initials is no longer working and returns undefined.
Any idea why this might be happening? Am I doing something wrong?

I just checked again and it seems upon hydration an observable proxy is created, but it is not the same type? it doesn't have the initials property set

I needed to specify how to de-serialize the object:
@format((data: Array<EnvironmentModel>) =>
    data.map(d => new EnvironmentModel(d.id, d.name))
  )```

Maybe adding a bit more of documentation to the readme would be useful for future user


NixOS 20.09 and onwards broke systemd predictable network interface naming

I believe that this change
https://github.com/NixOS/nixpkgs/commit/822918df4cec0ee14a657efb86a08bf7f457e647#diff-344943ac406571508ecc8d0311e237519c34548ffaf4eba83a15758a1ced65a4R57

nixos/scripted-networking: use udev to configure link MACAddress and MTUBytes

accidentally broke systemd's PredictableNetworkInterfaceNames.
For example, for my Nixops Hetzner machine, on 20.03, the default interface eth0 got renamed to enp6s0, but on 21.11 it stays eth0, making it lose Internet on the upgrade.

I noticed this because my Hetzner server lost Internet during the upgrade and I spent > 16 hours figuring out why.
diffing the output of udevadm info -e between 20.03 and 21.11, I found:
-E: ID_NET_LINK_FILE=/nix/store/vzbkzyw5x3ilqj6wg0q8l5gzcli5pn5f-systemd-243.7/lib/systemd/network/99-default.link
+E: ID_NET_LINK_FILE=/etc/systemd/network/40-eth0.link

On 20.03 that default file /nix/store/abc123...-systemd-243.7/lib/systemd/network/99-default.link contained:
[Match]
OriginalName=*

[Link]
NamePolicy=keep kernel database onboard slot path
MACAddressPolicy=persistent

This just says to do the usual systemd name policy finding as linked above.
But on 20.09 and onwards, /etc/systemd/network/40-eth0.link contains:
[Match]
OriginalName=eth0

[Link]

The [Link] section is empty; that is probably the reason why there is none of the expected systemd.log_level=debug output like this, and why no renaming is happening.
Reading https://www.freedesktop.org/software/systemd/man/systemd.link.html, I am not even sure if it's allowed for [Link] to be empty; the systemd docs don't seem to specify what happens in that case.
As an aside, running what's suggested in section **Example 3. Debugging NamePolicy= assignments on 21.11 produces:
Parsed configuration file /nix/store/q0881awy50g4srnnwasci37y2jk5sf99-systemd-249.5/lib/systemd/network/99-default.link
Parsed configuration file /etc/systemd/network/40-eth0.link
Created link configuration context.
ID_NET_DRIVER=igb
eth0: Failed to query name_assign_type: Invalid argument
eth0: Config file /etc/systemd/network/40-eth0.link is applied
eth0: Failed to get ACTION= property: No such file or directory
eth0: Could not apply link configuration, ignoring: No such file or directory
ID_NET_LINK_FILE=/etc/systemd/network/40-eth0.link

I am wondering:
Shouldn't the generated 40-eth0.link from that commit contain as default contents systemd's default values, namely:
NamePolicy=keep kernel database onboard slot path
MACAddressPolicy=persistent

Or was that intentionally not put in?
CC @flokli as author of the original commit
CC @aszlig because you might be interested in this

To elaborate a bit further on my specific problem with my Hetzner NixOps machine:
My machine has 2 network interfaces, a 1G (not in use) and a 10G (in use) one.
Nixops Hetzner integration generated into its nixops sqlite database this udev rule:
"extraRules": "ACTION==\"add\", SUBSYSTEM==\"net\", ATTR{address}==\"6c:b3:11:08:77:cc\", NAME=\"eth0\"\n"}, ...

as well as:
{"networking": ... "interfaces": {"eth0": {"ipv4": {"addresses": [...]

Because networking.interfaces.eth0 is defined, that quite-empty 40-eth0.link is generated (by this line):
[Match]
OriginalName=eth0

[Link]

Now, what's supposed to happen, and what worked in 20.03, is that

systemd would rename the 1G interface eth0 -> enp6s0
The nixops-hetzner-generated udev rule would rename the 10G interface eth1 -> eth0 based on the MAC address

But because of [Link] being empty, the rename (1) does not happen, so (2) fails with:
systemd-udevd[605]: eth1: Failed to rename network interface 3 from 'eth1' to 'eth0': File exists

Side note, I did notice that the 20.09 changelog introduced by that commit says:

The scripted networking system now uses .link files in /etc/systemd/network to configure mac address and link MTU, instead of the sometimes buggy network-link-* units, which have been removed. Bringing the interface up has been moved to the beginning of the network-addresses-* unit. Note this doesn't require systemd-networkd - it's udev that parses .link files. Extra care needs to be taken in the presence of legacy udev rules to rename interfaces, as MAC Address and MTU defined in these options can only match on the original link name. In such cases, you most likely want to create a 10-*.link file through systemd.network.links and set both name and MAC Address / MTU there.

I guess

Extra care needs to be taken in the presence of legacy udev rules to rename interfaces

Is somewhat relevant.
However, I don't think that the changelog gives any hint at the problem in the issue description: It doesn't mention that it would disable persistent renaming at all, which is why I suspect that this was an accident, and is a bug.

As another aside (cc @aszlig):

Nixops-hetzner renaming an interface to eth0 is evidently a bad idea. When somebody plugs in another card, that name can be taken. It should rename it to a name that Linux and Systemd will not create, like internet0, so that there can be no such conflict.
systemd docs have the same opinion in: https://www.freedesktop.org/software/systemd/man/systemd.link.html#Name=
".

Note that specifying a name that the kernel might use for another interface (for example "eth0") is dangerous because the name assignment done by udev will race with the assignment done by the kernel, and only one interface may use the name. Depending on the order of operations, either udev or the kernel will win, making the naming unpredictable. It is best to use some different prefix, for example "internal0"/"external0" or "lan0"/"lan1"/"lan3".

That part may need to be addressed in nixops-hetzner, perhaps as a combination of:

Switching from the generated udev extraRules for renaming to a .link file
Not using names eth0 as the target names as suggested.

One way to make this more graceful could have been to add the NamePolicy/MacAddressPolicy statements at https://github.com/NixOS/nixpkgs/commit/822918df4cec0ee14a657efb86a08bf7f457e647#diff-344943ac406571508ecc8d0311e237519c34548ffaf4eba83a15758a1ced65a4R59 (only for scripted networking).
Back then, we assumed it was cleaner to have people do that migration once, to the more stable predictable network interface names. It has been documented in the release notes, and with this being active since 20.09, there's not really a way to change it after the fact (and breaking people's setups again backwards to reintroduce the old naming scheme).
nixops-hetzner never did the necessary changes in their custom (and error-prone) config. This should be fixed there - either by seting above NamePolicy/MacAddressPolicy values by yourself, or update nixops-hetzner to not use the error-prone eth0 names.
Closing this one, it's not an nixpkgs issue anymore, but a nixops-hetzner bug.


set up performance integration tests to also run analysis server perf tests with the CFE

We have existing analysis server perf tests; we should duplicate these, and run the 2nd set with the --use-cfe cli option.

@sortie, when you're back from break, I think we're ready to duplicate the analysis server benchmarks to run in non-cfe and --use-cfe mode.


Fixup libuuid destdir

Trying alternatives to #8

the prefix-only commit does not work on my local setup. Same issue as before the PR.


injector:modulerr

I keep getting this error when trying to load this service.
I have included facebook as a dependency:
angular.module('myapp', ['facebook'])

And created this config:
.config(function(FacebookProvider) {
	var appID = 'XXXXXXXXXX';
	FacebookProvider.init(appID);
});

Am I doing something wrong?

Found the problem.
So I have all my configs separated out in separate files. So I had something like this:
// Facebook Config

angular.module('myapp.config')

.config(function(FacebookProvider) {
	var appID = 'XXXXXXXXXX';
	FacebookProvider.init(appID);
});

Then in my main app.js I have the configs included as a dependency.
angular.module('myapp', ['myapp.config', 'facebook'])

Apparently this doesn't work, after deleting the facebook config file and modifying my code to include the config right after the .module() call it works:
angular.module('myapp', ['myapp.config', 'facebook'])
	   .config(function(FacebookProvider) {
			var appID = 'XXXXXXXXXX';
			FacebookProvider.init(appID);
		});


please also add a barcode reader

there is a barcode reader in doc.
but when I install the library there is only a QR code reader

can you be more specific?

how can I scan a barcode.
there is only a scanQrCodes

please see https://github.com/rodgomesc/vision-camera-qrcode-scanner/issues/14


Port/data.matrix.notation

[x] depends on: #3215

This PR/issue depends on:

leanprover-community/mathlib4#3215
By Dependent Issues (🤖). Happy coding!

This PR/issue depends on:

leanprover-community/mathlib4#3215
By Dependent Issues (🤖). Happy coding!

I think the cause of the error is that em' is reduced to toTypeExpr m' and Qq tries to antiquote toTypeExpr m' nevertheless it doesn't have Q(_) type.
This problem is solved by replace let with have.

Let's wait for mathport support before considering merging this.

Let's wait for mathport support before considering merging this.

I think you want more than mathport support for this. Is the notation itself implemented yet? Otherwise I don't see how you can actually use it in the file.

Let's wait for mathport support before considering merging this.

Well… https://github.com/leanprover-community/mathport/pull/235 is ready now.

Let's wait for mathport support before considering merging this.

Well… leanprover-community/mathport#235 is ready now.

Great! What I mean is that we should wait for that to be merged and mathlib3port to be regenerated. Until then, all the downstream files in mathport will have holes in where this notation is used; so we don't want people trying to port them yet.

I merged the mathport PR. Hopefully tomorrow we will see what happens in mathlib3port.

This PR/issue depends on:

leanprover-community/mathlib4#3215
leanprover-community/mathlib4#3430
leanprover-community/mathport#235
By Dependent Issues (🤖). Happy coding!

This PR/issue depends on:

leanprover-community/mathlib4#3215
leanprover-community/mathlib4#3430
leanprover-community/mathport#235
leanprover-community/mathlib4#3485
By Dependent Issues (🤖). Happy coding!

bors merge


Make habitat-sim conda installable

🚀 Feature

Motivation
That involve multiple steps to clone, build and install habitat-sim and is limiting adoption of the framework by community dramatically.
Pitch
Produce and publish pre-built binaries of habitat-sim for Ubuntu and make it conda installable (similar to PyTorch install): conda install habitat_sim -c habitat

That will decrease framework usage threshold significantly and make our Continues Integration setup simpler.
Additional context
As reference point PyTorch night builds CI system can be checked and partially replicated.

Yes! This will be a high-impact feature!

Reference point to previous @erikwijmans effort to make it done: https://github.com/facebookresearch/habitat-sim/tree/conda-build.

@mpiseno -- now that #445 has landed, can we add instructions in the README on how to use the conda build?

@dhruvbatra Hello, I was trying to install Habitat sim onto my linux but for some reason it didn't build properly, so I was looking for an alternative solution when I came across this thread. I was wondering if you could point me to the conda-installable habitat sim solution. I tried
conda install -c aihabitat -c conda-forge habitat-sim
which I found in the README, but it produced the following errors:
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Package xorg-libx11 conflicts for:
habitat-sim -> xorg-libx11
Package bzip2 conflicts for:
python=3.6 -> bzip2[version='>=1.0.6,<2.0a0']
Package ld_impl_linux-64 conflicts for:
python=3.6 -> ld_impl_linux-64
Package python_abi conflicts for:
python=3.6 -> python_abi[version='*|3.6',build='*_pypy36_pp73|*_cp36m']
wheel -> python_abi=[build='*_cp38|*_cp36m|*_cp35m|*_cp27mu|*_cp34m|*_cp37m']
pip -> python_abi=[build='*_cp38|*_cp36m|*_cp35m|*_cp27mu|*_cp34m|*_cp37m']
certifi -> python_abi[version='2.7.*|3.6.*|3.7.*|3.8.*|3.6',build='*_pypy36_pp73|*_cp37m|*_cp36m|*_cp27mu|*_cp38']
habitat-sim -> python_abi=3.6[build=*_cp36m]
setuptools -> python_abi[version='3.6|3.6.*|3.7.*|3.8.*',build='*_pypy36_pp73|*_cp38|*_cp37m|*_cp36m']
Package ncurses conflicts for:
readline -> ncurses[version='5.9.*|6.0.*|>=6.0,<7.0a0|>=6.1,<6.2.0a0|>=6.1,<7.0a0']
sqlite -> ncurses[version='5.9|5.9.*|>=6.1,<6.2.0a0|>=6.2,<7.0a0']
libedit -> ncurses[version='5.9|6.0.*|>=6.1,<6.2.0a0|>=6.1,<7.0a0']
python=3.6 -> ncurses[version='5.9|5.9.*|6.0.*|>=6.0,<7.0a0|>=6.1,<6.2.0a0|>=6.1,<7.0a0|>=6.2,<7.0a0']
Package libstdcxx-ng conflicts for:
python=3.6 -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0']
libffi -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0']
sqlite -> libstdcxx-ng[version='>=4.9|>=7.3.0']
ncurses -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0']
Package scipy conflicts for:
habitat-sim -> scipy[version='>=1.3.0']
Package libffi conflicts for:
python=3.6 -> libffi[version='3.2.*|>=3.2.1,<3.3.0a0|>=3.2.1,<3.3a0']
Package quaternion conflicts for:
habitat-sim -> quaternion
Package attrs conflicts for:
habitat-sim -> attrs[version='>=19.1.0']
Package openssl conflicts for:
python=3.6 -> openssl[version='1.0.*|1.0.*,>=1.0.2l,<1.0.3a|>=1.0.2m,<1.0.3a|>=1.0.2n,<1.0.3a|>=1.0.2o,<1.0.3a|>=1.0.2p,<1.0.3a|>=1.1.1a,<1.1.2a|>=1.1.1c,<1.1.2a|>=1.1.1d,<1.1.2a|>=1.1.1e,<1.1.2a|>=1.1.1f,<1.1.2a|>=1.1.1g,<1.1.2a']
Package libgcc-ng conflicts for:
libedit -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
readline -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
tk -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
sqlite -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
openssl -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
zlib -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
xz -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
libffi -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
ncurses -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
python=3.6 -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
Package xorg-libxcursor conflicts for:
habitat-sim -> xorg-libxcursor
Package zlib conflicts for:
python=3.6 -> zlib[version='1.2.*|1.2.11|1.2.8|>=1.2.11,<1.3.0a0']
tk -> zlib[version='>=1.2.11,<1.3.0a0']
sqlite -> zlib[version='>=1.2.11,<1.3.0a0']
Package setuptools conflicts for:
pip -> setuptools
wheel -> setuptools
Package * conflicts for:
habitat-sim -> *[track_features=headless]
Package _libgcc_mutex conflicts for:
libgcc-ng -> _libgcc_mutex[version='*|0.1',build='conda_forge|main']
Package sqlite conflicts for:
python=3.6 -> sqlite[version='3.13.*|3.20.*|>=3.20.1,<4.0a0|>=3.22.0,<4.0a0|>=3.23.1,<4.0a0|>=3.24.0,<4.0a0|>=3.25.2,<4.0a0|>=3.25.3,<4.0a0|>=3.26.0,<4.0a0|>=3.28.0,<4.0a0|>=3.29.0,<4.0a0|>=3.30.1,<4.0a0|>=3.31.1,<4.0a0']
Package xorg-libxext conflicts for:
habitat-sim -> xorg-libxext
Package xorg-libxrandr conflicts for:
habitat-sim -> xorg-libxrandr
Package _openmp_mutex conflicts for:
libgcc-ng -> _openmp_mutex[version='>=4.5']
Package matplotlib conflicts for:
habitat-sim -> matplotlib
Package libxcb conflicts for:
habitat-sim -> libxcb
Package xorg-libxi conflicts for:
habitat-sim -> xorg-libxi
Package xz conflicts for:
python=3.6 -> xz[version='5.2.*|>=5.2.3,<5.3.0a0|>=5.2.3,<6.0a0|>=5.2.4,<5.3.0a0|>=5.2.4,<6.0a0|>=5.2.5,<5.3.0a0']
Package tk conflicts for:
python=3.6 -> tk[version='8.5.*|8.6.*|>=8.6.10,<8.7.0a0|>=8.6.7,<8.7.0a0|>=8.6.8,<8.7.0a0|>=8.6.9,<8.7.0a0']
Package wheel conflicts for:
pip -> wheel
Package numba conflicts for:
habitat-sim -> numba
Package certifi conflicts for:
setuptools -> certifi[version='>=2016.09|>=2016.9.26']
Package pillow conflicts for:
habitat-sim -> pillow
Package pypy3.6 conflicts for:
python=3.6 -> pypy3.6[version='7.3.0.*|7.3.1.*']
Package readline conflicts for:
python=3.6 -> readline[version='6.2.*|7.*|7.0|>=7.0,<8.0a0|>=8.0,<9.0a0']
sqlite -> readline[version='6.2.*|7.0.*|7.0|>=7.0,<8.0a0|>=8.0,<9.0a0']
Package xorg-libxinerama conflicts for:
habitat-sim -> xorg-libxinerama[version='>=1.1.4,<1.2.0a0']
Package pip conflicts for:
python=3.6 -> pip
Package xorg-libxau conflicts for:
habitat-sim -> xorg-libxau
Package numpy conflicts for:
habitat-sim -> numpy[version='>=1.13']
Package xorg-libxfixes conflicts for:
habitat-sim -> xorg-libxfixes
Package ca-certificates conflicts for:
openssl -> ca-certificates
Package libedit conflicts for:
sqlite -> libedit[version='>=3.1.20170329,<3.2.0a0|>=3.1.20181209,<3.2.0a0']
Package tqdm conflicts for:
habitat-sim -> tqdm
Package libgomp conflicts for:
libgcc-ng -> libgomp[version='7.3.0|9.2.0',build='h24d8f2e_4|h24d8f2e_5|h24d8f2e_2']

@shuishida Does your system have a display? The conda binaries for linux currently are for headless systems.
Does this work?
conda install -c aihabitat -c conda-forge habitat-sim headless

@mpiseno Ah thank you so much! Yes I have a display, but the headless version seems to work!! This is beautiful!

Now closed.

Hii, Can we install specific commit for habitat-sim via. conda command  conda install -c aihabitat -c conda-forge habitat-sim headless?

No.  We only have pre-compiled binaries for tagged versions or latest master through the nightly channel.

Thank you very much for the response, Sir. I really appreciate your help. Is there any way to install older version on google colab?

You can build from source on collab.  @Skylion007 @aclegg3 IIRC, one of you had scripts and/or a collab notebook that did at some point.  If that is correct, does it still exist?

This cell will install habitat-sim:
!wget -qO - https://apt.kitware.com/keys/kitware-archive-latest.asc | sudo apt-key add -
!apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main' #Add latest CMake version
# clang-9 llvm-9 lld-9
!apt update && apt install -y --no-install-recommends  libjpeg-dev libglm-dev libgl1-mesa-glx libegl1-mesa-dev mesa-utils  libbullet-dev cmake libomp-dev ninja-build libassimp-dev libeigen3-dev clang-9 lld-9  libclang-9-dev llvm-9-dev
#libxrandr-dev libxinerama-dev
# The default in Colab CMake 3.12 has a bug that prevents Habitat-Sim from compiling properly with CUDA
!rm -f /usr/local/bin/cmake #Delete Old CMake

%env HEADLESS=True
%cd /content/

!git clone https://github.com/facebookresearch/habitat-sim --depth 1 --recursive --jobs 16
%cd '/content/habitat-sim'
#CLANG'S LINKER IS A LOT FASTER WHICH SAVES A LOT OF TIME IN COLAB
%env CC=/usr/bin/clang-9
%env CXX=/usr/bin/clang++-9
%env LD=/usr/bin/lld-9
#We use a system wide precompiled version of ASSIMP to save compile time.
!python setup.py install --headless --bullet --cmake-args='-DUSE_SYSTEM_ASSIMP=ON'

Thank you for your quick response!! @erikwijmans and @Skylion007 . Habitat is really providing awesome platform for researcher working in embodied-AI.
Hey, I installed habitat-sim as given in above code and habitat-lab as given in https://github.com/facebookresearch/habitat-lab
And running following:
!python examples/example.py
But got following error:
2020-12-15 18:46:27,029 Initializing dataset PointNav-v1 2020-12-15 18:46:27,298 initializing sim Sim-v0 Traceback (most recent call last):   File "examples/example.py", line 28, in <module>     example()   File "examples/example.py", line 14, in example     config=habitat.get_config("configs/tasks/pointnav.yaml")   File "/content/habitat-lab/habitat/core/env.py", line 105, in __init__     id_sim=self._config.SIMULATOR.TYPE, config=self._config.SIMULATOR   File "/content/habitat-lab/habitat/sims/registration.py", line 19, in make_sim     return _sim(**kwargs)   File "/content/habitat-lab/habitat/sims/habitat_simulator/habitat_simulator.py", line 182, in __init__     self.sim_config = self.create_sim_config(self._sensor_suite)   File "/content/habitat-lab/habitat/sims/habitat_simulator/habitat_simulator.py", line 198, in create_sim_config     sim_config.scene.id = self.habitat_config.SCENE AttributeError: 'habitat_sim._ext.habitat_sim_bindings.SimulatorCon' object has no attribute 'scene' Exception ignored in: <bound method Simulator.__del__ of HabitatSim(config=NOTHING, agents=NOTHING, _num_total_frames=NOTHING, _default_agent_id=NOTHING, _Simulator__sensors=NOTHING, _initialized=NOTHING, _previous_step_time=NOTHING, _Simulator__last_state=NOTHING)> Traceback (most recent call last):   File "/usr/local/lib/python3.6/dist-packages/habitat_sim-0.1.6-py3.6-linux-x86_64.egg/habitat_sim/simulator.py", line 467, in __del__     self.close()   File "/usr/local/lib/python3.6/dist-packages/habitat_sim-0.1.6-py3.6-linux-x86_64.egg/habitat_sim/simulator.py", line 118, in close     for agent_sensorsuite in self.__sensors: AttributeError: 'HabitatSim' object has no attribute '_Simulator__sensors'
Is there anything I am missing?? Thank you in advance.

You will also need to be on a commit of habitat-lab that is compatible with that version of the simulator.  You need to be on a newer one it looks like.  I believe current master in habitat-lab is fine.

You will also need to be on a commit of habitat-lab that is compatible with that version of the simulator. You need to be on a newer one it looks like. I believe current master in habitat-lab is fine.

Hii Thanks for the support. I used current version of habitat-lab only.
Also, tried compatible git checkouts taking this as reference:https://github.com/devendrachaplot/Neural-SLAM
Then got following error:
2020-12-15 20:03:33,218 initializing sim Sim-v0
Traceback (most recent call last):
File "examples/example.py", line 25, in
example()
File "examples/example.py", line 11, in example
env = habitat.Env(config=habitat.get_config("configs/tasks/pointnav.yaml"))
File "/content/habitat-api/habitat/core/env.py", line 93, in init
id_sim=self._config.SIMULATOR.TYPE, config=self._config.SIMULATOR
File "/content/habitat-api/habitat/sims/registration.py", line 19, in make_sim
return _sim(**kwargs)
File "/content/habitat-api/habitat/sims/habitat_simulator/habitat_simulator.py", line 158, in init
self.sim_config = self.create_sim_config(self._sensor_suite)
File "/content/habitat-api/habitat/sims/habitat_simulator/habitat_simulator.py", line 171, in create_sim_config
sim_config.scene.id = self.config.SCENE
AttributeError: 'habitat_sim._ext.habitat_sim_bindings.SimulatorCon' object has no attribute 'scene'
Am I doing any mistake?  Is the code is only for some perticular habitat-sim git checkout??

Yes, that commit of habitat-lab requires an older commit of habitat-sim.  If you absolutely must use those two commits, this PR is what is missing from your habitat-lab version: https://github.com/facebookresearch/habitat-lab/pull/516

Hii thank you for this suggestion. I changed the code manually in to scene_id from scene.id where the code was breaking.
Now it is showing me following error:
2020-12-16 05:52:13,430 initializing sim Sim-v0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1216 05:52:13.433264  7440 ManagedContainerBase.cpp:19] ManagedContainerBase::convertFilenameToJSON : Filename : default changed to proposed JSON configuration filename : default.scene_dataset_config.json
I1216 05:52:13.433437  7440 AssetAttributesManager.cpp:117] Asset attributes (capsule3DSolid : capsule3DSolid_hemiRings_4_cylRings_1_segments_12_halfLen_0.75_useTexCoords_false_useTangents_false) created and registered.
I1216 05:52:13.433506  7440 AssetAttributesManager.cpp:117] Asset attributes (capsule3DWireframe : capsule3DWireframe_hemiRings_8_cylRings_1_segments_16_halfLen_1) created and registered.
I1216 05:52:13.433570  7440 AssetAttributesManager.cpp:117] Asset attributes (coneSolid : coneSolid_segments_12_halfLen_1.25_rings_1_useTexCoords_false_useTangents_false_capEnd_true) created and registered.
I1216 05:52:13.433611  7440 AssetAttributesManager.cpp:117] Asset attributes (coneWireframe : coneWireframe_segments_32_halfLen_1.25) created and registered.
I1216 05:52:13.433634  7440 AssetAttributesManager.cpp:117] Asset attributes (cubeSolid : cubeSolid) created and registered.
I1216 05:52:13.433653  7440 AssetAttributesManager.cpp:117] Asset attributes (cubeWireframe : cubeWireframe) created and registered.
I1216 05:52:13.433706  7440 AssetAttributesManager.cpp:117] Asset attributes (cylinderSolid : cylinderSolid_rings_1_segments_12_halfLen_1_useTexCoords_false_useTangents_false_capEnds_true) created and registered.
I1216 05:52:13.433756  7440 AssetAttributesManager.cpp:117] Asset attributes (cylinderWireframe : cylinderWireframe_rings_1_segments_32_halfLen_1) created and registered.
I1216 05:52:13.433782  7440 AssetAttributesManager.cpp:117] Asset attributes (icosphereSolid : icosphereSolid_subdivs_1) created and registered.
I1216 05:52:13.433805  7440 AssetAttributesManager.cpp:117] Asset attributes (icosphereWireframe : icosphereWireframe_subdivs_1) created and registered.
I1216 05:52:13.433838  7440 AssetAttributesManager.cpp:117] Asset attributes (uvSphereSolid : uvSphereSolid_rings_8_segments_16_useTexCoords_false_useTangents_false) created and registered.
I1216 05:52:13.433873  7440 AssetAttributesManager.cpp:117] Asset attributes (uvSphereWireframe : uvSphereWireframe_rings_16_segments_32) created and registered.
I1216 05:52:13.433889  7440 AssetAttributesManager.cpp:105] AssetAttributesManager::buildCtorFuncPtrMaps : Built default primitive asset templates : 12
I1216 05:52:13.434511  7440 SceneDatasetAttributesManager.cpp:23] File (default) not found, so new default dataset attributes created and registered.
I1216 05:52:13.434525  7440 MetadataMediator.cpp:47] MetadataMediator::createDataset : Dataset default successfully created.
I1216 05:52:13.438854  7440 PhysicsAttributesManager.cpp:27] File (./data/default.physics_config.json) not found, so new default physics manager attributes created and registered.
I1216 05:52:13.438887  7440 ManagedContainerBase.cpp:19] ManagedContainerBase::convertFilenameToJSON : Filename : data/scene_datasets/habitat-test-scenes/skokloster-castle.glb changed to proposed JSON configuration filename : data/scene_datasets/habitat-test-scenes/skokloster-castle.stage_config.json
I1216 05:52:13.438977  7440 AbstractObjectAttributesManagerBase.h:180] File (data/scene_datasets/habitat-test-scenes/skokloster-castle.glb) exists but is not a recognized config filename extension, so new default Stage attributes created and registered.
I1216 05:52:13.439000  7440 Simulator.cpp:153] Loading navmesh from data/scene_datasets/habitat-test-scenes/skokloster-castle.navmesh
I1216 05:52:13.439163  7440 Simulator.cpp:155] Loaded.
I1216 05:52:13.439195  7440 SceneGraph.h:93] Created DrawableGroup:
Platform::WindowlessEglApplication::tryCreateContext(): no EGL devices found, likely a driver issue; enable --magnum-gpu-validation to see additional info
WindowlessContext: Unable to create windowless context
How to solve this driver issue? Thank you in advance!!

If this is still on colab, you will need a GPU colab instance to run habitat-sim.

Thank you very much @erikwijmans , it is working now.

Hi,
I tried to install the habitat-sim using all the methods as mentioned in the respected website and even in this thread but nothing works for me. Either no response or package error as noted by @shuishida. Please guide me in resolving this issue.
Thanks

Hi,
I am having the issue of WindowlessContext: Unable to create windowless context (the same as mentioned in link. I want to use this simulator on my server PC which enables GPU in it. Please try to resolve this.
Thanks in advance.

Please open a fresh issue, answering the questions in the template.